Update README.md
Browse files
README.md
CHANGED
@@ -37,6 +37,6 @@ This repository contains all the intermediate checkpoints of OpenCoder-1.5B-Base
|
|
37 |
- `anneal_iter_0001000` - `anneal_iter_0011920`: Intermediate checkpoints during the annealing stage.
|
38 |
|
39 |
|
40 |
-
The number in each branch name indicates the corresponding current training step, where each step consumes
|
41 |
|
42 |
We use `pretrain_iter_0300000` as the starting point for the annealing stage, and use `anneal_iter_0010000` as the final base model.
|
|
|
37 |
- `anneal_iter_0001000` - `anneal_iter_0011920`: Intermediate checkpoints during the annealing stage.
|
38 |
|
39 |
|
40 |
+
The number in each branch name indicates the corresponding current training step, where each step consumes 838,8608 training tokens (2,048 batch size * 4,096 sequence length from `pretrain_iter_0001000` for `pretrain_iter_0130000`; 1,024 batch size * 8,192 sequence length for `pretrain_iter_0001000` - `pretrain_iter_0130000` and the whole annealing phase).
|
41 |
|
42 |
We use `pretrain_iter_0300000` as the starting point for the annealing stage, and use `anneal_iter_0010000` as the final base model.
|