Simingh commited on
Commit
b4a4e45
1 Parent(s): 330954b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -37,6 +37,6 @@ This repository contains all the intermediate checkpoints of OpenCoder-1.5B-Base
37
  - `anneal_iter_0001000` - `anneal_iter_0011920`: Intermediate checkpoints during the annealing stage.
38
 
39
 
40
- The number in each branch name indicates the corresponding current training step, where each step consumes 419,4304 training tokens (2,048 batch size * 4,096 sequence length from `pretrain_iter_0001000` for `pretrain_iter_0130000`; 1,024 batch size * 8,192 sequence length for `pretrain_iter_0001000` - `pretrain_iter_0130000` and the whole annealing phase).
41
 
42
  We use `pretrain_iter_0300000` as the starting point for the annealing stage, and use `anneal_iter_0010000` as the final base model.
 
37
  - `anneal_iter_0001000` - `anneal_iter_0011920`: Intermediate checkpoints during the annealing stage.
38
 
39
 
40
+ The number in each branch name indicates the corresponding current training step, where each step consumes 838,8608 training tokens (2,048 batch size * 4,096 sequence length from `pretrain_iter_0001000` for `pretrain_iter_0130000`; 1,024 batch size * 8,192 sequence length for `pretrain_iter_0001000` - `pretrain_iter_0130000` and the whole annealing phase).
41
 
42
  We use `pretrain_iter_0300000` as the starting point for the annealing stage, and use `anneal_iter_0010000` as the final base model.