Commit
•
d547cdc
1
Parent(s):
b476b46
new data
Browse files- logs/main_log.txt +6 -0
logs/main_log.txt
CHANGED
@@ -28216,3 +28216,9 @@ saving checkpoint at iteration 75000 to /gpfsscratch/rech/six/commun/synched_e
|
|
28216 |
[2021-10-02 15:42:38,338] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: /gpfsscratch/rech/six/commun/synched_exps/tr4c-1B3-rotary-oscar/checkpoints/global_step75000/mp_rank_00_model_states.pt
|
28217 |
successfully saved checkpoint at iteration 75000 to /gpfsscratch/rech/six/commun/synched_exps/tr4c-1B3-rotary-oscar/checkpoints
|
28218 |
time (ms) | save-checkpoint: 1703.03
|
|
|
|
|
|
|
|
|
|
|
|
|
|
28216 |
[2021-10-02 15:42:38,338] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: /gpfsscratch/rech/six/commun/synched_exps/tr4c-1B3-rotary-oscar/checkpoints/global_step75000/mp_rank_00_model_states.pt
|
28217 |
successfully saved checkpoint at iteration 75000 to /gpfsscratch/rech/six/commun/synched_exps/tr4c-1B3-rotary-oscar/checkpoints
|
28218 |
time (ms) | save-checkpoint: 1703.03
|
28219 |
+
iteration 75200/ 152972 | consumed samples: 33422784 | elapsed time per iteration (ms): 7684.7 | learning rate: 1.187E-04 | global batch size: 512 | lm loss: 2.852895E+00 | loss scale: 524288.0 | grad norm: 55204.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
28220 |
+
time (ms)
|
28221 |
+
iteration 75400/ 152972 | consumed samples: 33525184 | elapsed time per iteration (ms): 6753.5 | learning rate: 1.183E-04 | global batch size: 512 | lm loss: 2.844674E+00 | loss scale: 524288.0 | grad norm: 49166.981 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
28222 |
+
time (ms)
|
28223 |
+
iteration 75600/ 152972 | consumed samples: 33627584 | elapsed time per iteration (ms): 6782.4 | learning rate: 1.179E-04 | global batch size: 512 | lm loss: 2.847534E+00 | loss scale: 262144.0 | grad norm: 27896.990 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
|
28224 |
+
time (ms)
|