new data

logs/main_log.txt

@@ -32390,3 +32390,9 @@ saving checkpoint at iteration   87000 to /gpfsscratch/rech/six/commun/synched_e
 [2021-10-03 12:57:06,344] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: /gpfsscratch/rech/six/commun/synched_exps/tr4c-1B3-rotary-oscar/checkpoints/global_step87000/mp_rank_00_model_states.pt
   successfully saved checkpoint at iteration   87000 to /gpfsscratch/rech/six/commun/synched_exps/tr4c-1B3-rotary-oscar/checkpoints
 time (ms) | save-checkpoint: 1584.04
+ iteration    87200/  152972 | consumed samples:     39566784 | elapsed time per iteration (ms): 6950.1 | learning rate: 9.373E-05 | global batch size:   512 | lm loss: 2.822462E+00 | loss scale: 262144.0 | grad norm: 26305.608 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    87400/  152972 | consumed samples:     39669184 | elapsed time per iteration (ms): 6096.0 | learning rate: 9.331E-05 | global batch size:   512 | lm loss: 2.821524E+00 | loss scale: 524288.0 | grad norm: 53502.986 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+ iteration    87600/  152972 | consumed samples:     39771584 | elapsed time per iteration (ms): 6108.5 | learning rate: 9.290E-05 | global batch size:   512 | lm loss: 2.823395E+00 | loss scale: 524288.0 | grad norm: 53236.914 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)