new data

logs/main_log.txt

@@ -28222,3 +28222,12 @@ time (ms)
 time (ms)
  iteration    75600/  152972 | consumed samples:     33627584 | elapsed time per iteration (ms): 6782.4 | learning rate: 1.179E-04 | global batch size:   512 | lm loss: 2.847534E+00 | loss scale: 262144.0 | grad norm: 27896.990 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
 time (ms)
+ iteration    75800/  152972 | consumed samples:     33729984 | elapsed time per iteration (ms): 6779.8 | learning rate: 1.175E-04 | global batch size:   512 | lm loss: 2.845177E+00 | loss scale: 262144.0 | grad norm: 25938.193 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+[2021-10-02 17:35:29,446] [INFO] [logging.py:68:log_dist] [Rank 0] step=76000, skipped=163, lr=[0.00011703754771760277, 0.00011703754771760277], mom=[(0.9, 0.999), (0.9, 0.999)]
+ iteration    76000/  152972 | consumed samples:     33832384 | elapsed time per iteration (ms): 6758.6 | learning rate: 1.170E-04 | global batch size:   512 | lm loss: 2.846029E+00 | loss scale: 262144.0 | grad norm: 25893.255 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+time (ms)
+steps: 76000 loss: 2.8728 iter time (s): 0.003 samples/sec: 153587.986
+-------------------------------------------------------------------------------------------------
+ validation loss at iteration 76000 | lm loss value: 2.792851E+00 | lm loss PPL: 1.632750E+01 | 
+-------------------------------------------------------------------------------------------------