new data

logs/main_log.txt

@@ -13603,3 +13603,89 @@ time (ms) | save-checkpoint: 1132.57
 [2021-11-04 21:50:08,652] [INFO] [logging.py:68:log_dist] [Rank 0] step=40000, skipped=81, lr=[0.0001804599959837998, 0.0001804599959837998], mom=[(0.9, 0.999), (0.9, 0.999)]
  iteration    40000/  152972 | consumed samples:     15400384 | consumed tokens:  31539986432 | elapsed time per iteration (ms): 6080.1 | learning rate: 1.805E-04 | global batch size:   512 | lm loss: 2.046368E+00 | loss scale: 1048576.0 | grad norm: 79641.172 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
 steps: 40000 loss: 1.9234 iter time (s): 0.003 samples/sec: 168896.024
+-------------------------------------------------------------------------------------------------
+ validation loss at iteration 40000 | lm loss value: 2.018202E+00 | lm loss PPL: 7.524780E+00 | 
+-------------------------------------------------------------------------------------------------
+ iteration    40200/  152972 | consumed samples:     15502784 | consumed tokens:  31749701632 | elapsed time per iteration (ms): 7223.0 | learning rate: 1.802E-04 | global batch size:   512 | lm loss: 2.039917E+00 | loss scale: 1048576.0 | grad norm: 82712.329 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 |
+srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
+Killing subprocess 122540
+Killing subprocess 1284095
+Killing subprocess 908359
+Killing subprocess 252675
+Killing subprocess 1964291
+Killing subprocess 502625
+Killing subprocess 2721736
+Killing subprocess 122541
+Killing subprocess 803909
+Killing subprocess 1284096
+Killing subprocess 1964292
+Killing subprocess 908360
+Killing subprocess 2133881
+Killing subprocess 532510
+Killing subprocess 252676
+Killing subprocess 2791463
+Killing subprocess 1284097
+Killing subprocess 122542
+Killing subprocess 908361
+Killing subprocess 502626
+Killing subprocess 1964293
+Killing subprocess 803910
+Killing subprocess 2721737
+Killing subprocess 1288896
+Killing subprocess 252677
+Killing subprocess 122543
+Killing subprocess 1284099
+Killing subprocess 502627
+slurmstepd: error: *** STEP 1825190.0 ON r6i3n0 CANCELLED AT 2021-11-04T22:24:00 ***
+Killing subprocess 908362
+Killing subprocess 532511
+Killing subprocess 2721738
+Killing subprocess 2133882
+Killing subprocess 1964294
+Main process received SIGTERM, exiting
+Killing subprocess 2791464
+Killing subprocess 532512
+Killing subprocess 2133883
+Killing subprocess 502629
+Killing subprocess 803911
+Killing subprocess 803912
+Killing subprocess 532513
+Killing subprocess 2791465
+Main process received SIGTERM, exiting
+Killing subprocess 1288897
+Killing subprocess 2133884
+Killing subprocess 252678
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 2721739
+Main process received SIGTERM, exiting
+Killing subprocess 2791466
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 1288898
+Killing subprocess 1288899
+Main process received SIGTERM, exiting
+Killing subprocess 980734
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 980735
+Killing subprocess 980736
+Main process received SIGTERM, exiting
+Killing subprocess 980737
+Main process received SIGTERM, exiting
+Killing subprocess 429407
+Killing subprocess 2991515
+Killing subprocess 179727
+Killing subprocess 429408
+Killing subprocess 2991516
+Killing subprocess 179728
+Killing subprocess 429409
+Killing subprocess 429411
+Killing subprocess 2991517
+Killing subprocess 2991518
+Main process received SIGTERM, exiting
+Main process received SIGTERM, exiting
+Killing subprocess 179729
+Killing subprocess 179730
+Main process received SIGTERM, exiting