NeoXArgs.configure_distributed_args() using world size: 64 and model-parallel size: 1 > building HFTokenizer tokenizer ... > padded vocab (size: 50277) with 27 dummy tokens (new size: 50304) > setting tensorboard ... > initializing torch distributed ... > initializing model parallel with size 1 MPU DP: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63] MPU PP: [0] MPU PP: [1] MPU PP: [2] MPU PP: [3] MPU PP: [4] MPU PP: [5] MPU PP: [6] MPU PP: [7] MPU PP: [8] MPU PP: [9] MPU PP: [10] MPU PP: [11] MPU PP: [12] MPU PP: [13] MPU PP: [14] MPU PP: [15] MPU PP: [16] MPU PP: [17] MPU PP: [18] MPU PP: [19] MPU PP: [20] MPU PP: [21] MPU PP: [22] MPU PP: [23] MPU PP: [24] MPU PP: [25] MPU PP: [26] MPU PP: [27] MPU PP: [28] MPU PP: [29] MPU PP: [30] MPU PP: [31] MPU PP: [32] MPU PP: [33] MPU PP: [34] MPU PP: [35] MPU PP: [36] MPU PP: [37] MPU PP: [38] MPU PP: [39] MPU PP: [40] MPU PP: [41] MPU PP: [42] MPU PP: [43] MPU PP: [44] MPU PP: [45] MPU PP: [46] MPU PP: [47] MPU PP: [48] MPU PP: [49] MPU PP: [50] MPU PP: [51] MPU PP: [52] MPU PP: [53] MPU PP: [54] MPU PP: [55] MPU PP: [56] MPU PP: [57] MPU PP: [58] MPU PP: [59] MPU PP: [60] MPU PP: [61] MPU PP: [62] MPU PP: [63] MPU MP: [0] MPU MP: [1] MPU MP: [2] MPU MP: [3] MPU MP: [4] MPU MP: [5] MPU MP: [6] MPU MP: [7] MPU MP: [8] MPU MP: [9] MPU MP: [10] MPU MP: [11] MPU MP: [12] MPU MP: [13] MPU MP: [14] MPU MP: [15] MPU MP: [16] MPU MP: [17] MPU MP: [18] MPU MP: [19] MPU MP: [20] MPU MP: [21] MPU MP: [22] MPU MP: [23] MPU MP: [24] MPU MP: [25] MPU MP: [26] MPU MP: [27] MPU MP: [28] MPU MP: [29] MPU MP: [30] MPU MP: [31] MPU MP: [32] MPU MP: [33] MPU MP: [34] MPU MP: [35] MPU MP: [36] MPU MP: [37] MPU MP: [38] MPU MP: [39] MPU MP: [40] MPU MP: [41] MPU MP: [42] MPU MP: [43] MPU MP: [44] MPU MP: [45] MPU MP: [46] MPU MP: [47] MPU MP: [48] MPU MP: [49] MPU MP: [50] MPU MP: [51] MPU MP: [52] MPU MP: [53] MPU MP: [54] MPU MP: [55] MPU MP: [56] MPU MP: [57] MPU MP: [58] MPU MP: [59] MPU MP: [60] MPU MP: [61] MPU MP: [62] MPU MP: [63] > setting random seeds to 1234 ... building GPT2 model ... SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pipe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=46, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63} stage=0 layers=37 0: EmbeddingPipe 1: _pre_transformer_block 2: ParallelFlownetLayerPipe 3: ParallelFlownetLayerPipe 4: ParallelFlownetLayerPipe 5: ParallelFlownetLayerPipe 6: ParallelFlownetLayerPipe 7: ParallelFlownetLayerPipe 8: ParallelFlownetLayerPipe 9: ParallelFlownetLayerPipe 10: ParallelFlownetLayerPipe 11: ParallelFlownetLayerPipe 12: ParallelFlownetLayerPipe 13: ParallelFlownetLayerPipe 14: ParallelFlownetLayerPipe 15: ParallelFlownetLayerPipe 16: ParallelFlownetLayerPipe 17: ParallelFlownetLayerPipe 18: ParallelFlownetLayerPipe 19: ParallelFlownetLayerPipe 20: ParallelFlownetLayerPipe 21: ParallelFlownetLayerPipe 22: ParallelFlownetLayerPipe 23: ParallelFlownetLayerPipe 24: ParallelFlownetLayerPipe 25: ParallelFlownetLayerPipe 26: ParallelFlownetLayerPipe 27: ParallelFlownetLayerPipe 28: ParallelFlownetLayerPipe 29: ParallelFlownetLayerPipe 30: ParallelFlownetLayerPipe 31: ParallelFlownetLayerPipe 32: ParallelFlownetLayerPipe 33: ParallelFlownetLayerPipe 34: _post_transformer_block 35: NormPipe 36: EmbeddingPipe loss: partial Configuring Optimizer type: Adam with params: {'lr': 0.0006, 'betas': [0.9, 0.95], 'eps': 1e-08} > learning rate decay style: cosine DeepSpeed is enabled. > number of parameters on model parallel rank 0: 903252480 > total params: 903,252,480 Unable to load checkpoint. Loading checkpoint and starting from iteration 0 > building train, validation, and test datasets ... reading sizes... reading pointers... reading document index... creating numpy buffer of mmap... creating memory view of numpy buffer... > dataset split: train: document indices in [0, 130154259) total of 130154259 documents validation: document indices in [130154259, 134183803) total of 4029544 documents test: document indices in [134183803, 134318121) total of 134318 documents > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_doc_idx.npy > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_sample_idx.npy > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_train_indexmap_146432000ns_2048sl_1234s_shuffle_idx.npy loaded indexed file in 0.317 seconds total number of samples: 195915016 total number of epochs: 2 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_doc_idx.npy > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_sample_idx.npy > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_valid_indexmap_2969600ns_2048sl_1234s_shuffle_idx.npy loaded indexed file in 0.112 seconds total number of samples: 3097460 total number of epochs: 1 > loading doc-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_doc_idx.npy > loading sample-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_sample_idx.npy > loading shuffle-idx mapping from /u/wangh/workspace/dataset/language_dataset/pile/pile_0.87_deduped_text_document_test_indexmap_102400ns_2048sl_1234s_shuffle_idx.npy loaded indexed file in 0.079 seconds total number of samples: 102462 total number of epochs: 1 setting training data start iteration to 0 setting validation data start iteration to 0 done with setups ... time (ms) | model and optimizer: 3443.03 | train/valid/test data iterators: 5438.07 training ... WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step0 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4 samples/sec: 346.770 | iteration 500/ 143000 | elapsed time per iteration (ms): 2953.0 | learning rate: 2.098E-04 | approx flops per GPU: 75.0TFLOPS | lm_loss: 6.376106E+00 | loss scale: 4096.0 | number of skipped iterations: 0 | number of nan iterations: 0 | after 500 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11453.5703125 | reserved: 13716.0 | max reserved: 13716.0 time (ms) | forward: 616.85 | backward: 2216.34 | backward-backward: 2216.27 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 3.13 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16 samples/sec: 352.134 | iteration 1000/ 143000 | elapsed time per iteration (ms): 2908.0 | learning rate: 4.196E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 3.774834E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.82 | backward: 2207.45 | backward-backward: 2207.38 | backward-allreduce: 0.00 | optimizer: 79.73 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32 samples/sec: 352.377 | iteration 1500/ 143000 | elapsed time per iteration (ms): 2906.0 | learning rate: 6.000E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 3.132075E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.61 | backward: 2210.01 | backward-backward: 2209.94 | backward-allreduce: 0.00 | optimizer: 79.72 | batch generator: 2.42 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64 samples/sec: 352.701 | iteration 2000/ 143000 | elapsed time per iteration (ms): 2903.3 | learning rate: 6.000E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.867739E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.86 | backward: 2206.59 | backward-backward: 2206.51 | backward-allreduce: 0.00 | optimizer: 79.71 | batch generator: 2.43 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128 samples/sec: 352.790 | iteration 2500/ 143000 | elapsed time per iteration (ms): 2902.6 | learning rate: 5.999E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.722186E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.26 | backward: 2206.18 | backward-backward: 2206.11 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.37 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step256 samples/sec: 349.630 | iteration 3000/ 143000 | elapsed time per iteration (ms): 2928.8 | learning rate: 5.998E-04 | approx flops per GPU: 75.7TFLOPS | lm_loss: 2.633660E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.85 | backward: 2233.73 | backward-backward: 2233.65 | backward-allreduce: 0.00 | optimizer: 79.63 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step500 samples/sec: 352.908 | iteration 3500/ 143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 5.997E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.576675E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.49 | backward: 2206.91 | backward-backward: 2206.84 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step512 samples/sec: 352.294 | iteration 4000/ 143000 | elapsed time per iteration (ms): 2906.7 | learning rate: 5.996E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.532798E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.75 | backward: 2211.48 | backward-backward: 2211.41 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1000 samples/sec: 353.086 | iteration 4500/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.994E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.499583E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.59 | backward: 2205.48 | backward-backward: 2205.41 | backward-allreduce: 0.00 | optimizer: 79.66 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step1500 samples/sec: 353.079 | iteration 5000/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 5.992E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.472691E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.12 | backward: 2205.31 | backward-backward: 2205.24 | backward-allreduce: 0.00 | optimizer: 79.31 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2000 --------------------------------------------------------------------------------------------------------- validation results at iteration 5000 | lm_loss value: 2.411079E+00 | lm_loss_ppl value: 1.114599E+01 | --------------------------------------------------------------------------------------------------------- samples/sec: 338.121 | iteration 5500/ 143000 | elapsed time per iteration (ms): 3028.5 | learning rate: 5.989E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.450628E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.69 | backward: 2204.89 | backward-backward: 2204.82 | backward-allreduce: 0.00 | optimizer: 79.71 | batch generator: 2.74 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step2500 samples/sec: 352.386 | iteration 6000/ 143000 | elapsed time per iteration (ms): 2905.9 | learning rate: 5.986E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.431382E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.51 | backward: 2211.14 | backward-backward: 2211.07 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step3000 samples/sec: 352.866 | iteration 6500/ 143000 | elapsed time per iteration (ms): 2902.0 | learning rate: 5.983E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.413347E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.32 | backward: 2207.21 | backward-backward: 2207.13 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step3500 samples/sec: 353.044 | iteration 7000/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 5.979E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.399783E+00 | loss scale: 65536.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.57 | backward: 2206.24 | backward-backward: 2206.17 | backward-allreduce: 0.00 | optimizer: 79.28 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4000 samples/sec: 353.070 | iteration 7500/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 5.976E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.388037E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.42 | backward: 2205.85 | backward-backward: 2205.78 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step4500 samples/sec: 350.078 | iteration 8000/ 143000 | elapsed time per iteration (ms): 2925.1 | learning rate: 5.971E-04 | approx flops per GPU: 75.7TFLOPS | lm_loss: 2.376209E+00 | loss scale: 131072.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.41 | backward: 2229.35 | backward-backward: 2229.28 | backward-allreduce: 0.00 | optimizer: 79.72 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step5000 samples/sec: 353.087 | iteration 8500/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.967E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.366794E+00 | loss scale: 65536.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.16 | backward: 2204.86 | backward-backward: 2204.79 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step5500 samples/sec: 352.920 | iteration 9000/ 143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.962E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.356318E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.31 | backward: 2205.70 | backward-backward: 2205.62 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.37 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step6000 samples/sec: 353.012 | iteration 9500/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 5.957E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.347873E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.18 | backward: 2205.41 | backward-backward: 2205.33 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.42 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step6500 samples/sec: 351.834 | iteration 10000/ 143000 | elapsed time per iteration (ms): 2910.5 | learning rate: 5.951E-04 | approx flops per GPU: 76.1TFLOPS | lm_loss: 2.339790E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.60 | backward: 2214.82 | backward-backward: 2214.74 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.62 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step7000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 10000 | lm_loss value: 2.289003E+00 | lm_loss_ppl value: 9.865094E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.135 | iteration 10500/ 143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 5.946E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.333742E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.31 | backward: 2205.22 | backward-backward: 2205.15 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.78 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step7500 samples/sec: 352.921 | iteration 11000/ 143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.939E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.327170E+00 | loss scale: 131072.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.67 | backward: 2205.55 | backward-backward: 2205.48 | backward-allreduce: 0.00 | optimizer: 79.70 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8000 samples/sec: 353.044 | iteration 11500/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 5.933E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.319703E+00 | loss scale: 32768.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 610.05 | backward: 2205.54 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.16 | batch generator: 2.37 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step8500 samples/sec: 352.928 | iteration 12000/ 143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 5.926E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.316716E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.27 | backward: 2206.04 | backward-backward: 2205.97 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.37 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step9000 samples/sec: 352.587 | iteration 12500/ 143000 | elapsed time per iteration (ms): 2904.2 | learning rate: 5.919E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.309714E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.70 | backward: 2209.49 | backward-backward: 2209.42 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step9500 samples/sec: 352.916 | iteration 13000/ 143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.912E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.304286E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.50 | backward: 2205.78 | backward-backward: 2205.71 | backward-allreduce: 0.00 | optimizer: 79.33 | batch generator: 2.41 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step10000 samples/sec: 352.269 | iteration 13500/ 143000 | elapsed time per iteration (ms): 2906.9 | learning rate: 5.904E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.299688E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.55 | backward: 2211.04 | backward-backward: 2210.97 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.43 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step10500 samples/sec: 352.845 | iteration 14000/ 143000 | elapsed time per iteration (ms): 2902.1 | learning rate: 5.896E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.296754E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.46 | backward: 2206.06 | backward-backward: 2205.98 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.38 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step11000 samples/sec: 352.970 | iteration 14500/ 143000 | elapsed time per iteration (ms): 2901.1 | learning rate: 5.887E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.291494E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.69 | backward: 2206.35 | backward-backward: 2206.28 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step11500 samples/sec: 352.892 | iteration 15000/ 143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 5.879E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.288373E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.95 | backward: 2206.62 | backward-backward: 2206.54 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step12000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 15000 | lm_loss value: 2.235826E+00 | lm_loss_ppl value: 9.354208E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.137 | iteration 15500/ 143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 5.870E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.282740E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.83 | backward: 2205.90 | backward-backward: 2205.83 | backward-allreduce: 0.00 | optimizer: 79.63 | batch generator: 2.80 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step12500 samples/sec: 352.894 | iteration 16000/ 143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 5.860E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.279487E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.46 | backward: 2206.08 | backward-backward: 2206.01 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.42 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step13000 samples/sec: 352.987 | iteration 16500/ 143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 5.851E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.277207E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.07 | backward: 2205.59 | backward-backward: 2205.52 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step13500 samples/sec: 352.909 | iteration 17000/ 143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 5.841E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.272039E+00 | loss scale: 131072.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.39 | backward: 2206.05 | backward-backward: 2205.98 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step14000 samples/sec: 353.060 | iteration 17500/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.830E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.269935E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.54 | backward: 2204.93 | backward-backward: 2204.86 | backward-allreduce: 0.00 | optimizer: 79.30 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step14500 samples/sec: 352.898 | iteration 18000/ 143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 5.820E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.265185E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.88 | backward: 2206.55 | backward-backward: 2206.48 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step15000 samples/sec: 353.008 | iteration 18500/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 5.809E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.264649E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.41 | backward: 2205.34 | backward-backward: 2205.27 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step15500 samples/sec: 352.880 | iteration 19000/ 143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 5.798E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.260992E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.88 | backward: 2206.13 | backward-backward: 2206.06 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16000 samples/sec: 353.063 | iteration 19500/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 5.786E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.258127E+00 | loss scale: 65536.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.66 | backward: 2205.94 | backward-backward: 2205.87 | backward-allreduce: 0.00 | optimizer: 79.32 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step16500 samples/sec: 352.917 | iteration 20000/ 143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.774E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.254986E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.85 | backward: 2206.64 | backward-backward: 2206.57 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step17000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 20000 | lm_loss value: 2.205550E+00 | lm_loss_ppl value: 9.075239E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 335.977 | iteration 20500/ 143000 | elapsed time per iteration (ms): 3047.8 | learning rate: 5.762E-04 | approx flops per GPU: 72.7TFLOPS | lm_loss: 2.253958E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.77 | backward: 2205.69 | backward-backward: 2205.62 | backward-allreduce: 0.00 | optimizer: 79.78 | batch generator: 6.67 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step17500 samples/sec: 352.881 | iteration 21000/ 143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 5.750E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.249801E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.97 | backward: 2206.61 | backward-backward: 2206.53 | backward-allreduce: 0.00 | optimizer: 79.76 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step18000 samples/sec: 351.926 | iteration 21500/ 143000 | elapsed time per iteration (ms): 2909.7 | learning rate: 5.737E-04 | approx flops per GPU: 76.1TFLOPS | lm_loss: 2.249513E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 617.36 | backward: 2206.78 | backward-backward: 2206.71 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 9.92 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step18500 samples/sec: 352.934 | iteration 22000/ 143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 5.724E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.247319E+00 | loss scale: 131072.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.90 | backward: 2206.51 | backward-backward: 2206.44 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step19000 samples/sec: 353.014 | iteration 22500/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 5.711E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.245720E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.74 | backward: 2205.88 | backward-backward: 2205.81 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step19500 samples/sec: 352.890 | iteration 23000/ 143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 5.697E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.243116E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.26 | backward: 2206.25 | backward-backward: 2206.17 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.38 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step20000 samples/sec: 352.974 | iteration 23500/ 143000 | elapsed time per iteration (ms): 2901.1 | learning rate: 5.683E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.241707E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.76 | backward: 2206.25 | backward-backward: 2206.18 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step20500 samples/sec: 352.923 | iteration 24000/ 143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 5.669E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.242003E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.99 | backward: 2206.31 | backward-backward: 2206.24 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step21000 samples/sec: 352.905 | iteration 24500/ 143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 5.655E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.237709E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.82 | backward: 2206.67 | backward-backward: 2206.60 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.38 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step21500 samples/sec: 352.945 | iteration 25000/ 143000 | elapsed time per iteration (ms): 2901.3 | learning rate: 5.640E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.236009E+00 | loss scale: 65536.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.49 | backward: 2205.98 | backward-backward: 2205.91 | backward-allreduce: 0.00 | optimizer: 79.34 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step22000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 25000 | lm_loss value: 2.192039E+00 | lm_loss_ppl value: 8.953446E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 337.969 | iteration 25500/ 143000 | elapsed time per iteration (ms): 3029.9 | learning rate: 5.625E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.234498E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.67 | backward: 2206.27 | backward-backward: 2206.20 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 2.79 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step22500 samples/sec: 352.955 | iteration 26000/ 143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 5.609E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.233741E+00 | loss scale: 131072.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.84 | backward: 2205.22 | backward-backward: 2205.14 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step23000 samples/sec: 352.802 | iteration 26500/ 143000 | elapsed time per iteration (ms): 2902.5 | learning rate: 5.594E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.230550E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.58 | backward: 2206.58 | backward-backward: 2206.51 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step23500 samples/sec: 352.196 | iteration 27000/ 143000 | elapsed time per iteration (ms): 2907.5 | learning rate: 5.578E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.229256E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.68 | backward: 2211.71 | backward-backward: 2211.64 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step24000 samples/sec: 352.787 | iteration 27500/ 143000 | elapsed time per iteration (ms): 2902.6 | learning rate: 5.562E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.227592E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.56 | backward: 2206.68 | backward-backward: 2206.61 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step24500 samples/sec: 352.828 | iteration 28000/ 143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 5.545E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.226016E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.37 | backward: 2206.40 | backward-backward: 2206.33 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step25000 samples/sec: 352.809 | iteration 28500/ 143000 | elapsed time per iteration (ms): 2902.4 | learning rate: 5.528E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.224761E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.65 | backward: 2206.90 | backward-backward: 2206.83 | backward-allreduce: 0.00 | optimizer: 79.42 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step25500 samples/sec: 352.819 | iteration 29000/ 143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 5.511E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.222556E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.84 | backward: 2206.07 | backward-backward: 2206.00 | backward-allreduce: 0.00 | optimizer: 79.79 | batch generator: 2.41 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step26000 samples/sec: 352.801 | iteration 29500/ 143000 | elapsed time per iteration (ms): 2902.5 | learning rate: 5.494E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.222526E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.60 | backward: 2206.68 | backward-backward: 2206.61 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.35 samples/sec: 352.290 | iteration 30000/ 143000 | elapsed time per iteration (ms): 2906.7 | learning rate: 5.476E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.219844E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | after 30000 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11451.5068359375 | reserved: 13710.0 | max reserved: 13710.0 time (ms) | forward: 617.72 | backward: 2207.65 | backward-backward: 2207.58 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 5.17 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step27000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 30000 | lm_loss value: 2.171570E+00 | lm_loss_ppl value: 8.772044E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 337.953 | iteration 30500/ 143000 | elapsed time per iteration (ms): 3030.0 | learning rate: 5.459E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.218577E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.60 | backward: 2205.52 | backward-backward: 2205.44 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 2.65 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step27500 samples/sec: 353.089 | iteration 31000/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.440E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.218100E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.25 | backward: 2205.63 | backward-backward: 2205.56 | backward-allreduce: 0.00 | optimizer: 79.80 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step28000 samples/sec: 353.054 | iteration 31500/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.422E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.214409E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.42 | backward: 2205.83 | backward-backward: 2205.76 | backward-allreduce: 0.00 | optimizer: 79.81 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step28500 samples/sec: 353.107 | iteration 32000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 5.403E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.214736E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.32 | backward: 2205.26 | backward-backward: 2205.19 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step29000 samples/sec: 353.060 | iteration 32500/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.385E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.213073E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.50 | backward: 2205.52 | backward-backward: 2205.45 | backward-allreduce: 0.00 | optimizer: 79.97 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step29500 samples/sec: 353.124 | iteration 33000/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 5.365E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.213183E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.34 | backward: 2205.10 | backward-backward: 2205.03 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step30000 samples/sec: 353.007 | iteration 33500/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 5.346E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.210692E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.95 | backward: 2205.39 | backward-backward: 2205.31 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step30500 samples/sec: 353.089 | iteration 34000/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 5.326E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.211081E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.58 | backward: 2205.18 | backward-backward: 2205.11 | backward-allreduce: 0.00 | optimizer: 79.84 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step31000 samples/sec: 353.024 | iteration 34500/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 5.306E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.208467E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.54 | backward: 2205.59 | backward-backward: 2205.51 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step31500 samples/sec: 353.171 | iteration 35000/ 143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 5.286E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.206911E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.37 | backward: 2204.80 | backward-backward: 2204.73 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 35000 | lm_loss value: 2.161903E+00 | lm_loss_ppl value: 8.687654E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.232 | iteration 35500/ 143000 | elapsed time per iteration (ms): 3027.5 | learning rate: 5.266E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.205729E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.79 | backward: 2204.63 | backward-backward: 2204.55 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.69 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step32500 samples/sec: 353.216 | iteration 36000/ 143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 5.245E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.204554E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.47 | backward: 2204.01 | backward-backward: 2203.94 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step33000 samples/sec: 353.212 | iteration 36500/ 143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 5.224E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.202843E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.73 | backward: 2204.00 | backward-backward: 2203.92 | backward-allreduce: 0.00 | optimizer: 79.94 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step33500 samples/sec: 353.282 | iteration 37000/ 143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 5.203E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.202124E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.55 | backward: 2203.61 | backward-backward: 2203.53 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step34000 samples/sec: 352.516 | iteration 37500/ 143000 | elapsed time per iteration (ms): 2904.8 | learning rate: 5.182E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.201600E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.81 | backward: 2209.54 | backward-backward: 2209.47 | backward-allreduce: 0.00 | optimizer: 79.92 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step34500 samples/sec: 353.205 | iteration 38000/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 5.160E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.201355E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.55 | backward: 2204.24 | backward-backward: 2204.17 | backward-allreduce: 0.00 | optimizer: 79.80 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step35000 samples/sec: 353.133 | iteration 38500/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 5.139E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.200124E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.78 | backward: 2204.53 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 79.96 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step35500 samples/sec: 353.179 | iteration 39000/ 143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 5.116E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.200712E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.59 | backward: 2204.03 | backward-backward: 2203.96 | backward-allreduce: 0.00 | optimizer: 80.06 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step36000 samples/sec: 353.148 | iteration 39500/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 5.094E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.197089E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.78 | backward: 2204.26 | backward-backward: 2204.18 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step36500 samples/sec: 353.290 | iteration 40000/ 143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 5.072E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.195795E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.56 | backward: 2203.77 | backward-backward: 2203.70 | backward-allreduce: 0.00 | optimizer: 79.66 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step37000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 40000 | lm_loss value: 2.145396E+00 | lm_loss_ppl value: 8.545420E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.263 | iteration 40500/ 143000 | elapsed time per iteration (ms): 3027.2 | learning rate: 5.049E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.194291E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.04 | backward: 2204.35 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.70 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step37500 samples/sec: 353.221 | iteration 41000/ 143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 5.026E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.194061E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.87 | backward: 2204.04 | backward-backward: 2203.96 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step38000 samples/sec: 353.054 | iteration 41500/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 5.003E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.193298E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.12 | backward: 2204.52 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step38500 samples/sec: 353.114 | iteration 42000/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 4.980E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.191050E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.98 | backward: 2204.12 | backward-backward: 2204.04 | backward-allreduce: 0.00 | optimizer: 80.02 | batch generator: 2.38 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step39000 samples/sec: 353.115 | iteration 42500/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 4.956E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.191622E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.17 | backward: 2204.08 | backward-backward: 2204.01 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step39500 samples/sec: 353.193 | iteration 43000/ 143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 4.932E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.189374E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.07 | backward: 2204.72 | backward-backward: 2204.65 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step40000 samples/sec: 353.062 | iteration 43500/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 4.909E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.189497E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.04 | backward: 2204.63 | backward-backward: 2204.56 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.44 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step40500 samples/sec: 353.130 | iteration 44000/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 4.884E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.186470E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.15 | backward: 2204.83 | backward-backward: 2204.76 | backward-allreduce: 0.00 | optimizer: 79.97 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step41000 samples/sec: 353.135 | iteration 44500/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 4.860E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.185729E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.37 | backward: 2204.91 | backward-backward: 2204.83 | backward-allreduce: 0.00 | optimizer: 79.83 | batch generator: 2.43 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step41500 samples/sec: 353.202 | iteration 45000/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 4.836E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.185482E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.06 | backward: 2204.52 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 79.95 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step42000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 45000 | lm_loss value: 2.138723E+00 | lm_loss_ppl value: 8.488594E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.317 | iteration 45500/ 143000 | elapsed time per iteration (ms): 3026.7 | learning rate: 4.811E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.183970E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.23 | backward: 2204.88 | backward-backward: 2204.81 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.69 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step42500 samples/sec: 353.201 | iteration 46000/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 4.786E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.182952E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.04 | backward: 2204.59 | backward-backward: 2204.52 | backward-allreduce: 0.00 | optimizer: 79.79 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step43000 samples/sec: 353.141 | iteration 46500/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 4.761E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.181557E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.25 | backward: 2204.76 | backward-backward: 2204.69 | backward-allreduce: 0.00 | optimizer: 79.97 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step43500 samples/sec: 353.227 | iteration 47000/ 143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 4.736E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.180339E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.04 | backward: 2204.50 | backward-backward: 2204.42 | backward-allreduce: 0.00 | optimizer: 79.94 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step44000 samples/sec: 353.206 | iteration 47500/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 4.710E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.180814E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.59 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step44500 samples/sec: 353.223 | iteration 48000/ 143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 4.685E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.179525E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.96 | backward: 2203.51 | backward-backward: 2203.44 | backward-allreduce: 0.00 | optimizer: 79.87 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step45000 samples/sec: 353.153 | iteration 48500/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 4.659E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.178233E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.21 | backward: 2203.92 | backward-backward: 2203.85 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step45500 samples/sec: 353.266 | iteration 49000/ 143000 | elapsed time per iteration (ms): 2898.7 | learning rate: 4.633E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.178110E+00 | loss scale: 16384.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 610.00 | backward: 2203.66 | backward-backward: 2203.59 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step46000 samples/sec: 353.088 | iteration 49500/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 4.607E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.177734E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.21 | backward: 2204.44 | backward-backward: 2204.36 | backward-allreduce: 0.00 | optimizer: 79.98 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step46500 samples/sec: 353.138 | iteration 50000/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 4.581E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.174503E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.04 | backward: 2203.97 | backward-backward: 2203.90 | backward-allreduce: 0.00 | optimizer: 80.00 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step47000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 50000 | lm_loss value: 2.130526E+00 | lm_loss_ppl value: 8.419295E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.257 | iteration 50500/ 143000 | elapsed time per iteration (ms): 3027.3 | learning rate: 4.554E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.175228E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.28 | backward: 2203.95 | backward-backward: 2203.88 | backward-allreduce: 0.00 | optimizer: 80.03 | batch generator: 2.77 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step47500 samples/sec: 353.186 | iteration 51000/ 143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 4.527E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.173995E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.98 | backward: 2203.62 | backward-backward: 2203.55 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step48000 samples/sec: 352.949 | iteration 51500/ 143000 | elapsed time per iteration (ms): 2901.3 | learning rate: 4.501E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.171973E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.30 | backward: 2204.71 | backward-backward: 2204.64 | backward-allreduce: 0.00 | optimizer: 80.28 | batch generator: 2.45 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step48500 samples/sec: 353.153 | iteration 52000/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 4.474E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.171407E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.09 | backward: 2204.00 | backward-backward: 2203.93 | backward-allreduce: 0.00 | optimizer: 79.89 | batch generator: 2.42 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step49000 samples/sec: 353.082 | iteration 52500/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 4.447E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.173119E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.28 | backward: 2204.49 | backward-backward: 2204.41 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.41 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step49500 samples/sec: 353.104 | iteration 53000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 4.420E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.170325E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.08 | backward: 2204.37 | backward-backward: 2204.30 | backward-allreduce: 0.00 | optimizer: 79.98 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step50000 samples/sec: 353.031 | iteration 53500/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 4.392E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.169852E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.22 | backward: 2204.69 | backward-backward: 2204.62 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step50500 samples/sec: 353.111 | iteration 54000/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 4.365E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.168136E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.97 | backward: 2204.53 | backward-backward: 2204.46 | backward-allreduce: 0.00 | optimizer: 79.87 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step51000 samples/sec: 353.074 | iteration 54500/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 4.337E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.167851E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.24 | backward: 2204.42 | backward-backward: 2204.34 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step51500 samples/sec: 353.128 | iteration 55000/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 4.310E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.166386E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.86 | backward: 2204.40 | backward-backward: 2204.33 | backward-allreduce: 0.00 | optimizer: 79.85 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step52000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 55000 | lm_loss value: 2.120372E+00 | lm_loss_ppl value: 8.334240E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.261 | iteration 55500/ 143000 | elapsed time per iteration (ms): 3027.2 | learning rate: 4.282E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.166138E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.92 | backward: 2204.37 | backward-backward: 2204.29 | backward-allreduce: 0.00 | optimizer: 80.07 | batch generator: 2.70 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step52500 samples/sec: 353.047 | iteration 56000/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 4.254E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.164145E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.80 | backward: 2205.04 | backward-backward: 2204.96 | backward-allreduce: 0.00 | optimizer: 79.90 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step53000 samples/sec: 352.928 | iteration 56500/ 143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 4.226E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.163596E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.90 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.86 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step53500 samples/sec: 353.058 | iteration 57000/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 4.198E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.162641E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.08 | backward: 2205.96 | backward-backward: 2205.88 | backward-allreduce: 0.00 | optimizer: 79.88 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step54000 samples/sec: 353.075 | iteration 57500/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 4.170E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.162332E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.29 | backward: 2205.46 | backward-backward: 2205.39 | backward-allreduce: 0.00 | optimizer: 80.01 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step54500 samples/sec: 353.009 | iteration 58000/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 4.141E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.162577E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.11 | backward: 2206.19 | backward-backward: 2206.12 | backward-allreduce: 0.00 | optimizer: 79.84 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step55000 samples/sec: 353.009 | iteration 58500/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 4.113E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.157890E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.29 | backward: 2205.56 | backward-backward: 2205.49 | backward-allreduce: 0.00 | optimizer: 79.99 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step55500 samples/sec: 353.007 | iteration 59000/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 4.084E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.160383E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.06 | backward: 2206.43 | backward-backward: 2206.36 | backward-allreduce: 0.00 | optimizer: 79.82 | batch generator: 2.24 samples/sec: 352.461 | iteration 59500/ 143000 | elapsed time per iteration (ms): 2905.3 | learning rate: 4.055E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.158436E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | after 59500 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11451.5068359375 | reserved: 13710.0 | max reserved: 13710.0 time (ms) | forward: 612.28 | backward: 2212.08 | backward-backward: 2212.00 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.38 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step56500 samples/sec: 352.901 | iteration 60000/ 143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 4.027E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.158576E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.15 | backward: 2205.89 | backward-backward: 2205.81 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step57000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 60000 | lm_loss value: 2.111970E+00 | lm_loss_ppl value: 8.264506E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.134 | iteration 60500/ 143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 3.998E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.156245E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.96 | backward: 2205.88 | backward-backward: 2205.81 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.68 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step57500 samples/sec: 352.976 | iteration 61000/ 143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 3.969E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.155501E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.04 | backward: 2206.12 | backward-backward: 2206.05 | backward-allreduce: 0.00 | optimizer: 79.28 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step58000 samples/sec: 353.005 | iteration 61500/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 3.940E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.157141E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.63 | backward: 2206.19 | backward-backward: 2206.12 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step58500 samples/sec: 352.935 | iteration 62000/ 143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 3.911E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.151945E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.81 | backward: 2206.55 | backward-backward: 2206.48 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step59000 samples/sec: 353.034 | iteration 62500/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 3.881E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.153375E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.26 | backward: 2206.32 | backward-backward: 2206.25 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step59500 samples/sec: 352.947 | iteration 63000/ 143000 | elapsed time per iteration (ms): 2901.3 | learning rate: 3.852E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.149092E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.42 | backward: 2206.85 | backward-backward: 2206.78 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step60000 samples/sec: 353.107 | iteration 63500/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.823E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.150221E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.62 | backward: 2205.37 | backward-backward: 2205.30 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step60500 samples/sec: 353.025 | iteration 64000/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 3.793E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.151737E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.86 | backward: 2205.68 | backward-backward: 2205.61 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step61000 samples/sec: 353.219 | iteration 64500/ 143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 3.764E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.147897E+00 | loss scale: 65536.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.77 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.26 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step61500 samples/sec: 353.069 | iteration 65000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 3.735E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.149497E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.94 | backward: 2205.06 | backward-backward: 2204.99 | backward-allreduce: 0.00 | optimizer: 79.48 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step62000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 65000 | lm_loss value: 2.102544E+00 | lm_loss_ppl value: 8.186968E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.324 | iteration 65500/ 143000 | elapsed time per iteration (ms): 3026.7 | learning rate: 3.705E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.147435E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.95 | backward: 2204.30 | backward-backward: 2204.23 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.73 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step62500 samples/sec: 353.106 | iteration 66000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.675E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.146142E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.14 | backward: 2204.75 | backward-backward: 2204.68 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step63000 samples/sec: 353.163 | iteration 66500/ 143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 3.646E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.153322E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.93 | backward: 2204.37 | backward-backward: 2204.30 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step63500 samples/sec: 353.119 | iteration 67000/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 3.616E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.146076E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.00 | backward: 2204.76 | backward-backward: 2204.69 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64000 samples/sec: 353.197 | iteration 67500/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 3.586E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.145408E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.70 | backward: 2204.53 | backward-backward: 2204.46 | backward-allreduce: 0.00 | optimizer: 79.41 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step64500 samples/sec: 353.109 | iteration 68000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.557E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.141154E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.22 | backward: 2204.85 | backward-backward: 2204.78 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step65000 samples/sec: 353.140 | iteration 68500/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 3.527E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.142211E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.89 | backward: 2204.71 | backward-backward: 2204.64 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step65500 samples/sec: 353.111 | iteration 69000/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 3.497E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.140988E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.09 | backward: 2205.02 | backward-backward: 2204.95 | backward-allreduce: 0.00 | optimizer: 79.26 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step66000 samples/sec: 353.136 | iteration 69500/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 3.467E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.141401E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.92 | backward: 2204.74 | backward-backward: 2204.67 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step66500 samples/sec: 353.052 | iteration 70000/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 3.437E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.142745E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.18 | backward: 2205.15 | backward-backward: 2205.08 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step67000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 70000 | lm_loss value: 2.095315E+00 | lm_loss_ppl value: 8.128005E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.207 | iteration 70500/ 143000 | elapsed time per iteration (ms): 3027.7 | learning rate: 3.407E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.141310E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.08 | backward: 2205.06 | backward-backward: 2204.99 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.76 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step67500 samples/sec: 352.648 | iteration 71000/ 143000 | elapsed time per iteration (ms): 2903.7 | learning rate: 3.377E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.139759E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.59 | backward: 2204.78 | backward-backward: 2204.71 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step68000 samples/sec: 344.420 | iteration 71500/ 143000 | elapsed time per iteration (ms): 2973.1 | learning rate: 3.348E-04 | approx flops per GPU: 74.5TFLOPS | lm_loss: 2.137903E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.05 | backward: 2204.68 | backward-backward: 2204.61 | backward-allreduce: 0.00 | optimizer: 79.25 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step68500 samples/sec: 353.074 | iteration 72000/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 3.318E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.141233E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.01 | backward: 2205.31 | backward-backward: 2205.23 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.38 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step69000 samples/sec: 353.094 | iteration 72500/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 3.288E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.136018E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.69 | backward: 2205.20 | backward-backward: 2205.13 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.37 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step69500 samples/sec: 353.016 | iteration 73000/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 3.258E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.136095E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.22 | backward: 2205.11 | backward-backward: 2205.04 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step70000 samples/sec: 353.134 | iteration 73500/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 3.228E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.136326E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.06 | backward: 2204.52 | backward-backward: 2204.45 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step70500 samples/sec: 353.105 | iteration 74000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.198E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.134349E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.22 | backward: 2204.81 | backward-backward: 2204.74 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.36 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step71000 samples/sec: 353.159 | iteration 74500/ 143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 3.168E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.133106E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.04 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step71500 samples/sec: 353.099 | iteration 75000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 3.138E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.132001E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.22 | backward: 2204.73 | backward-backward: 2204.66 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step72000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 75000 | lm_loss value: 2.086347E+00 | lm_loss_ppl value: 8.055432E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.216 | iteration 75500/ 143000 | elapsed time per iteration (ms): 3027.7 | learning rate: 3.108E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.129546E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.07 | backward: 2204.26 | backward-backward: 2204.19 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.74 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step72500 samples/sec: 353.084 | iteration 76000/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 3.078E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.130378E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.35 | backward: 2204.71 | backward-backward: 2204.64 | backward-allreduce: 0.00 | optimizer: 79.42 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step73000 samples/sec: 353.130 | iteration 76500/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 3.049E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.129836E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.09 | backward: 2204.74 | backward-backward: 2204.66 | backward-allreduce: 0.00 | optimizer: 79.44 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step73500 samples/sec: 353.016 | iteration 77000/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 3.019E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.129329E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.32 | backward: 2205.14 | backward-backward: 2205.07 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.38 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step74000 samples/sec: 353.189 | iteration 77500/ 143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 2.989E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.127165E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.12 | backward: 2204.10 | backward-backward: 2204.03 | backward-allreduce: 0.00 | optimizer: 79.45 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step74500 samples/sec: 352.115 | iteration 78000/ 143000 | elapsed time per iteration (ms): 2908.1 | learning rate: 2.959E-04 | approx flops per GPU: 76.2TFLOPS | lm_loss: 2.125618E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 612.20 | backward: 2210.20 | backward-backward: 2210.13 | backward-allreduce: 0.00 | optimizer: 79.98 | batch generator: 2.80 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step75000 samples/sec: 352.992 | iteration 78500/ 143000 | elapsed time per iteration (ms): 2900.9 | learning rate: 2.930E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.124752E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.11 | backward: 2203.71 | backward-backward: 2203.64 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step75500 samples/sec: 353.035 | iteration 79000/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 2.900E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.124064E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.27 | backward: 2205.40 | backward-backward: 2205.33 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.37 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step76000 samples/sec: 353.161 | iteration 79500/ 143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 2.870E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.124916E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.13 | backward: 2204.26 | backward-backward: 2204.19 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step76500 samples/sec: 353.076 | iteration 80000/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 2.841E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.123121E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.38 | backward: 2204.68 | backward-backward: 2204.61 | backward-allreduce: 0.00 | optimizer: 79.59 | batch generator: 2.39 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step77000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 80000 | lm_loss value: 2.076035E+00 | lm_loss_ppl value: 7.972794E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.235 | iteration 80500/ 143000 | elapsed time per iteration (ms): 3027.5 | learning rate: 2.812E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.123672E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.02 | backward: 2204.81 | backward-backward: 2204.74 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.72 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step77500 samples/sec: 353.067 | iteration 81000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 2.782E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.121719E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.56 | backward: 2205.67 | backward-backward: 2205.60 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step78000 samples/sec: 353.116 | iteration 81500/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 2.753E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.122009E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.43 | backward: 2205.23 | backward-backward: 2205.16 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step78500 samples/sec: 353.068 | iteration 82000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 2.724E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.120729E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.54 | backward: 2206.03 | backward-backward: 2205.95 | backward-allreduce: 0.00 | optimizer: 79.31 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step79000 samples/sec: 353.094 | iteration 82500/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 2.694E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.118500E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.46 | backward: 2205.51 | backward-backward: 2205.44 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step79500 samples/sec: 353.043 | iteration 83000/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 2.665E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.117742E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.96 | backward: 2205.44 | backward-backward: 2205.37 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step80000 samples/sec: 353.113 | iteration 83500/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 2.636E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.119061E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.22 | backward: 2204.66 | backward-backward: 2204.58 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step80500 samples/sec: 353.104 | iteration 84000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 2.607E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.117008E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 610.06 | backward: 2205.13 | backward-backward: 2205.06 | backward-allreduce: 0.00 | optimizer: 79.29 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step81000 samples/sec: 353.060 | iteration 84500/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 2.578E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.114706E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 610.05 | backward: 2205.29 | backward-backward: 2205.22 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step81500 samples/sec: 353.105 | iteration 85000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 2.550E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.114689E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.21 | backward: 2204.64 | backward-backward: 2204.57 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step82000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 85000 | lm_loss value: 2.068848E+00 | lm_loss_ppl value: 7.915702E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.150 | iteration 85500/ 143000 | elapsed time per iteration (ms): 3028.2 | learning rate: 2.521E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.112462E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.06 | backward: 2205.70 | backward-backward: 2205.62 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.76 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step82500 samples/sec: 353.077 | iteration 86000/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 2.492E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.112222E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.21 | backward: 2204.92 | backward-backward: 2204.84 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.35 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step83000 samples/sec: 353.025 | iteration 86500/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 2.464E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.113981E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.99 | backward: 2205.67 | backward-backward: 2205.60 | backward-allreduce: 0.00 | optimizer: 79.29 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step83500 samples/sec: 353.073 | iteration 87000/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 2.435E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.111177E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.18 | backward: 2204.98 | backward-backward: 2204.91 | backward-allreduce: 0.00 | optimizer: 79.62 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step84000 samples/sec: 352.984 | iteration 87500/ 143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 2.407E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.109004E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.99 | backward: 2205.97 | backward-backward: 2205.90 | backward-allreduce: 0.00 | optimizer: 79.43 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step84500 samples/sec: 353.019 | iteration 88000/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 2.379E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.110119E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.07 | backward: 2205.51 | backward-backward: 2205.43 | backward-allreduce: 0.00 | optimizer: 79.60 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step85000 samples/sec: 352.963 | iteration 88500/ 143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 2.351E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.108484E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.67 | backward: 2206.40 | backward-backward: 2206.32 | backward-allreduce: 0.00 | optimizer: 79.45 | batch generator: 2.32 samples/sec: 352.470 | iteration 89000/ 143000 | elapsed time per iteration (ms): 2905.2 | learning rate: 2.323E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.108585E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | after 89000 iterations memory (MB) | allocated: 3746.75439453125 | max allocated: 11575.26025390625 | reserved: 13710.0 | max reserved: 13710.0 time (ms) | forward: 616.27 | backward: 2208.27 | backward-backward: 2208.19 | backward-allreduce: 0.00 | optimizer: 79.34 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step86000 samples/sec: 352.898 | iteration 89500/ 143000 | elapsed time per iteration (ms): 2901.7 | learning rate: 2.295E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.106150E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.45 | backward: 2207.18 | backward-backward: 2207.11 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step86500 samples/sec: 352.933 | iteration 90000/ 143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 2.267E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.105758E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.27 | backward: 2207.37 | backward-backward: 2207.29 | backward-allreduce: 0.00 | optimizer: 79.23 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step87000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 90000 | lm_loss value: 2.062608E+00 | lm_loss_ppl value: 7.866459E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.063 | iteration 90500/ 143000 | elapsed time per iteration (ms): 3029.0 | learning rate: 2.240E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.104899E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.41 | backward: 2207.09 | backward-backward: 2207.02 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.58 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step87500 samples/sec: 353.067 | iteration 91000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 2.212E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.103225E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.30 | backward: 2205.97 | backward-backward: 2205.90 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step88000 samples/sec: 352.923 | iteration 91500/ 143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 2.185E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.103656E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.50 | backward: 2206.87 | backward-backward: 2206.80 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step88500 samples/sec: 352.970 | iteration 92000/ 143000 | elapsed time per iteration (ms): 2901.1 | learning rate: 2.158E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.102270E+00 | loss scale: 8192.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 609.18 | backward: 2207.41 | backward-backward: 2207.33 | backward-allreduce: 0.00 | optimizer: 79.08 | batch generator: 2.20 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step89000 samples/sec: 352.823 | iteration 92500/ 143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 2.131E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.101580E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.45 | backward: 2207.84 | backward-backward: 2207.77 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.20 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step89500 samples/sec: 352.670 | iteration 93000/ 143000 | elapsed time per iteration (ms): 2903.6 | learning rate: 2.104E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.102271E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.20 | backward: 2208.67 | backward-backward: 2208.60 | backward-allreduce: 0.00 | optimizer: 80.12 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step90000 samples/sec: 352.881 | iteration 93500/ 143000 | elapsed time per iteration (ms): 2901.8 | learning rate: 2.077E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.098684E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.18 | backward: 2207.48 | backward-backward: 2207.41 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step90500 samples/sec: 352.937 | iteration 94000/ 143000 | elapsed time per iteration (ms): 2901.4 | learning rate: 2.050E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.098608E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.02 | backward: 2207.43 | backward-backward: 2207.36 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step91000 samples/sec: 352.822 | iteration 94500/ 143000 | elapsed time per iteration (ms): 2902.3 | learning rate: 2.024E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.099217E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.26 | backward: 2207.85 | backward-backward: 2207.78 | backward-allreduce: 0.00 | optimizer: 79.77 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step91500 samples/sec: 352.954 | iteration 95000/ 143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 1.998E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.096398E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.07 | backward: 2207.19 | backward-backward: 2207.12 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step92000 ---------------------------------------------------------------------------------------------------------- validation results at iteration 95000 | lm_loss value: 2.051986E+00 | lm_loss_ppl value: 7.783340E+00 | ---------------------------------------------------------------------------------------------------------- samples/sec: 338.135 | iteration 95500/ 143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 1.972E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.096801E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.28 | backward: 2206.82 | backward-backward: 2206.75 | backward-allreduce: 0.00 | optimizer: 79.24 | batch generator: 2.65 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step92500 samples/sec: 353.224 | iteration 96000/ 143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 1.946E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.097106E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.15 | backward: 2205.10 | backward-backward: 2205.03 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step93000 samples/sec: 353.030 | iteration 96500/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 1.920E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.094771E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.29 | backward: 2206.26 | backward-backward: 2206.19 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step93500 samples/sec: 353.177 | iteration 97000/ 143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 1.894E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.093601E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.13 | backward: 2205.23 | backward-backward: 2205.16 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step94000 samples/sec: 353.113 | iteration 97500/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 1.869E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.093295E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.35 | backward: 2205.70 | backward-backward: 2205.63 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step94500 samples/sec: 352.656 | iteration 98000/ 143000 | elapsed time per iteration (ms): 2903.7 | learning rate: 1.843E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.091473E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.11 | backward: 2205.78 | backward-backward: 2205.71 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step95000 samples/sec: 353.158 | iteration 98500/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 1.818E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.090431E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.32 | backward: 2205.61 | backward-backward: 2205.54 | backward-allreduce: 0.00 | optimizer: 79.22 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step95500 samples/sec: 353.155 | iteration 99000/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 1.793E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.090558E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.13 | backward: 2205.53 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step96000 samples/sec: 353.124 | iteration 99500/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 1.769E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.091439E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.39 | backward: 2205.55 | backward-backward: 2205.48 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step96500 samples/sec: 352.840 | iteration 100000/ 143000 | elapsed time per iteration (ms): 2902.2 | learning rate: 1.744E-04 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.089060E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.18 | backward: 2207.83 | backward-backward: 2207.76 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step97000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 100000 | lm_loss value: 2.045376E+00 | lm_loss_ppl value: 7.732068E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.133 | iteration 100500/ 143000 | elapsed time per iteration (ms): 3028.4 | learning rate: 1.720E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.087061E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.37 | backward: 2206.72 | backward-backward: 2206.64 | backward-allreduce: 0.00 | optimizer: 79.45 | batch generator: 2.63 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step97500 samples/sec: 353.093 | iteration 101000/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 1.696E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.086114E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.11 | backward: 2206.10 | backward-backward: 2206.02 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step98000 samples/sec: 353.044 | iteration 101500/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 1.672E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.086666E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.33 | backward: 2205.98 | backward-backward: 2205.91 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step98500 samples/sec: 353.053 | iteration 102000/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 1.648E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.085056E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.14 | backward: 2206.50 | backward-backward: 2206.43 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step99000 samples/sec: 353.010 | iteration 102500/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 1.624E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.084948E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.32 | backward: 2206.03 | backward-backward: 2205.96 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step99500 samples/sec: 353.093 | iteration 103000/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 1.601E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.083879E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.21 | backward: 2205.75 | backward-backward: 2205.68 | backward-allreduce: 0.00 | optimizer: 79.69 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step100000 samples/sec: 352.955 | iteration 103500/ 143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 1.578E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.083274E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.38 | backward: 2206.88 | backward-backward: 2206.81 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step100500 samples/sec: 353.067 | iteration 104000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.555E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.082632E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.18 | backward: 2206.20 | backward-backward: 2206.13 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step101000 samples/sec: 353.020 | iteration 104500/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 1.532E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.081282E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.35 | backward: 2206.60 | backward-backward: 2206.53 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step101500 samples/sec: 353.071 | iteration 105000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.509E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.082003E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.17 | backward: 2206.15 | backward-backward: 2206.08 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step102000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 105000 | lm_loss value: 2.036453E+00 | lm_loss_ppl value: 7.663379E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.098 | iteration 105500/ 143000 | elapsed time per iteration (ms): 3028.7 | learning rate: 1.487E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.080755E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.44 | backward: 2206.85 | backward-backward: 2206.77 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.64 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step102500 samples/sec: 352.999 | iteration 106000/ 143000 | elapsed time per iteration (ms): 2900.9 | learning rate: 1.465E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.078865E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.23 | backward: 2206.51 | backward-backward: 2206.44 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step103000 samples/sec: 353.002 | iteration 106500/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 1.443E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.078916E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.50 | backward: 2206.11 | backward-backward: 2206.04 | backward-allreduce: 0.00 | optimizer: 79.71 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step103500 samples/sec: 353.200 | iteration 107000/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 1.422E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.078951E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.27 | backward: 2205.09 | backward-backward: 2205.02 | backward-allreduce: 0.00 | optimizer: 79.39 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step104000 samples/sec: 353.138 | iteration 107500/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 1.400E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.077511E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.43 | backward: 2205.37 | backward-backward: 2205.29 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step104500 samples/sec: 353.109 | iteration 108000/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 1.379E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.075537E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.30 | backward: 2205.52 | backward-backward: 2205.45 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step105000 samples/sec: 353.074 | iteration 108500/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.358E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.075166E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.45 | backward: 2205.40 | backward-backward: 2205.33 | backward-allreduce: 0.00 | optimizer: 79.39 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step105500 samples/sec: 353.125 | iteration 109000/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 1.338E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.074447E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.20 | backward: 2205.49 | backward-backward: 2205.42 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step106000 samples/sec: 353.017 | iteration 109500/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 1.317E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.073662E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.41 | backward: 2206.34 | backward-backward: 2206.26 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step106500 samples/sec: 352.979 | iteration 110000/ 143000 | elapsed time per iteration (ms): 2901.0 | learning rate: 1.297E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.075984E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.31 | backward: 2206.66 | backward-backward: 2206.58 | backward-allreduce: 0.00 | optimizer: 79.57 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step107000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 110000 | lm_loss value: 2.029312E+00 | lm_loss_ppl value: 7.608849E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.003 | iteration 110500/ 143000 | elapsed time per iteration (ms): 3029.6 | learning rate: 1.277E-04 | approx flops per GPU: 73.1TFLOPS | lm_loss: 2.073421E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.47 | backward: 2207.30 | backward-backward: 2207.23 | backward-allreduce: 0.00 | optimizer: 79.75 | batch generator: 2.65 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step107500 samples/sec: 353.136 | iteration 111000/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 1.257E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.072429E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.23 | backward: 2205.60 | backward-backward: 2205.53 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.21 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step108000 samples/sec: 353.108 | iteration 111500/ 143000 | elapsed time per iteration (ms): 2900.0 | learning rate: 1.238E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.070852E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.38 | backward: 2205.52 | backward-backward: 2205.45 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.21 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step108500 samples/sec: 353.153 | iteration 112000/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 1.219E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.067711E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.25 | backward: 2205.53 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.38 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step109000 samples/sec: 353.072 | iteration 112500/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.200E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.069139E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.40 | backward: 2205.82 | backward-backward: 2205.75 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step109500 samples/sec: 353.082 | iteration 113000/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.181E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.066094E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.35 | backward: 2205.73 | backward-backward: 2205.66 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step110000 samples/sec: 353.081 | iteration 113500/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.163E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.068288E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.54 | backward: 2205.97 | backward-backward: 2205.89 | backward-allreduce: 0.00 | optimizer: 79.22 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step110500 samples/sec: 353.020 | iteration 114000/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 1.144E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.066782E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.39 | backward: 2205.94 | backward-backward: 2205.87 | backward-allreduce: 0.00 | optimizer: 79.76 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step111000 samples/sec: 353.041 | iteration 114500/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 1.127E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.066362E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.49 | backward: 2206.20 | backward-backward: 2206.13 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step111500 samples/sec: 353.061 | iteration 115000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 1.109E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.065361E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.38 | backward: 2205.99 | backward-backward: 2205.92 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step112000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 115000 | lm_loss value: 2.016774E+00 | lm_loss_ppl value: 7.514043E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.209 | iteration 115500/ 143000 | elapsed time per iteration (ms): 3027.7 | learning rate: 1.092E-04 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.063500E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.42 | backward: 2205.58 | backward-backward: 2205.51 | backward-allreduce: 0.00 | optimizer: 79.72 | batch generator: 2.64 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step112500 samples/sec: 353.004 | iteration 116000/ 143000 | elapsed time per iteration (ms): 2900.8 | learning rate: 1.075E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.064721E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.42 | backward: 2206.27 | backward-backward: 2206.20 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step113000 samples/sec: 353.084 | iteration 116500/ 143000 | elapsed time per iteration (ms): 2900.2 | learning rate: 1.058E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.064561E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.59 | backward: 2205.72 | backward-backward: 2205.65 | backward-allreduce: 0.00 | optimizer: 79.24 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step113500 samples/sec: 352.924 | iteration 117000/ 143000 | elapsed time per iteration (ms): 2901.5 | learning rate: 1.041E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.061940E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.42 | backward: 2206.61 | backward-backward: 2206.54 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step114000 samples/sec: 353.029 | iteration 117500/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 1.025E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.062451E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.61 | backward: 2206.10 | backward-backward: 2206.03 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step114500 samples/sec: 352.905 | iteration 118000/ 143000 | elapsed time per iteration (ms): 2901.6 | learning rate: 1.009E-04 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.061479E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.44 | backward: 2207.04 | backward-backward: 2206.97 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.28 samples/sec: 352.810 | iteration 118500/ 143000 | elapsed time per iteration (ms): 2902.4 | learning rate: 9.933E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.060447E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | after 118500 iterations memory (MB) | allocated: 3623.939453125 | max allocated: 11451.5068359375 | reserved: 13710.0 | max reserved: 13710.0 time (ms) | forward: 611.70 | backward: 2209.83 | backward-backward: 2209.76 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step115500 samples/sec: 353.042 | iteration 119000/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 9.778E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.058811E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.03 | backward: 2205.13 | backward-backward: 2205.06 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.27 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step116000 samples/sec: 353.040 | iteration 119500/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 9.627E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.059651E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.77 | backward: 2205.63 | backward-backward: 2205.56 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.28 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step116500 samples/sec: 353.094 | iteration 120000/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 9.478E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.058310E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.97 | backward: 2205.14 | backward-backward: 2205.07 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step117000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 120000 | lm_loss value: 2.013359E+00 | lm_loss_ppl value: 7.488429E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.314 | iteration 120500/ 143000 | elapsed time per iteration (ms): 3026.8 | learning rate: 9.334E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.056560E+00 | loss scale: 16384.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 610.09 | backward: 2204.59 | backward-backward: 2204.52 | backward-allreduce: 0.00 | optimizer: 79.06 | batch generator: 2.76 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step117500 samples/sec: 353.022 | iteration 121000/ 143000 | elapsed time per iteration (ms): 2900.7 | learning rate: 9.191E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.057362E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.17 | backward: 2205.14 | backward-backward: 2205.07 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.37 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step118000 samples/sec: 353.183 | iteration 121500/ 143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 9.051E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.056591E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.83 | backward: 2204.49 | backward-backward: 2204.42 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step118500 samples/sec: 353.131 | iteration 122000/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 8.914E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.055705E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.11 | backward: 2204.43 | backward-backward: 2204.36 | backward-allreduce: 0.00 | optimizer: 79.68 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step119000 samples/sec: 353.233 | iteration 122500/ 143000 | elapsed time per iteration (ms): 2898.9 | learning rate: 8.781E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.056121E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.95 | backward: 2204.14 | backward-backward: 2204.07 | backward-allreduce: 0.00 | optimizer: 79.40 | batch generator: 2.31 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step119500 samples/sec: 353.173 | iteration 123000/ 143000 | elapsed time per iteration (ms): 2899.4 | learning rate: 8.650E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.056422E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.10 | backward: 2204.31 | backward-backward: 2204.24 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step120000 samples/sec: 353.259 | iteration 123500/ 143000 | elapsed time per iteration (ms): 2898.7 | learning rate: 8.522E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.054773E+00 | loss scale: 16384.0 | number of skipped iterations: 3 | number of nan iterations: 0 | time (ms) | forward: 609.95 | backward: 2204.21 | backward-backward: 2204.14 | backward-allreduce: 0.00 | optimizer: 79.19 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step120500 samples/sec: 353.137 | iteration 124000/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 8.398E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.053937E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 610.21 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.34 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step121000 samples/sec: 353.167 | iteration 124500/ 143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 8.276E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.052470E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.99 | backward: 2204.39 | backward-backward: 2204.32 | backward-allreduce: 0.00 | optimizer: 79.67 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step121500 samples/sec: 353.132 | iteration 125000/ 143000 | elapsed time per iteration (ms): 2899.8 | learning rate: 8.157E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.053600E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.88 | backward: 2204.79 | backward-backward: 2204.72 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step122000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 125000 | lm_loss value: 2.008578E+00 | lm_loss_ppl value: 7.452709E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.435 | iteration 125500/ 143000 | elapsed time per iteration (ms): 3025.7 | learning rate: 8.041E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.052524E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.53 | backward: 2203.95 | backward-backward: 2203.88 | backward-allreduce: 0.00 | optimizer: 79.37 | batch generator: 2.65 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step122500 samples/sec: 353.238 | iteration 126000/ 143000 | elapsed time per iteration (ms): 2898.9 | learning rate: 7.929E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051470E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.68 | backward: 2204.45 | backward-backward: 2204.38 | backward-allreduce: 0.00 | optimizer: 79.32 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step123000 samples/sec: 353.280 | iteration 126500/ 143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 7.819E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051416E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.48 | backward: 2203.99 | backward-backward: 2203.92 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step123500 samples/sec: 353.207 | iteration 127000/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 7.713E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051840E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.66 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step124000 samples/sec: 353.283 | iteration 127500/ 143000 | elapsed time per iteration (ms): 2898.5 | learning rate: 7.609E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.050111E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.53 | backward: 2203.98 | backward-backward: 2203.91 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step124500 samples/sec: 353.206 | iteration 128000/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 7.509E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.051122E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.78 | backward: 2204.40 | backward-backward: 2204.33 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step125000 samples/sec: 353.334 | iteration 128500/ 143000 | elapsed time per iteration (ms): 2898.1 | learning rate: 7.412E-05 | approx flops per GPU: 76.5TFLOPS | lm_loss: 2.049028E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.05 | backward: 2204.39 | backward-backward: 2204.32 | backward-allreduce: 0.00 | optimizer: 79.20 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step125500 samples/sec: 353.224 | iteration 129000/ 143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 7.318E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.049796E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.32 | backward: 2204.66 | backward-backward: 2204.60 | backward-allreduce: 0.00 | optimizer: 79.51 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step126000 samples/sec: 353.378 | iteration 129500/ 143000 | elapsed time per iteration (ms): 2897.7 | learning rate: 7.228E-05 | approx flops per GPU: 76.5TFLOPS | lm_loss: 2.046924E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.03 | backward: 2204.02 | backward-backward: 2203.95 | backward-allreduce: 0.00 | optimizer: 79.30 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step126500 samples/sec: 353.229 | iteration 130000/ 143000 | elapsed time per iteration (ms): 2899.0 | learning rate: 7.140E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.048169E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.67 | backward: 2204.25 | backward-backward: 2204.18 | backward-allreduce: 0.00 | optimizer: 79.56 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step127000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 130000 | lm_loss value: 2.005995E+00 | lm_loss_ppl value: 7.433485E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.434 | iteration 130500/ 143000 | elapsed time per iteration (ms): 3025.7 | learning rate: 7.056E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.048540E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.73 | backward: 2204.00 | backward-backward: 2203.93 | backward-allreduce: 0.00 | optimizer: 79.24 | batch generator: 2.63 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step127500 samples/sec: 353.168 | iteration 131000/ 143000 | elapsed time per iteration (ms): 2899.5 | learning rate: 6.974E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.048352E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.99 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128000 samples/sec: 353.268 | iteration 131500/ 143000 | elapsed time per iteration (ms): 2898.7 | learning rate: 6.896E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.046191E+00 | loss scale: 65536.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.81 | backward: 2203.79 | backward-backward: 2203.72 | backward-allreduce: 0.00 | optimizer: 79.55 | batch generator: 2.32 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step128500 samples/sec: 352.743 | iteration 132000/ 143000 | elapsed time per iteration (ms): 2903.0 | learning rate: 6.822E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.046235E+00 | loss scale: 32768.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.96 | backward: 2208.10 | backward-backward: 2208.03 | backward-allreduce: 0.00 | optimizer: 79.23 | batch generator: 2.33 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step129000 samples/sec: 353.251 | iteration 132500/ 143000 | elapsed time per iteration (ms): 2898.8 | learning rate: 6.750E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.046550E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.71 | backward: 2204.34 | backward-backward: 2204.27 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.29 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step129500 samples/sec: 353.110 | iteration 133000/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 6.681E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.045019E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.84 | backward: 2205.06 | backward-backward: 2204.99 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.25 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step130000 samples/sec: 352.687 | iteration 133500/ 143000 | elapsed time per iteration (ms): 2903.4 | learning rate: 6.616E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.044678E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 612.96 | backward: 2205.08 | backward-backward: 2205.01 | backward-allreduce: 0.00 | optimizer: 79.77 | batch generator: 2.40 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step130500 samples/sec: 353.041 | iteration 134000/ 143000 | elapsed time per iteration (ms): 2900.5 | learning rate: 6.554E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.043597E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.57 | backward: 2205.57 | backward-backward: 2205.50 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step131000 samples/sec: 353.273 | iteration 134500/ 143000 | elapsed time per iteration (ms): 2898.6 | learning rate: 6.496E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.043592E+00 | loss scale: 65536.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 608.95 | backward: 2204.99 | backward-backward: 2204.92 | backward-allreduce: 0.00 | optimizer: 79.33 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step131500 samples/sec: 353.157 | iteration 135000/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 6.440E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.043424E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.14 | backward: 2205.64 | backward-backward: 2205.57 | backward-allreduce: 0.00 | optimizer: 79.33 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step132000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 135000 | lm_loss value: 2.000182E+00 | lm_loss_ppl value: 7.390400E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.294 | iteration 135500/ 143000 | elapsed time per iteration (ms): 3027.0 | learning rate: 6.388E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.042341E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.49 | backward: 2205.20 | backward-backward: 2205.13 | backward-allreduce: 0.00 | optimizer: 79.46 | batch generator: 2.62 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step132500 samples/sec: 353.116 | iteration 136000/ 143000 | elapsed time per iteration (ms): 2899.9 | learning rate: 6.339E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.044158E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.83 | backward: 2205.09 | backward-backward: 2205.02 | backward-allreduce: 0.00 | optimizer: 79.54 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step133000 samples/sec: 353.214 | iteration 136500/ 143000 | elapsed time per iteration (ms): 2899.1 | learning rate: 6.293E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.041776E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.19 | backward: 2204.87 | backward-backward: 2204.80 | backward-allreduce: 0.00 | optimizer: 79.53 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step133500 samples/sec: 352.813 | iteration 137000/ 143000 | elapsed time per iteration (ms): 2902.4 | learning rate: 6.251E-05 | approx flops per GPU: 76.3TFLOPS | lm_loss: 2.043309E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.14 | backward: 2206.15 | backward-backward: 2206.08 | backward-allreduce: 0.00 | optimizer: 79.52 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step134000 samples/sec: 353.197 | iteration 137500/ 143000 | elapsed time per iteration (ms): 2899.2 | learning rate: 6.212E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040981E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 608.93 | backward: 2205.39 | backward-backward: 2205.32 | backward-allreduce: 0.00 | optimizer: 79.34 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step134500 samples/sec: 353.091 | iteration 138000/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 6.176E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040547E+00 | loss scale: 8192.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.11 | backward: 2206.17 | backward-backward: 2206.11 | backward-allreduce: 0.00 | optimizer: 79.30 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step135000 samples/sec: 353.149 | iteration 138500/ 143000 | elapsed time per iteration (ms): 2899.6 | learning rate: 6.144E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.042986E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.07 | backward: 2205.53 | backward-backward: 2205.46 | backward-allreduce: 0.00 | optimizer: 79.61 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step135500 samples/sec: 353.067 | iteration 139000/ 143000 | elapsed time per iteration (ms): 2900.3 | learning rate: 6.115E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.041756E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.23 | backward: 2205.98 | backward-backward: 2205.91 | backward-allreduce: 0.00 | optimizer: 79.64 | batch generator: 2.26 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136000 samples/sec: 353.187 | iteration 139500/ 143000 | elapsed time per iteration (ms): 2899.3 | learning rate: 6.089E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040638E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.07 | backward: 2205.21 | backward-backward: 2205.14 | backward-allreduce: 0.00 | optimizer: 79.65 | batch generator: 2.30 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step136500 samples/sec: 353.088 | iteration 140000/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 6.066E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039759E+00 | loss scale: 32768.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.19 | backward: 2205.90 | backward-backward: 2205.83 | backward-allreduce: 0.00 | optimizer: 79.58 | batch generator: 2.24 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step137000 ----------------------------------------------------------------------------------------------------------- validation results at iteration 140000 | lm_loss value: 1.998048E+00 | lm_loss_ppl value: 7.374645E+00 | ----------------------------------------------------------------------------------------------------------- samples/sec: 338.334 | iteration 140500/ 143000 | elapsed time per iteration (ms): 3026.6 | learning rate: 6.047E-05 | approx flops per GPU: 73.2TFLOPS | lm_loss: 2.040289E+00 | loss scale: 32768.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.00 | backward: 2205.39 | backward-backward: 2205.32 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.58 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step137500 samples/sec: 353.029 | iteration 141000/ 143000 | elapsed time per iteration (ms): 2900.6 | learning rate: 6.031E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.040376E+00 | loss scale: 16384.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.20 | backward: 2206.47 | backward-backward: 2206.40 | backward-allreduce: 0.00 | optimizer: 79.36 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step138000 samples/sec: 353.137 | iteration 141500/ 143000 | elapsed time per iteration (ms): 2899.7 | learning rate: 6.018E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039720E+00 | loss scale: 16384.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.03 | backward: 2205.80 | backward-backward: 2205.73 | backward-allreduce: 0.00 | optimizer: 79.49 | batch generator: 2.23 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step138500 samples/sec: 353.059 | iteration 142000/ 143000 | elapsed time per iteration (ms): 2900.4 | learning rate: 6.009E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.041119E+00 | loss scale: 16384.0 | number of skipped iterations: 2 | number of nan iterations: 0 | time (ms) | forward: 609.17 | backward: 2206.66 | backward-backward: 2206.59 | backward-allreduce: 0.00 | optimizer: 79.17 | batch generator: 2.21 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step139000 samples/sec: 353.093 | iteration 142500/ 143000 | elapsed time per iteration (ms): 2900.1 | learning rate: 6.003E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039393E+00 | loss scale: 8192.0 | number of skipped iterations: 1 | number of nan iterations: 0 | time (ms) | forward: 609.09 | backward: 2206.25 | backward-backward: 2206.18 | backward-allreduce: 0.00 | optimizer: 79.39 | batch generator: 2.22 WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step139500 samples/sec: 352.952 | iteration 143000/ 143000 | elapsed time per iteration (ms): 2901.2 | learning rate: 6.000E-05 | approx flops per GPU: 76.4TFLOPS | lm_loss: 2.039866E+00 | loss scale: 8192.0 | number of skipped iterations: 0 | number of nan iterations: 0 | time (ms) | forward: 609.83 | backward: 2206.30 | backward-backward: 2206.23 | backward-allreduce: 0.00 | optimizer: 79.50 | batch generator: 2.25 --------------------------------------------------------------------------------------------------------------------------- validation results at the end of training for val data | lm_loss value: 1.999463E+00 | lm_loss_ppl value: 7.385092E+00 | --------------------------------------------------------------------------------------------------------------------------- WARNING: Deleting old checkpoints: /u/wangh/workspace/project/NextModel/gpt-neox/work_dirs/flownet/900M_nogptj_nobias_nomalinit_wtying_nonparamln_finalwithparam/checkpoints/global_step140000 ---------------------------------------------------------------------------------------------------------------------- test results at the end of training for test data | lm_loss value: 1.992931E+00 | lm_loss_ppl value: 7.337004E+00 | ----------------------------------------------------------------------------------------------------------------------