25: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 10: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 4: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 12: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 16: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 9: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 22: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 26: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 19: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 21: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 30: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 28: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 14: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 2: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 21: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 3: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 21: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 10: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 1: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 10: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 21: START 2058110: Thu Nov 24 16:46:14 EET 2022 10: START 2058110: Thu Nov 24 16:46:14 EET 2022 23: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 20: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 18: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 24: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 13: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 0: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 31: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 17: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 29: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 5: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 7: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 15: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 8: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 11: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 6: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 27: Model parameters: d_model 640 ffw_size 2560 kv_size 64 n_heads 10 n_layers 10 25: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 25: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 4: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 4: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 25: START 2058110: Thu Nov 24 16:46:14 EET 2022 4: START 2058110: Thu Nov 24 16:46:14 EET 2022 16: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 16: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 22: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 16: START 2058110: Thu Nov 24 16:46:14 EET 2022 22: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 22: START 2058110: Thu Nov 24 16:46:14 EET 2022 26: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 26: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 26: START 2058110: Thu Nov 24 16:46:14 EET 2022 12: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 12: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 12: START 2058110: Thu Nov 24 16:46:14 EET 2022 19: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 19: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 21: 21: 21: ======================= ROCm System Management Interface ======================= 21: ================================= Concise Info ================================= 21: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 21: 0 46.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 21: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 21: 2 34.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 21: 3 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 21: 4 42.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 21: 5 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 21: 6 43.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 21: 7 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 21: ================================================================================ 21: ============================= End of ROCm SMI Log ============================== 19: START 2058110: Thu Nov 24 16:46:14 EET 2022 10: 10: 10: ======================= ROCm System Management Interface ======================= 10: ================================= Concise Info ================================= 10: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 10: 0 42.0c 98.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 10: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 10: 2 42.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 10: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 10: 4 45.0c 77.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 10: 5 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 10: 6 41.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 10: 7 39.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 10: ================================================================================ 10: ============================= End of ROCm SMI Log ============================== 9: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 9: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 30: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 30: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 9: START 2058110: Thu Nov 24 16:46:14 EET 2022 30: START 2058110: Thu Nov 24 16:46:14 EET 2022 28: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 28: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 28: START 2058110: Thu Nov 24 16:46:14 EET 2022 4: 4: 4: ======================= ROCm System Management Interface ======================= 4: ================================= Concise Info ================================= 4: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 4: 0 41.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 1 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 2 40.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 3 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 4 44.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 5 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 6 42.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: ================================================================================ 4: ============================= End of ROCm SMI Log ============================== 25: 25: 25: ======================= ROCm System Management Interface ======================= 25: ================================= Concise Info ================================= 25: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 25: 0 45.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 25: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 25: 2 42.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 25: 3 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 25: 4 47.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 25: 5 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 25: 6 39.0c 97.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 25: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 25: ================================================================================ 25: ============================= End of ROCm SMI Log ============================== 14: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 14: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 14: START 2058110: Thu Nov 24 16:46:14 EET 2022 2: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 2: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 2: START 2058110: Thu Nov 24 16:46:14 EET 2022 3: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 3: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 3: START 2058110: Thu Nov 24 16:46:14 EET 2022 1: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 1: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 1: START 2058110: Thu Nov 24 16:46:14 EET 2022 23: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 23: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 23: START 2058110: Thu Nov 24 16:46:14 EET 2022 20: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 20: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 18: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 20: START 2058110: Thu Nov 24 16:46:14 EET 2022 18: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 18: START 2058110: Thu Nov 24 16:46:14 EET 2022 24: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 24: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 13: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 24: START 2058110: Thu Nov 24 16:46:14 EET 2022 13: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 13: START 2058110: Thu Nov 24 16:46:14 EET 2022 0: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 0: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 0: START 2058110: Thu Nov 24 16:46:14 EET 2022 31: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 31: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 17: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 17: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 31: START 2058110: Thu Nov 24 16:46:14 EET 2022 17: START 2058110: Thu Nov 24 16:46:14 EET 2022 29: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 29: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 5: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 29: START 2058110: Thu Nov 24 16:46:14 EET 2022 5: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 22: 22: 22: ======================= ROCm System Management Interface ======================= 22: ================================= Concise Info ================================= 22: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 22: 0 48.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 22: 1 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 22: 2 45.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 22: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 22: 4 44.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 22: 5 50.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 22: 6 41.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 22: 7 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 22: ================================================================================ 22: ============================= End of ROCm SMI Log ============================== 5: START 2058110: Thu Nov 24 16:46:14 EET 2022 7: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 16: 16: 16: ======================= ROCm System Management Interface ======================= 16: ================================= Concise Info ================================= 16: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 16: 0 41.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 16: 1 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 16: 2 34.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 16: 3 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 16: 4 46.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 16: 5 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 16: 6 44.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 16: 7 40.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 16: ================================================================================ 16: ============================= End of ROCm SMI Log ============================== 7: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 26: 26: 26: ======================= ROCm System Management Interface ======================= 26: ================================= Concise Info ================================= 26: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 26: 0 50.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 26: 1 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 26: 2 42.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 26: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 26: 4 44.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 26: 5 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 26: 6 40.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 26: 7 40.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 26: ================================================================================ 26: ============================= End of ROCm SMI Log ============================== 12: 12: 12: ======================= ROCm System Management Interface ======================= 12: ================================= Concise Info ================================= 12: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 12: 0 45.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 12: 1 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 12: 2 45.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 12: 3 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 12: 4 45.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 12: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 12: 6 47.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 12: 7 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 12: ================================================================================ 12: ============================= End of ROCm SMI Log ============================== 7: START 2058110: Thu Nov 24 16:46:14 EET 2022 15: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 15: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 19: 19: 19: ======================= ROCm System Management Interface ======================= 19: ================================= Concise Info ================================= 19: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 19: 0 47.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 19: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 19: 2 46.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 19: 3 40.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 19: 4 43.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 19: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 19: 6 43.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 19: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 19: ================================================================================ 19: ============================= End of ROCm SMI Log ============================== 15: START 2058110: Thu Nov 24 16:46:14 EET 2022 8: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 8: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 8: START 2058110: Thu Nov 24 16:46:14 EET 2022 11: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 11: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 9: 9: 9: ======================= ROCm System Management Interface ======================= 9: ================================= Concise Info ================================= 9: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 9: 0 46.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 9: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 9: 2 41.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 9: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 9: 4 44.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 9: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 9: 6 46.0c 83.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 9: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 9: ================================================================================ 9: ============================= End of ROCm SMI Log ============================== 11: START 2058110: Thu Nov 24 16:46:14 EET 2022 30: 30: 30: ======================= ROCm System Management Interface ======================= 30: ================================= Concise Info ================================= 30: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 30: 0 45.0c 97.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 30: 1 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 30: 2 43.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 30: 3 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 30: 4 42.0c 99.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 30: 5 38.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 30: 6 40.0c 98.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 30: 7 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 30: ================================================================================ 30: ============================= End of ROCm SMI Log ============================== 6: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 6: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 6: START 2058110: Thu Nov 24 16:46:14 EET 2022 27: Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 10 --hidden-size 640 --num-attention-heads 10 --kv-channels 64 --ffn-hidden-size 2560 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 1 --global-batch-size 256 --train-samples 9_703_701 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-1 --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 9_703_701 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 10 --save-interval 1000 --eval-interval 1000 --eval-iters 1 --tensorboard-dir tensorboard_83m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_83m --load checkpoints_83m --data-path /scratch/project_462000119/data/pile/megatron_data/meg-gp 27: t2_pile_text_document --data-impl mmap --split 949,50,1 --deepspeed --deepspeed_config ds_configs/2058110.json --zero-stage 0 27: START 2058110: Thu Nov 24 16:46:14 EET 2022 28: 28: 28: ======================= ROCm System Management Interface ======================= 28: ================================= Concise Info ================================= 28: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 28: 0 44.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 28: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 28: 2 41.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 28: 3 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 28: 4 43.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 28: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 28: 6 42.0c 97.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 28: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 28: ================================================================================ 28: ============================= End of ROCm SMI Log ============================== 2: 2: 2: ======================= ROCm System Management Interface ======================= 2: ================================= Concise Info ================================= 2: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2: 0 42.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 1 50.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 2 37.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 3 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 4 42.0c 97.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 6 44.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: ================================================================================ 2: ============================= End of ROCm SMI Log ============================== 14: 14: 14: ======================= ROCm System Management Interface ======================= 14: ================================= Concise Info ================================= 14: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 14: 0 42.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 14: 1 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 14: 2 44.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 14: 3 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 14: 4 39.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 14: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 14: 6 44.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 14: 7 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 14: ================================================================================ 14: ============================= End of ROCm SMI Log ============================== 3: 3: 3: ======================= ROCm System Management Interface ======================= 3: ================================= Concise Info ================================= 3: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 3: 0 47.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 1 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 2 42.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 3 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 4 40.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 5 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 6 45.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: ================================================================================ 3: ============================= End of ROCm SMI Log ============================== 1: 1: 1: ======================= ROCm System Management Interface ======================= 1: ================================= Concise Info ================================= 1: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 1: 0 44.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 1 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 2 41.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 3 40.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 4 42.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 5 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 6 40.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: ================================================================================ 1: ============================= End of ROCm SMI Log ============================== 23: 23: 23: ======================= ROCm System Management Interface ======================= 23: ================================= Concise Info ================================= 23: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 23: 0 48.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 23: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 23: 2 43.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 23: 3 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 23: 4 39.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 23: 5 51.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 23: 6 41.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 23: 7 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 23: ================================================================================ 23: ============================= End of ROCm SMI Log ============================== 20: 20: 20: ======================= ROCm System Management Interface ======================= 20: ================================= Concise Info ================================= 20: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 20: 0 47.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 20: 1 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 20: 2 41.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 20: 3 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 20: 4 46.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 20: 5 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 20: 6 45.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 20: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 20: ================================================================================ 20: ============================= End of ROCm SMI Log ============================== 24: 24: 24: ======================= ROCm System Management Interface ======================= 24: ================================= Concise Info ================================= 24: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 24: 0 44.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 24: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 24: 2 40.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 24: 3 40.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 24: 4 43.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 24: 5 53.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 24: 6 46.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 24: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 24: ================================================================================ 24: ============================= End of ROCm SMI Log ============================== 18: 18: 18: ======================= ROCm System Management Interface ======================= 18: ================================= Concise Info ================================= 18: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 18: 0 42.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 18: 1 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 18: 2 42.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 18: 3 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 18: 4 43.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 18: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 18: 6 46.0c 82.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 18: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 18: ================================================================================ 18: ============================= End of ROCm SMI Log ============================== 13: 13: 13: ======================= ROCm System Management Interface ======================= 13: ================================= Concise Info ================================= 13: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 13: 0 44.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 13: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 13: 2 37.0c 81.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 13: 3 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 13: 4 39.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 13: 5 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 13: 6 37.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 13: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 13: ================================================================================ 13: ============================= End of ROCm SMI Log ============================== 0: 0: 0: ======================= ROCm System Management Interface ======================= 0: ================================= Concise Info ================================= 0: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 0: 0 43.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 1 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 2 38.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 4 37.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 6 37.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 7 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: ================================================================================ 0: ============================= End of ROCm SMI Log ============================== 31: 31: 31: ======================= ROCm System Management Interface ======================= 31: ================================= Concise Info ================================= 31: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 31: 0 46.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 31: 1 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 31: 2 42.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 31: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 31: 4 44.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 31: 5 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 31: 6 41.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 31: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 31: ================================================================================ 31: ============================= End of ROCm SMI Log ============================== 29: 29: 29: ======================= ROCm System Management Interface ======================= 29: ================================= Concise Info ================================= 29: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 29: 0 49.0c 98.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 29: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 29: 2 41.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 29: 3 39.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 29: 4 42.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 29: 5 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 29: 6 40.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 29: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 29: ================================================================================ 29: ============================= End of ROCm SMI Log ============================== 17: 17: 17: ======================= ROCm System Management Interface ======================= 17: ================================= Concise Info ================================= 17: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 17: 0 46.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 17: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 17: 2 40.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 17: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 17: 4 43.0c 82.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 17: 5 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 17: 6 42.0c 84.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 17: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 17: ================================================================================ 17: ============================= End of ROCm SMI Log ============================== 5: 5: 5: ======================= ROCm System Management Interface ======================= 5: ================================= Concise Info ================================= 5: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 5: 0 46.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 2 45.0c 83.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 3 40.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 4 42.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 6 41.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: ================================================================================ 5: ============================= End of ROCm SMI Log ============================== 15: 15: 15: ======================= ROCm System Management Interface ======================= 15: ================================= Concise Info ================================= 15: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 15: 0 41.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 15: 1 53.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 15: 2 39.0c 100.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 15: 3 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 15: 4 44.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 15: 5 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 15: 6 40.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 15: 7 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 15: ================================================================================ 15: ============================= End of ROCm SMI Log ============================== 7: 7: 7: ======================= ROCm System Management Interface ======================= 7: ================================= Concise Info ================================= 7: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 7: 0 50.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 1 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 2 41.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 3 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 4 47.0c 81.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 5 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 6 39.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: ================================================================================ 7: ============================= End of ROCm SMI Log ============================== 8: 8: 8: ======================= ROCm System Management Interface ======================= 8: ================================= Concise Info ================================= 8: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 8: 0 43.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 8: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 8: 2 43.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 8: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 8: 4 38.0c 100.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 8: 5 53.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 8: 6 41.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 8: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 8: ================================================================================ 8: ============================= End of ROCm SMI Log ============================== 6: 6: 6: ======================= ROCm System Management Interface ======================= 6: ================================= Concise Info ================================= 6: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 6: 0 44.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 2 42.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 3 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 4 42.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 5 50.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 6 40.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 7 38.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: ================================================================================ 6: ============================= End of ROCm SMI Log ============================== 11: 11: 11: ======================= ROCm System Management Interface ======================= 11: ================================= Concise Info ================================= 11: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 11: 0 44.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 11: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 11: 2 40.0c 84.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 11: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 11: 4 46.0c 80.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 11: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 11: 6 38.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 11: 7 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 11: ================================================================================ 11: ============================= End of ROCm SMI Log ============================== 27: 27: 27: ======================= ROCm System Management Interface ======================= 27: ================================= Concise Info ================================= 27: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 27: 0 43.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 27: 1 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 27: 2 42.0c 94.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 27: 3 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 27: 4 38.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 27: 5 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 27: 6 48.0c 80.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 27: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 27: ================================================================================ 27: ============================= End of ROCm SMI Log ============================== 16: Launching on nid005397 (16/32), master nid005360 port 9999, GPUs 8, CUDA: True 0: Launching on nid005360 (0/32), master nid005360 port 9999, GPUs 8, CUDA: True 25: Launching on nid005417 (25/32), master nid005360 port 9999, GPUs 8, CUDA: True 22: Launching on nid005414 (22/32), master nid005360 port 9999, GPUs 8, CUDA: True 29: Launching on nid005421 (29/32), master nid005360 port 9999, GPUs 8, CUDA: True 13: Launching on nid005373 (13/32), master nid005360 port 9999, GPUs 8, CUDA: True 19: Launching on nid005411 (19/32), master nid005360 port 9999, GPUs 8, CUDA: True 8: Launching on nid005368 (8/32), master nid005360 port 9999, GPUs 8, CUDA: True 27: Launching on nid005419 (27/32), master nid005360 port 9999, GPUs 8, CUDA: True 1: Launching on nid005361 (1/32), master nid005360 port 9999, GPUs 8, CUDA: True 14: Launching on nid005395 (14/32), master nid005360 port 9999, GPUs 8, CUDA: True 4: Launching on nid005364 (4/32), master nid005360 port 9999, GPUs 8, CUDA: True 17: Launching on nid005398 (17/32), master nid005360 port 9999, GPUs 8, CUDA: True 23: Launching on nid005415 (23/32), master nid005360 port 9999, GPUs 8, CUDA: True 10: Launching on nid005370 (10/32), master nid005360 port 9999, GPUs 8, CUDA: True 31: Launching on nid005423 (31/32), master nid005360 port 9999, GPUs 8, CUDA: True 5: Launching on nid005365 (5/32), master nid005360 port 9999, GPUs 8, CUDA: True 18: Launching on nid005399 (18/32), master nid005360 port 9999, GPUs 8, CUDA: True 7: Launching on nid005367 (7/32), master nid005360 port 9999, GPUs 8, CUDA: True 11: Launching on nid005371 (11/32), master nid005360 port 9999, GPUs 8, CUDA: True 2: Launching on nid005362 (2/32), master nid005360 port 9999, GPUs 8, CUDA: True 26: Launching on nid005418 (26/32), master nid005360 port 9999, GPUs 8, CUDA: True 28: Launching on nid005420 (28/32), master nid005360 port 9999, GPUs 8, CUDA: True 15: Launching on nid005396 (15/32), master nid005360 port 9999, GPUs 8, CUDA: True 9: Launching on nid005369 (9/32), master nid005360 port 9999, GPUs 8, CUDA: True 30: Launching on nid005422 (30/32), master nid005360 port 9999, GPUs 8, CUDA: True 12: Launching on nid005372 (12/32), master nid005360 port 9999, GPUs 8, CUDA: True 20: Launching on nid005412 (20/32), master nid005360 port 9999, GPUs 8, CUDA: True 21: Launching on nid005413 (21/32), master nid005360 port 9999, GPUs 8, CUDA: True 24: Launching on nid005416 (24/32), master nid005360 port 9999, GPUs 8, CUDA: True 3: Launching on nid005363 (3/32), master nid005360 port 9999, GPUs 8, CUDA: True 6: Launching on nid005366 (6/32), master nid005360 port 9999, GPUs 8, CUDA: True 0: using world size: 256, data-parallel-size: 256, tensor-model-parallel size: 1, pipeline-model-parallel size: 1 0: accumulate and all-reduce gradients in fp32 for bfloat16 data type. 0: using torch.bfloat16 for parameters ... 0: ------------------------ arguments ------------------------ 0: abort_on_unmet_fused_kernel_constraints ......... False 0: accumulate_allreduce_grads_in_fp32 .............. True 0: adam_beta1 ...................................... 0.9 0: adam_beta2 ...................................... 0.999 0: adam_eps ........................................ 1e-08 0: adlr_autoresume ................................. False 0: adlr_autoresume_interval ........................ 1000 0: apply_query_key_layer_scaling ................... True 0: apply_residual_connection_post_layernorm ........ False 0: attention_dropout ............................... 0.1 0: attention_softmax_in_fp32 ....................... False 0: bert_binary_head ................................ True 0: bert_load ....................................... None 0: bf16 ............................................ True 0: bias_dropout_fusion ............................. True 0: bias_gelu_fusion ................................ True 0: biencoder_projection_dim ........................ 0 0: biencoder_shared_query_context_model ............ False 0: block_data_path ................................. None 0: checkpoint_activations .......................... True 0: checkpoint_in_cpu ............................... False 0: checkpoint_num_layers ........................... 1 0: clip_grad ....................................... 1.0 0: codecarbon_dir .................................. None 0: consumed_train_samples .......................... 0 0: consumed_train_tokens ........................... 0 0: consumed_valid_samples .......................... 0 0: contigious_checkpointing ........................ False 0: cpu_optimizer ................................... False 0: cpu_torch_adam .................................. False 0: curriculum_learning ............................. False 0: data_impl ....................................... mmap 0: data_parallel_size .............................. 256 0: data_path ....................................... ['/scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document'] 0: dataloader_type ................................. single 0: DDP_impl ........................................ local 0: decoder_seq_length .............................. None 0: deepscale ....................................... False 0: deepscale_config ................................ None 0: deepspeed ....................................... True 0: deepspeed_activation_checkpointing .............. False 0: deepspeed_config ................................ ds_configs/2058110.json 0: deepspeed_mpi ................................... False 0: distribute_checkpointed_activations ............. False 0: distributed_backend ............................. nccl 0: embed_layernorm ................................. False 0: embedding_path .................................. None 0: encoder_seq_length .............................. 2048 0: eod_mask_loss ................................... False 0: eval_interval ................................... 1000 0: eval_iters ...................................... 1 0: eval_only ....................................... None 0: evidence_data_path .............................. None 0: exit_duration_in_mins ........................... None 0: exit_interval ................................... None 0: ffn_hidden_size ................................. 2560 0: finetune ........................................ False 0: fp16 ............................................ False 0: fp16_lm_cross_entropy ........................... False 0: fp32_residual_connection ........................ False 0: gigaflos_no_embeds .............................. 0 0: global_batch_size ............................... 256 0: glu_activation .................................. None 0: hidden_dropout .................................. 0.1 0: hidden_size ..................................... 640 0: hysteresis ...................................... 2 0: ict_head_size ................................... None 0: ict_load ........................................ None 0: img_dim ......................................... 224 0: indexer_batch_size .............................. 128 0: indexer_log_interval ............................ 1000 0: inference ....................................... False 0: init_method_std ................................. 0.02 0: init_method_xavier_uniform ...................... False 0: initial_loss_scale .............................. 4294967296 0: kill_switch_path ................................ kill-switch-1 0: kv_channels ..................................... 64 0: layer_norm_fusion ............................... True 0: layernorm_epsilon ............................... 1e-05 0: lazy_mpu_init ................................... None 0: load ............................................ checkpoints_83m 0: local_rank ...................................... None 0: log_batch_size_to_tensorboard ................... True 0: log_interval .................................... 10 0: log_learning_rate_to_tensorboard ................ True 0: log_level ....................................... None 0: log_level_replica ............................... None 0: log_loss_scale_to_tensorboard ................... True 0: log_num_zeros_in_grad ........................... False 0: log_params_norm ................................. False 0: log_path ........................................ None 0: log_timers_to_tensorboard ....................... True 0: log_validation_ppl_to_tensorboard ............... True 0: loss_on_targets_only ............................ False 0: loss_scale ...................................... 12.0 0: loss_scale_window ............................... 1000 0: lr .............................................. 0.0002 0: lr_decay_iters .................................. None 0: lr_decay_samples ................................ 9703701 0: lr_decay_style .................................. cosine 0: lr_decay_tokens ................................. None 0: lr_warmup_fraction .............................. None 0: lr_warmup_iters ................................. 0 0: lr_warmup_samples ............................... 0 0: make_vocab_size_divisible_by .................... 128 0: mask_prob ....................................... 0.15 0: masked_softmax_fusion ........................... True 0: max_position_embeddings ......................... 2048 0: mean_noise_span_length .......................... None 0: memory_centric_tiled_linear ..................... False 0: merge_file ...................................... gpt2/merges.txt 0: micro_batch_size ................................ 1 0: min_loss_scale .................................. 1.0 0: min_lr .......................................... 2e-05 0: mmap_warmup ..................................... False 0: no_load_optim ................................... None 0: no_load_rng ..................................... None 0: no_save_optim ................................... None 0: no_save_rng ..................................... None 0: noise_density ................................... None 0: num_attention_heads ............................. 10 0: num_channels .................................... 3 0: num_classes ..................................... 1000 0: num_layers ...................................... 10 0: num_layers_per_virtual_pipeline_stage ........... None 0: num_workers ..................................... 2 0: onnx_safe ....................................... None 0: openai_gelu ..................................... False 0: optimizer ....................................... adam 0: optimizer_fusion ................................ True 0: override_lr_scheduler ........................... False 0: pad_vocab_size_to ............................... None 0: params_dtype .................................... torch.bfloat16 0: partition_activations ........................... False 0: patch_dim ....................................... 16 0: pipeline_model_parallel_size .................... 1 0: position_embedding_type ......................... PositionEmbeddingType.absolute 0: pp_partition_method ............................. None 0: profile_backward ................................ False 0: query_in_block_prob ............................. 0.1 0: rampup_batch_size ............................... None 0: rank ............................................ 0 0: remote_device ................................... none 0: reset_attention_mask ............................ False 0: reset_position_ids .............................. False 0: retriever_report_topk_accuracies ................ [] 0: retriever_score_scaling ......................... False 0: retriever_seq_length ............................ 256 0: reweight_loss_based_on_position_frequency ....... False 0: sample_rate ..................................... 1.0 0: save ............................................ checkpoints_83m 0: save_interval ................................... 1000 0: scatter_gather_tensors_in_pipeline .............. True 0: scattered_embeddings ............................ False 0: seed ............................................ 1234 0: seq_length ...................................... 2048 0: sgd_momentum .................................... 0.9 0: short_seq_prob .................................. 0.1 0: skip_train_iteration_range ...................... None 0: split ........................................... 949,50,1 0: split_transformers .............................. False 0: sync_tp_duplicated_parameters ................... False 0: synchronize_each_layer .......................... False 0: tensor_model_parallel_size ...................... 1 0: tensorboard_dir ................................. tensorboard_83m 0: tensorboard_log_interval ........................ 1 0: tensorboard_queue_size .......................... 5 0: test_weighted_split_names ....................... None 0: test_weighted_split_paths ....................... None 0: test_weighted_split_paths_path .................. None 0: test_weighted_split_splits ...................... None 0: test_weighted_split_weights ..................... None 0: tile_factor ..................................... 1 0: titles_data_path ................................ None 0: tokenizer_name_or_path .......................... None 0: tokenizer_type .................................. GPT2BPETokenizer 0: train_iters ..................................... None 0: train_samples ................................... 9703701 0: train_tokens .................................... None 0: train_weighted_split_paths ...................... None 0: train_weighted_split_paths_path ................. None 0: universal_checkpoint ............................ False 0: use_bnb_optimizer ............................... False 0: use_checkpoint_lr_scheduler ..................... False 0: use_contiguous_buffers_in_ddp ................... True 0: use_cpu_initialization .......................... None 0: use_one_sent_docs ............................... False 0: use_pin_memory .................................. False 0: valid_num_workers ............................... 2 0: valid_weighted_split_names ...................... None 0: valid_weighted_split_paths ...................... None 0: valid_weighted_split_paths_path ................. None 0: valid_weighted_split_splits ..................... None 0: valid_weighted_split_weights .................... None 0: virtual_pipeline_model_parallel_size ............ None 0: vocab_extra_ids ................................. 0 0: vocab_file ...................................... gpt2/vocab.json 0: weight_decay .................................... 0.1 0: world_size ...................................... 256 0: zero_allgather_bucket_size ...................... 0.0 0: zero_contigious_gradients ....................... False 0: zero_reduce_bucket_size ......................... 0.0 0: zero_reduce_scatter ............................. False 0: zero_stage ...................................... 0 0: -------------------- end of arguments --------------------- 0: setting number of micro-batches to constant 1 0: > building GPT2BPETokenizer tokenizer ... 0: > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304) 0: DeepSpeed general environment info: 0: torch install path ............... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch'] 0: torch version .................... 1.13.0+rocm5.2 0: torch cuda version ............... None 0: torch hip version ................ 5.2.21151-afdc89f8 0: nvcc version ..................... None 0: deepspeed install path ........... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/deepspeed'] 0: deepspeed info ................... 0.7.5, unknown, unknown 0: deepspeed wheel compiled w. ...... torch 1.13, hip 5.1 0: **** Git info for Megatron: git_hash=unknown git_branch=unknown **** 0: > initializing torch distributed ... 0: [2022-11-24 16:47:37,578] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl 31: > setting tensorboard ... 0: > initializing tensor model parallel with size 1 0: > initializing pipeline model parallel with size 1 0: > setting random seeds to 1234 ... 0: > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234 0: > compiling dataset index builder ... 0: make: Entering directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data' 0: make: Nothing to be done for 'default'. 0: make: Leaving directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data' 0: >>> done with dataset index builder. Compilation time: 0.073 seconds 0: WARNING: constraints for invoking optimized fused softmax kernel are not met. We default back to unfused kernel invocations. 0: > compiling and loading fused kernels ... 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.cpp [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified] 0: Total number of unsupported CUDA function calls: 0 0: 0: 0: Total number of replaced kernel launches: 87 0: [1/1] c++ scaled_upper_triang_masked_softmax_hip.o scaled_upper_triang_masked_softmax_hip.cuda.o -shared -L/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/lib -lc10 -lc10_hip -ltorch_cpu -ltorch_hip -ltorch -ltorch_python -L/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/lib -lamdhip64 -o scaled_upper_triang_masked_softmax_cuda.so 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.cpp [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified] 0: Total number of unsupported CUDA function calls: 0 0: 0: 0: Total number of replaced kernel launches: 63 0: ninja: no work to do. 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda_kernel.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_hip_kernel.hip [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified] 0: Total number of unsupported CUDA function calls: 0 0: 0: 0: Total number of replaced kernel launches: 67 0: [1/1] c++ layer_norm_hip_kernel.cuda.o layer_norm_cuda.o -shared -L/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/lib -lc10 -lc10_hip -ltorch_cpu -ltorch_hip -ltorch -ltorch_python -L/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/lib -lamdhip64 -o fused_mix_prec_layer_norm_cuda.so 0: >>> done with compiling and loading fused kernels. Compilation time: 27.278 seconds 0: time to initialize megatron (seconds): 50.883 0: [after megatron is initialized] datetime: 2022-11-24 16:48:18 0: building GPT model ... 0: [2022-11-24 16:48:18,961] [INFO] [utils.py:827:see_memory_usage] Before Building Model 0: [2022-11-24 16:48:18,962] [INFO] [utils.py:828:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB 0: [2022-11-24 16:48:18,962] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.74 GB, percent = 6.3% 0: SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None 0: Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pi 0: pe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=4 0: 6, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63, ProcessCoord(pipe=0, data=64, model=0): 64, ProcessCoord(pipe=0, data=65, model=0): 65, ProcessCoord(pipe=0, data=66, model=0): 66, ProcessCoord(pipe=0, data=67, model=0): 67, ProcessCoord(pipe=0, data=68, model=0): 68, ProcessCoord(pipe=0, data=69, model=0): 0: 69, ProcessCoord(pipe=0, data=70, model=0): 70, ProcessCoord(pipe=0, data=71, model=0): 71, ProcessCoord(pipe=0, data=72, model=0): 72, ProcessCoord(pipe=0, data=73, model=0): 73, ProcessCoord(pipe=0, data=74, model=0): 74, ProcessCoord(pipe=0, data=75, model=0): 75, ProcessCoord(pipe=0, data=76, model=0): 76, ProcessCoord(pipe=0, data=77, model=0): 77, ProcessCoord(pipe=0, data=78, model=0): 78, ProcessCoord(pipe=0, data=79, model=0): 79, ProcessCoord(pipe=0, data=80, model=0): 80, ProcessCoord(pipe=0, data=81, model=0): 81, ProcessCoord(pipe=0, data=82, model=0): 82, ProcessCoord(pipe=0, data=83, model=0): 83, ProcessCoord(pipe=0, data=84, model=0): 84, ProcessCoord(pipe=0, data=85, model=0): 85, ProcessCoord(pipe=0, data=86, model=0): 86, ProcessCoord(pipe=0, data=87, model=0): 87, ProcessCoord(pipe=0, data=88, model=0): 88, ProcessCoord(pipe=0, data=89, model=0): 89, ProcessCoord(pipe=0, data=90, model=0): 90, ProcessCoord(pipe=0, data=91, model=0): 91, ProcessCoord(pipe=0, data=92, model=0): 92, Process 0: Coord(pipe=0, data=93, model=0): 93, ProcessCoord(pipe=0, data=94, model=0): 94, ProcessCoord(pipe=0, data=95, model=0): 95, ProcessCoord(pipe=0, data=96, model=0): 96, ProcessCoord(pipe=0, data=97, model=0): 97, ProcessCoord(pipe=0, data=98, model=0): 98, ProcessCoord(pipe=0, data=99, model=0): 99, ProcessCoord(pipe=0, data=100, model=0): 100, ProcessCoord(pipe=0, data=101, model=0): 101, ProcessCoord(pipe=0, data=102, model=0): 102, ProcessCoord(pipe=0, data=103, model=0): 103, ProcessCoord(pipe=0, data=104, model=0): 104, ProcessCoord(pipe=0, data=105, model=0): 105, ProcessCoord(pipe=0, data=106, model=0): 106, ProcessCoord(pipe=0, data=107, model=0): 107, ProcessCoord(pipe=0, data=108, model=0): 108, ProcessCoord(pipe=0, data=109, model=0): 109, ProcessCoord(pipe=0, data=110, model=0): 110, ProcessCoord(pipe=0, data=111, model=0): 111, ProcessCoord(pipe=0, data=112, model=0): 112, ProcessCoord(pipe=0, data=113, model=0): 113, ProcessCoord(pipe=0, data=114, model=0): 114, ProcessCoord(pipe=0, data=115, mo 0: del=0): 115, ProcessCoord(pipe=0, data=116, model=0): 116, ProcessCoord(pipe=0, data=117, model=0): 117, ProcessCoord(pipe=0, data=118, model=0): 118, ProcessCoord(pipe=0, data=119, model=0): 119, ProcessCoord(pipe=0, data=120, model=0): 120, ProcessCoord(pipe=0, data=121, model=0): 121, ProcessCoord(pipe=0, data=122, model=0): 122, ProcessCoord(pipe=0, data=123, model=0): 123, ProcessCoord(pipe=0, data=124, model=0): 124, ProcessCoord(pipe=0, data=125, model=0): 125, ProcessCoord(pipe=0, data=126, model=0): 126, ProcessCoord(pipe=0, data=127, model=0): 127, ProcessCoord(pipe=0, data=128, model=0): 128, ProcessCoord(pipe=0, data=129, model=0): 129, ProcessCoord(pipe=0, data=130, model=0): 130, ProcessCoord(pipe=0, data=131, model=0): 131, ProcessCoord(pipe=0, data=132, model=0): 132, ProcessCoord(pipe=0, data=133, model=0): 133, ProcessCoord(pipe=0, data=134, model=0): 134, ProcessCoord(pipe=0, data=135, model=0): 135, ProcessCoord(pipe=0, data=136, model=0): 136, ProcessCoord(pipe=0, data=137, model=0): 137, 0: ProcessCoord(pipe=0, data=138, model=0): 138, ProcessCoord(pipe=0, data=139, model=0): 139, ProcessCoord(pipe=0, data=140, model=0): 140, ProcessCoord(pipe=0, data=141, model=0): 141, ProcessCoord(pipe=0, data=142, model=0): 142, ProcessCoord(pipe=0, data=143, model=0): 143, ProcessCoord(pipe=0, data=144, model=0): 144, ProcessCoord(pipe=0, data=145, model=0): 145, ProcessCoord(pipe=0, data=146, model=0): 146, ProcessCoord(pipe=0, data=147, model=0): 147, ProcessCoord(pipe=0, data=148, model=0): 148, ProcessCoord(pipe=0, data=149, model=0): 149, ProcessCoord(pipe=0, data=150, model=0): 150, ProcessCoord(pipe=0, data=151, model=0): 151, ProcessCoord(pipe=0, data=152, model=0): 152, ProcessCoord(pipe=0, data=153, model=0): 153, ProcessCoord(pipe=0, data=154, model=0): 154, ProcessCoord(pipe=0, data=155, model=0): 155, ProcessCoord(pipe=0, data=156, model=0): 156, ProcessCoord(pipe=0, data=157, model=0): 157, ProcessCoord(pipe=0, data=158, model=0): 158, ProcessCoord(pipe=0, data=159, model=0): 159, ProcessCoor 0: d(pipe=0, data=160, model=0): 160, ProcessCoord(pipe=0, data=161, model=0): 161, ProcessCoord(pipe=0, data=162, model=0): 162, ProcessCoord(pipe=0, data=163, model=0): 163, ProcessCoord(pipe=0, data=164, model=0): 164, ProcessCoord(pipe=0, data=165, model=0): 165, ProcessCoord(pipe=0, data=166, model=0): 166, ProcessCoord(pipe=0, data=167, model=0): 167, ProcessCoord(pipe=0, data=168, model=0): 168, ProcessCoord(pipe=0, data=169, model=0): 169, ProcessCoord(pipe=0, data=170, model=0): 170, ProcessCoord(pipe=0, data=171, model=0): 171, ProcessCoord(pipe=0, data=172, model=0): 172, ProcessCoord(pipe=0, data=173, model=0): 173, ProcessCoord(pipe=0, data=174, model=0): 174, ProcessCoord(pipe=0, data=175, model=0): 175, ProcessCoord(pipe=0, data=176, model=0): 176, ProcessCoord(pipe=0, data=177, model=0): 177, ProcessCoord(pipe=0, data=178, model=0): 178, ProcessCoord(pipe=0, data=179, model=0): 179, ProcessCoord(pipe=0, data=180, model=0): 180, ProcessCoord(pipe=0, data=181, model=0): 181, ProcessCoord(pipe=0, da 0: ta=182, model=0): 182, ProcessCoord(pipe=0, data=183, model=0): 183, ProcessCoord(pipe=0, data=184, model=0): 184, ProcessCoord(pipe=0, data=185, model=0): 185, ProcessCoord(pipe=0, data=186, model=0): 186, ProcessCoord(pipe=0, data=187, model=0): 187, ProcessCoord(pipe=0, data=188, model=0): 188, ProcessCoord(pipe=0, data=189, model=0): 189, ProcessCoord(pipe=0, data=190, model=0): 190, ProcessCoord(pipe=0, data=191, model=0): 191, ProcessCoord(pipe=0, data=192, model=0): 192, ProcessCoord(pipe=0, data=193, model=0): 193, ProcessCoord(pipe=0, data=194, model=0): 194, ProcessCoord(pipe=0, data=195, model=0): 195, ProcessCoord(pipe=0, data=196, model=0): 196, ProcessCoord(pipe=0, data=197, model=0): 197, ProcessCoord(pipe=0, data=198, model=0): 198, ProcessCoord(pipe=0, data=199, model=0): 199, ProcessCoord(pipe=0, data=200, model=0): 200, ProcessCoord(pipe=0, data=201, model=0): 201, ProcessCoord(pipe=0, data=202, model=0): 202, ProcessCoord(pipe=0, data=203, model=0): 203, ProcessCoord(pipe=0, data=204, mode 0: l=0): 204, ProcessCoord(pipe=0, data=205, model=0): 205, ProcessCoord(pipe=0, data=206, model=0): 206, ProcessCoord(pipe=0, data=207, model=0): 207, ProcessCoord(pipe=0, data=208, model=0): 208, ProcessCoord(pipe=0, data=209, model=0): 209, ProcessCoord(pipe=0, data=210, model=0): 210, ProcessCoord(pipe=0, data=211, model=0): 211, ProcessCoord(pipe=0, data=212, model=0): 212, ProcessCoord(pipe=0, data=213, model=0): 213, ProcessCoord(pipe=0, data=214, model=0): 214, ProcessCoord(pipe=0, data=215, model=0): 215, ProcessCoord(pipe=0, data=216, model=0): 216, ProcessCoord(pipe=0, data=217, model=0): 217, ProcessCoord(pipe=0, data=218, model=0): 218, ProcessCoord(pipe=0, data=219, model=0): 219, ProcessCoord(pipe=0, data=220, model=0): 220, ProcessCoord(pipe=0, data=221, model=0): 221, ProcessCoord(pipe=0, data=222, model=0): 222, ProcessCoord(pipe=0, data=223, model=0): 223, ProcessCoord(pipe=0, data=224, model=0): 224, ProcessCoord(pipe=0, data=225, model=0): 225, ProcessCoord(pipe=0, data=226, model=0): 226, P 0: rocessCoord(pipe=0, data=227, model=0): 227, ProcessCoord(pipe=0, data=228, model=0): 228, ProcessCoord(pipe=0, data=229, model=0): 229, ProcessCoord(pipe=0, data=230, model=0): 230, ProcessCoord(pipe=0, data=231, model=0): 231, ProcessCoord(pipe=0, data=232, model=0): 232, ProcessCoord(pipe=0, data=233, model=0): 233, ProcessCoord(pipe=0, data=234, model=0): 234, ProcessCoord(pipe=0, data=235, model=0): 235, ProcessCoord(pipe=0, data=236, model=0): 236, ProcessCoord(pipe=0, data=237, model=0): 237, ProcessCoord(pipe=0, data=238, model=0): 238, ProcessCoord(pipe=0, data=239, model=0): 239, ProcessCoord(pipe=0, data=240, model=0): 240, ProcessCoord(pipe=0, data=241, model=0): 241, ProcessCoord(pipe=0, data=242, model=0): 242, ProcessCoord(pipe=0, data=243, model=0): 243, ProcessCoord(pipe=0, data=244, model=0): 244, ProcessCoord(pipe=0, data=245, model=0): 245, ProcessCoord(pipe=0, data=246, model=0): 246, ProcessCoord(pipe=0, data=247, model=0): 247, ProcessCoord(pipe=0, data=248, model=0): 248, ProcessCoord( 0: pipe=0, data=249, model=0): 249, ProcessCoord(pipe=0, data=250, model=0): 250, ProcessCoord(pipe=0, data=251, model=0): 251, ProcessCoord(pipe=0, data=252, model=0): 252, ProcessCoord(pipe=0, data=253, model=0): 253, ProcessCoord(pipe=0, data=254, model=0): 254, ProcessCoord(pipe=0, data=255, model=0): 255} 0: [2022-11-24 16:48:27,708] [INFO] [module.py:366:_partition_layers] Partitioning pipeline stages with method type:transformer 0: stage=0 layers=17 0: 0: _to_float16 0: 1: EmbeddingPipe 0: 2: 0: 3: ParallelTransformerLayerPipe 0: 4: ParallelTransformerLayerPipe 0: 5: ParallelTransformerLayerPipe 0: 6: ParallelTransformerLayerPipe 0: 7: ParallelTransformerLayerPipe 0: 8: ParallelTransformerLayerPipe 0: 9: ParallelTransformerLayerPipe 0: 10: ParallelTransformerLayerPipe 0: 11: ParallelTransformerLayerPipe 0: 12: ParallelTransformerLayerPipe 0: 13: undo 0: 14: MixedFusedLayerNorm 0: 15: EmbeddingPipe 0: 16: float16_to_fp32 0: loss: CrossEntropy 0: [2022-11-24 16:48:28,049] [INFO] [utils.py:827:see_memory_usage] After Building Model 0: [2022-11-24 16:48:28,049] [INFO] [utils.py:828:see_memory_usage] MA 0.16 GB Max_MA 0.16 GB CA 0.17 GB Max_CA 0 GB 0: [2022-11-24 16:48:28,049] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.75 GB, percent = 6.3% 0: setting training iterations to 37905 0: > learning rate decay style: cosine 0: DeepSpeed is enabled. 0: [2022-11-24 16:48:28,051] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.5, git-hash=unknown, git-branch=unknown 0: [2022-11-24 16:48:46,668] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False 0: [2022-11-24 16:48:46,668] [INFO] [logging.py:68:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer 0: [2022-11-24 16:48:46,668] [INFO] [logging.py:68:log_dist] [Rank 0] Using client Optimizer as basic optimizer 0: [2022-11-24 16:48:46,671] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam 0: [2022-11-24 16:48:46,671] [INFO] [logging.py:68:log_dist] [Rank 0] Creating BF16 optimizer 0: [2022-11-24 16:48:46,714] [INFO] [utils.py:827:see_memory_usage] begin bf16_optimizer 0: [2022-11-24 16:48:46,714] [INFO] [utils.py:828:see_memory_usage] MA 0.16 GB Max_MA 0.16 GB CA 0.17 GB Max_CA 0 GB 0: [2022-11-24 16:48:46,714] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.45 GB, percent = 6.4% 4: ninja: no work to do. 4: Time to load utils op: 0.17708468437194824 seconds 0: ninja: no work to do. 3: Time to load utils op: 0.11601424217224121 seconds 3: Time to load utils op: 0.11609864234924316 seconds 3: Time to load utils op: 0.11621832847595215 secondsTime to load utils op: 0.11625456809997559 seconds 3: Time to load utils op: 0.1162266731262207 seconds 3: 3: Time to load utils op: 0.11626839637756348 seconds 3: Time to load utils op: 0.11626553535461426 secondsTime to load utils op: 0.11624693870544434 seconds 3: 7: Time to load utils op: 0.12700867652893066 seconds 7: Time to load utils op: 0.12777209281921387 seconds 7: Time to load utils op: 0.12772297859191895 seconds 7: Time to load utils op: 0.127913236618042 secondsTime to load utils op: 0.12756681442260742 seconds 7: 7: Time to load utils op: 0.3271341323852539 secondsTime to load utils op: 0.12742066383361816 seconds 7: 0: Time to load utils op: 0.15517687797546387 secondsTime to load utils op: 0.3155193328857422 seconds 0: 8: Time to load utils op: 0.14496707916259766 seconds 8: Time to load utils op: 0.14435553550720215 seconds 8: Time to load utils op: 0.14549040794372559 seconds 8: Time to load utils op: 0.14443707466125488 seconds 8: Time to load utils op: 0.14553284645080566 secondsTime to load utils op: 0.1457219123840332 secondsTime to load utils op: 0.14519882202148438 secondsTime to load utils op: 0.14483428001403809 seconds 8: 8: 8: 6: Time to load utils op: 0.14590668678283691 secondsTime to load utils op: 0.14591073989868164 seconds 6: 6: Time to load utils op: 0.14593815803527832 seconds 6: Time to load utils op: 0.14595603942871094 seconds 6: Time to load utils op: 0.1459643840789795 secondsTime to load utils op: 0.145965576171875 secondsTime to load utils op: 0.14595961570739746 seconds 6: 6: 6: Time to load utils op: 0.14600419998168945 seconds 5: Time to load utils op: 0.14726996421813965 seconds 5: Time to load utils op: 0.1473560333251953 seconds 5: Time to load utils op: 0.14739084243774414 secondsTime to load utils op: 0.14739131927490234 seconds 5: Time to load utils op: 0.14742493629455566 seconds 5: 5: Time to load utils op: 0.14744186401367188 seconds 5: Time to load utils op: 0.14751839637756348 secondsTime to load utils op: 0.14750051498413086 seconds 5: 16: Time to load utils op: 0.13729500770568848 secondsTime to load utils op: 0.1372973918914795 seconds 16: 16: Time to load utils op: 0.13730716705322266 seconds 16: Time to load utils op: 0.13731718063354492 seconds 16: Time to load utils op: 0.1373615264892578 seconds 16: Time to load utils op: 0.13737010955810547 seconds 16: Time to load utils op: 0.13736939430236816 seconds 16: Time to load utils op: 0.1374502182006836 seconds 15: Time to load utils op: 0.1429305076599121 seconds 15: Time to load utils op: 0.14311552047729492 secondsTime to load utils op: 0.14341402053833008 seconds 15: 15: Time to load utils op: 0.1434311866760254 seconds 15: Time to load utils op: 0.3444797992706299 seconds 15: Time to load utils op: 0.1437084674835205 seconds 15: Time to load utils op: 0.14346814155578613 secondsTime to load utils op: 0.1438133716583252 seconds 15: 19: Time to load utils op: 0.13953089714050293 secondsTime to load utils op: 0.1404881477355957 secondsTime to load utils op: 0.1403672695159912 seconds 19: 19: 19: Time to load utils op: 0.14014077186584473 seconds 19: Time to load utils op: 0.14109420776367188 secondsTime to load utils op: 0.1396772861480713 seconds 19: 19: Time to load utils op: 0.14025545120239258 seconds 19: Time to load utils op: 0.14082789421081543 seconds 17: Time to load utils op: 0.13805294036865234 seconds 21: Time to load utils op: 0.13415122032165527 seconds 17: Time to load utils op: 0.13831496238708496 secondsTime to load utils op: 0.138322114944458 seconds 17: 17: Time to load utils op: 0.13832950592041016 seconds 17: Time to load utils op: 0.13834071159362793 seconds 21: Time to load utils op: 0.13420438766479492 seconds 17: Time to load utils op: 0.13834881782531738 seconds 21: Time to load utils op: 0.13423395156860352 seconds 17: Time to load utils op: 0.13846492767333984 seconds 30: Time to load utils op: 0.12216401100158691 secondsTime to load utils op: 0.12218475341796875 seconds 30: 30: Time to load utils op: 0.12219834327697754 secondsTime to load utils op: 0.12221026420593262 seconds 30: Time to load utils op: 0.12221336364746094 seconds 30: 21: Time to load utils op: 0.13427352905273438 seconds 21: Time to load utils op: 0.13429832458496094 seconds 17: Time to load utils op: 0.13850975036621094 seconds 21: Time to load utils op: 0.1343522071838379 secondsTime to load utils op: 0.13437390327453613 seconds 21: 30: Time to load utils op: 0.12223291397094727 secondsTime to load utils op: 0.12221789360046387 seconds 30: 21: Time to load utils op: 0.13450980186462402 seconds 30: Time to load utils op: 0.12225770950317383 seconds 10: Time to load utils op: 0.1536097526550293 secondsTime to load utils op: 0.15436029434204102 seconds 10: 24: Time to load utils op: 0.13448357582092285 seconds 24: Time to load utils op: 0.13433313369750977 seconds 24: Time to load utils op: 0.1347808837890625 seconds 10: Time to load utils op: 0.15385198593139648 seconds 24: Time to load utils op: 0.13492512702941895 seconds 10: Time to load utils op: 0.1535172462463379 seconds 10: Time to load utils op: 0.15333342552185059 secondsTime to load utils op: 0.1542823314666748 seconds 10: 24: Time to load utils op: 0.13518786430358887 seconds 24: Time to load utils op: 0.13525962829589844 secondsTime to load utils op: 0.13480138778686523 seconds 24: 10: Time to load utils op: 0.1548328399658203 seconds 10: Time to load utils op: 0.1542038917541504 seconds 12: Time to load utils op: 0.14644289016723633 seconds 12: Time to load utils op: 0.14647507667541504 seconds 12: Time to load utils op: 0.14647293090820312 seconds 24: Time to load utils op: 0.33760643005371094 seconds 12: Time to load utils op: 0.14649510383605957 secondsTime to load utils op: 0.14652419090270996 seconds 12: 12: Time to load utils op: 0.14653253555297852 seconds 12: Time to load utils op: 0.14655017852783203 secondsTime to load utils op: 0.14653730392456055 seconds 12: 18: Time to load utils op: 0.14335298538208008 seconds 23: Time to load utils op: 0.13634371757507324 secondsTime to load utils op: 0.13624191284179688 seconds 23: 18: Time to load utils op: 0.14327621459960938 seconds 18: Time to load utils op: 0.14335989952087402 seconds 18: Time to load utils op: 0.14354848861694336 seconds 18: Time to load utils op: 0.3456759452819824 secondsTime to load utils op: 0.14377093315124512 seconds 18: 18: Time to load utils op: 0.14340615272521973 seconds 18: Time to load utils op: 0.14334821701049805 seconds 23: Time to load utils op: 0.13701438903808594 secondsTime to load utils op: 0.1370222568511963 seconds 23: 23: Time to load utils op: 0.13665485382080078 secondsTime to load utils op: 0.13657522201538086 seconds 23: 13: Time to load utils op: 0.1468040943145752 seconds 23: Time to load utils op: 0.13694286346435547 seconds 23: Time to load utils op: 0.3393666744232178 seconds 13: Time to load utils op: 0.14685726165771484 seconds 13: Time to load utils op: 0.14688849449157715 seconds 13: Time to load utils op: 0.14688515663146973 secondsTime to load utils op: 0.14688539505004883 seconds 13: 13: Time to load utils op: 0.14690351486206055 secondsTime to load utils op: 0.14693307876586914 secondsTime to load utils op: 0.14690041542053223 seconds 13: 13: 11: Time to load utils op: 0.153411865234375 secondsTime to load utils op: 0.15296483039855957 seconds 11: 11: Time to load utils op: 0.1534419059753418 seconds 11: Time to load utils op: 0.15268874168395996 seconds 11: Time to load utils op: 0.15269923210144043 secondsTime to load utils op: 0.15274572372436523 secondsTime to load utils op: 0.15274620056152344 seconds 11: 11: 11: Time to load utils op: 0.15272760391235352 seconds 20: Time to load utils op: 0.13745641708374023 seconds 20: Time to load utils op: 0.1374659538269043 seconds 20: Time to load utils op: 0.137481689453125 seconds 20: Time to load utils op: 0.13753461837768555 seconds 20: Time to load utils op: 0.13750123977661133 secondsTime to load utils op: 0.1375572681427002 seconds 20: 20: Time to load utils op: 0.13755583763122559 seconds 20: Time to load utils op: 0.13753771781921387 seconds 14: Time to load utils op: 0.15096664428710938 secondsTime to load utils op: 0.15097522735595703 seconds 14: 14: Time to load utils op: 0.15092039108276367 seconds 14: Time to load utils op: 0.15152335166931152 secondsTime to load utils op: 0.15031909942626953 seconds 14: 22: Time to load utils op: 0.1361558437347412 seconds 14: Time to load utils op: 0.15049171447753906 seconds 26: Time to load utils op: 0.13489651679992676 seconds 14: Time to load utils op: 0.1504826545715332 seconds 14: Time to load utils op: 0.15019774436950684 seconds 22: Time to load utils op: 0.1361548900604248 seconds 22: Time to load utils op: 0.13616156578063965 seconds 22: Time to load utils op: 0.1362152099609375 seconds 22: Time to load utils op: 0.13622760772705078 secondsTime to load utils op: 0.13623046875 seconds 22: 22: Time to load utils op: 0.13623332977294922 secondsTime to load utils op: 0.1362297534942627 seconds 22: 26: Time to load utils op: 0.13396310806274414 seconds 26: Time to load utils op: 0.13511180877685547 secondsTime to load utils op: 0.1347517967224121 seconds 26: 27: Time to load utils op: 0.12834429740905762 seconds 27: Time to load utils op: 0.12833070755004883 seconds 26: Time to load utils op: 0.13502764701843262 seconds 26: Time to load utils op: 0.13512611389160156 seconds 26: Time to load utils op: 0.1341254711151123 secondsTime to load utils op: 0.13462090492248535 seconds 26: 27: Time to load utils op: 0.1284017562866211 seconds 27: Time to load utils op: 0.12842416763305664 seconds 31: Time to load utils op: 0.12362456321716309 seconds 27: Time to load utils op: 0.12842416763305664 secondsTime to load utils op: 0.12843608856201172 seconds 27: 27: Time to load utils op: 0.12841320037841797 seconds 27: Time to load utils op: 0.12847304344177246 seconds 31: Time to load utils op: 0.12364721298217773 seconds 31: Time to load utils op: 0.12368392944335938 seconds 31: Time to load utils op: 0.12370085716247559 secondsTime to load utils op: 0.12369012832641602 seconds 31: 31: Time to load utils op: 0.1237022876739502 seconds 31: Time to load utils op: 0.12372922897338867 seconds 31: Time to load utils op: 0.12371683120727539 seconds 29: Time to load utils op: 0.12737131118774414 secondsTime to load utils op: 0.12736201286315918 secondsTime to load utils op: 0.12734675407409668 secondsTime to load utils op: 0.1273491382598877 seconds 29: 29: 29: 29: Time to load utils op: 0.12736272811889648 seconds 29: Time to load utils op: 0.127485990524292 secondsTime to load utils op: 0.1275181770324707 seconds 29: 29: Time to load utils op: 0.12752676010131836 seconds 25: Time to load utils op: 0.13324618339538574 seconds 25: Time to load utils op: 0.13322782516479492 seconds 25: Time to load utils op: 0.13324642181396484 seconds 25: Time to load utils op: 0.13326811790466309 secondsTime to load utils op: 0.1332552433013916 seconds 25: Time to load utils op: 0.1332552433013916 seconds 25: Time to load utils op: 0.13326454162597656 seconds 25: 25: Time to load utils op: 0.1332705020904541 seconds 9: Time to load utils op: 0.15452337265014648 secondsTime to load utils op: 0.1545546054840088 secondsTime to load utils op: 0.1545543670654297 seconds 9: 9: 9: Time to load utils op: 0.15456819534301758 seconds 9: Time to load utils op: 0.15456318855285645 seconds 28: Time to load utils op: 0.12966537475585938 seconds 28: Time to load utils op: 0.12964653968811035 seconds 28: Time to load utils op: 0.12966084480285645 seconds 9: Time to load utils op: 0.15457701683044434 seconds 9: Time to load utils op: 0.1545724868774414 seconds 9: Time to load utils op: 0.15459895133972168 seconds 28: Time to load utils op: 0.12967300415039062 seconds 28: Time to load utils op: 0.12972545623779297 seconds 28: Time to load utils op: 0.12970399856567383 secondsTime to load utils op: 0.12970614433288574 seconds 28: 28: Time to load utils op: 0.12971973419189453 seconds 0: [2022-11-24 16:48:47,072] [INFO] [utils.py:827:see_memory_usage] before initializing group 0 0: [2022-11-24 16:48:47,073] [INFO] [utils.py:828:see_memory_usage] MA 0.16 GB Max_MA 0.16 GB CA 0.17 GB Max_CA 0 GB 0: [2022-11-24 16:48:47,073] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.46 GB, percent = 6.4% 0: Time to load utils op: 0.20273232460021973 seconds 0: Time to load utils op: 0.20279455184936523 seconds 0: Time to load utils op: 0.2019975185394287 seconds 0: Time to load utils op: 0.20264458656311035 seconds 0: Time to load utils op: 0.20212078094482422 seconds 0: Time to load utils op: 0.20218539237976074 seconds 4: Time to load utils op: 0.20360946655273438 seconds 4: Time to load utils op: 0.20368170738220215 seconds 4: Time to load utils op: 0.20443964004516602 seconds 4: Time to load utils op: 0.20513439178466797 seconds 4: Time to load utils op: 0.20534348487854004 seconds 4: Time to load utils op: 0.20382976531982422 seconds 4: Time to load utils op: 0.20466899871826172 seconds 7: Time to load utils op: 0.2027416229248047 seconds 3: Time to load utils op: 0.0006225109100341797 seconds 3: Time to load utils op: 0.0006654262542724609 seconds 3: Time to load utils op: 0.00048160552978515625 seconds 3: Time to load utils op: 0.0005125999450683594 seconds 3: Time to load utils op: 0.00038313865661621094 seconds 3: Time to load utils op: 0.00046944618225097656 seconds 3: Time to load utils op: 0.0005297660827636719 seconds 3: Time to load utils op: 0.0005877017974853516 seconds 1: Time to load utils op: 0.21785259246826172 seconds 1: Time to load utils op: 0.2180500030517578 seconds 1: Time to load utils op: 0.21805572509765625 seconds 1: Time to load utils op: 0.21825242042541504 seconds 1: Time to load utils op: 0.21827125549316406 seconds 1: Time to load utils op: 0.21838760375976562 seconds 1: Time to load utils op: 0.21842694282531738 secondsTime to load utils op: 0.21840143203735352 seconds 1: 2: Time to load utils op: 0.21813321113586426 seconds 2: Time to load utils op: 0.21816205978393555 seconds 2: Time to load utils op: 0.2181856632232666 seconds 2: Time to load utils op: 0.21822118759155273 seconds 2: Time to load utils op: 0.2183074951171875 seconds 2: Time to load utils op: 0.21835041046142578 seconds 2: Time to load utils op: 0.21836137771606445 seconds 2: Time to load utils op: 0.21844220161437988 seconds 24: Time to load utils op: 0.0005009174346923828 seconds 6: Time to load utils op: 0.0007183551788330078 seconds 6: Time to load utils op: 0.0008821487426757812 seconds 6: Time to load utils op: 0.0009953975677490234 seconds 6: Time to load utils op: 0.0009686946868896484 seconds 6: Time to load utils op: 0.0009870529174804688 seconds 6: Time to load utils op: 0.001033782958984375 seconds 6: Time to load utils op: 0.0007448196411132812 seconds 6: Time to load utils op: 0.0010640621185302734 seconds 0: Time to load utils op: 0.00044655799865722656 seconds 0: Time to load utils op: 0.0004749298095703125 seconds 0: Time to load utils op: 0.000446319580078125 seconds 0: Time to load utils op: 0.0004405975341796875 seconds 0: Time to load utils op: 0.00044226646423339844 seconds 0: Time to load utils op: 0.0004475116729736328 seconds 0: Time to load utils op: 0.0005106925964355469 seconds 18: Time to load utils op: 0.00075531005859375 seconds 18: Time to load utils op: 0.0005426406860351562 seconds 18: Time to load utils op: 0.0005679130554199219 seconds 18: Time to load utils op: 0.0008511543273925781 seconds 18: Time to load utils op: 0.0005943775177001953 secondsTime to load utils op: 0.0005137920379638672 seconds 18: 18: Time to load utils op: 0.0005307197570800781 seconds 18: Time to load utils op: 0.0008978843688964844 seconds 5: Time to load utils op: 0.0008461475372314453 seconds 30: Time to load utils op: 0.0006594657897949219 seconds 5: Time to load utils op: 0.000995635986328125 seconds 5: Time to load utils op: 0.001058340072631836 seconds 5: Time to load utils op: 0.001195669174194336 secondsTime to load utils op: 0.0011115074157714844 seconds 5: 30: Time to load utils op: 0.0005331039428710938 seconds 5: Time to load utils op: 0.001251220703125 seconds 5: Time to load utils op: 0.0012450218200683594 seconds 5: Time to load utils op: 0.0013458728790283203 seconds 30: Time to load utils op: 0.0006711483001708984 secondsTime to load utils op: 0.0004971027374267578 seconds 30: Time to load utils op: 0.00041031837463378906 seconds 30: Time to load utils op: 0.00039887428283691406 seconds 30: 30: Time to load utils op: 0.0003838539123535156 seconds 30: Time to load utils op: 0.0003426074981689453 seconds 19: Time to load utils op: 0.0008356571197509766 seconds 14: Time to load utils op: 0.0006852149963378906 seconds 14: Time to load utils op: 0.0006647109985351562 seconds 14: Time to load utils op: 0.0003542900085449219 seconds 19: Time to load utils op: 0.0013628005981445312 seconds 19: Time to load utils op: 0.0013082027435302734 seconds 19: Time to load utils op: 0.0013000965118408203 seconds 14: Time to load utils op: 0.00034308433532714844 seconds 19: Time to load utils op: 0.0013349056243896484 secondsTime to load utils op: 0.0013065338134765625 seconds 19: 19: Time to load utils op: 0.0013110637664794922 seconds 14: Time to load utils op: 0.0003590583801269531 seconds 14: Time to load utils op: 0.00041222572326660156 secondsTime to load utils op: 0.00041484832763671875 seconds 14: 19: Time to load utils op: 0.0013556480407714844 seconds 14: Time to load utils op: 0.00044465065002441406 seconds 20: Time to load utils op: 0.0006361007690429688 seconds 20: Time to load utils op: 0.0006611347198486328 seconds 20: Time to load utils op: 0.0004029273986816406 seconds 20: Time to load utils op: 0.0004608631134033203 seconds 20: Time to load utils op: 0.000644683837890625 seconds 20: Time to load utils op: 0.0007281303405761719 seconds 20: Time to load utils op: 0.0007371902465820312 seconds 2: Time to load utils op: 0.0010793209075927734 seconds 2: Time to load utils op: 0.0013005733489990234 seconds 20: Time to load utils op: 0.0007424354553222656 seconds 31: Time to load utils op: 0.0011219978332519531 seconds 2: Time to load utils op: 0.0012559890747070312 seconds 2: Time to load utils op: 0.0014624595642089844 seconds 2: Time to load utils op: 0.0014941692352294922 seconds 2: Time to load utils op: 0.001378774642944336 seconds 28: Time to load utils op: 0.0007712841033935547 seconds 2: Time to load utils op: 0.0013756752014160156 seconds 2: Time to load utils op: 0.001329183578491211 seconds 17: Time to load utils op: 0.0006861686706542969 seconds 28: Time to load utils op: 0.0008535385131835938 seconds 31: Time to load utils op: 0.0012233257293701172 seconds 31: Time to load utils op: 0.0013387203216552734 seconds 28: Time to load utils op: 0.00087738037109375 seconds 31: Time to load utils op: 0.0013499259948730469 seconds 29: Time to load utils op: 0.0006382465362548828 seconds 31: Time to load utils op: 0.0013713836669921875 seconds 15: Time to load utils op: 0.0009982585906982422 seconds 17: Time to load utils op: 0.00058746337890625 seconds 29: Time to load utils op: 0.0007512569427490234 secondsTime to load utils op: 0.0004987716674804688 seconds 29: 31: Time to load utils op: 0.0013813972473144531 seconds 31: Time to load utils op: 0.0011987686157226562 seconds 31: Time to load utils op: 0.0013689994812011719 seconds 28: Time to load utils op: 0.000934600830078125 secondsTime to load utils op: 0.0010402202606201172 secondsTime to load utils op: 0.0008528232574462891 seconds 28: 28: 28: Time to load utils op: 0.001102447509765625 seconds 29: Time to load utils op: 0.0006892681121826172 seconds 29: Time to load utils op: 0.0006597042083740234 seconds 29: Time to load utils op: 0.0008661746978759766 secondsTime to load utils op: 0.0007770061492919922 seconds 29: 17: Time to load utils op: 0.00037288665771484375 seconds 28: Time to load utils op: 0.0010721683502197266 seconds 29: Time to load utils op: 0.0009703636169433594 seconds 15: Time to load utils op: 0.0012845993041992188 seconds 15: Time to load utils op: 0.001226663589477539 secondsTime to load utils op: 0.0013134479522705078 secondsTime to load utils op: 0.0012278556823730469 seconds 15: 15: 17: Time to load utils op: 0.0006208419799804688 secondsTime to load utils op: 0.00040841102600097656 seconds 17: Time to load utils op: 0.00048089027404785156 seconds 17: 15: Time to load utils op: 0.0012292861938476562 seconds 15: Time to load utils op: 0.0012519359588623047 seconds 17: Time to load utils op: 0.0005898475646972656 seconds 15: Time to load utils op: 0.0012640953063964844 seconds 17: Time to load utils op: 0.0004405975341796875 seconds 10: Time to load utils op: 0.0006566047668457031 seconds 10: Time to load utils op: 0.0006957054138183594 seconds 10: Time to load utils op: 0.00045418739318847656 seconds 10: Time to load utils op: 0.0004553794860839844 seconds 10: Time to load utils op: 0.0005128383636474609 secondsTime to load utils op: 0.000415802001953125 seconds 10: 10: Time to load utils op: 0.0004889965057373047 seconds 10: Time to load utils op: 0.0004954338073730469 seconds 13: Time to load utils op: 0.0007350444793701172 seconds 13: Time to load utils op: 0.0004532337188720703 seconds 13: Time to load utils op: 0.0004391670227050781 seconds 13: Time to load utils op: 0.0009684562683105469 seconds 13: Time to load utils op: 0.0009961128234863281 seconds 13: Time to load utils op: 0.0009813308715820312 seconds 13: Time to load utils op: 0.0010597705841064453 seconds 13: Time to load utils op: 0.0007081031799316406 seconds 26: Time to load utils op: 0.0009002685546875 seconds 26: Time to load utils op: 0.0013873577117919922 seconds 26: Time to load utils op: 0.0013957023620605469 seconds 26: Time to load utils op: 0.001407623291015625 secondsTime to load utils op: 0.0013422966003417969 seconds 26: 26: Time to load utils op: 0.0012769699096679688 seconds 26: Time to load utils op: 0.0014882087707519531 seconds 26: Time to load utils op: 0.001359701156616211 seconds 0: [2022-11-24 16:48:47,171] [INFO] [utils.py:827:see_memory_usage] after initializing group 0 0: [2022-11-24 16:48:47,172] [INFO] [utils.py:828:see_memory_usage] MA 0.37 GB Max_MA 0.37 GB CA 0.48 GB Max_CA 0 GB 0: [2022-11-24 16:48:47,173] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.58 GB, percent = 6.5% 24: Time to load utils op: 0.000415802001953125 secondsTime to load utils op: 0.00035762786865234375 seconds 24: 24: Time to load utils op: 0.00038814544677734375 seconds 24: Time to load utils op: 0.0003902912139892578 seconds 24: Time to load utils op: 0.00041484832763671875 seconds 24: Time to load utils op: 0.00039386749267578125 seconds 24: Time to load utils op: 0.0003886222839355469 seconds 0: [2022-11-24 16:48:47,212] [INFO] [utils.py:827:see_memory_usage] before initializing group 1 0: [2022-11-24 16:48:47,213] [INFO] [utils.py:828:see_memory_usage] MA 0.37 GB Max_MA 0.37 GB CA 0.48 GB Max_CA 0 GB 0: [2022-11-24 16:48:47,213] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.6 GB, percent = 6.5% 0: [2022-11-24 16:48:47,246] [INFO] [utils.py:827:see_memory_usage] after initializing group 1 0: [2022-11-24 16:48:47,247] [INFO] [utils.py:828:see_memory_usage] MA 0.46 GB Max_MA 0.46 GB CA 0.58 GB Max_CA 1 GB 0: [2022-11-24 16:48:47,247] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.6 GB, percent = 6.5% 0: [2022-11-24 16:48:47,278] [INFO] [utils.py:827:see_memory_usage] before initializing group 2 0: [2022-11-24 16:48:47,278] [INFO] [utils.py:828:see_memory_usage] MA 0.46 GB Max_MA 0.46 GB CA 0.58 GB Max_CA 1 GB 0: [2022-11-24 16:48:47,278] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.6 GB, percent = 6.5% 0: [2022-11-24 16:48:47,311] [INFO] [utils.py:827:see_memory_usage] after initializing group 2 0: [2022-11-24 16:48:47,312] [INFO] [utils.py:828:see_memory_usage] MA 0.46 GB Max_MA 0.46 GB CA 0.58 GB Max_CA 1 GB 0: [2022-11-24 16:48:47,312] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.6 GB, percent = 6.5% 0: [2022-11-24 16:48:47,342] [INFO] [utils.py:827:see_memory_usage] before initialize_optimizer 0: [2022-11-24 16:48:47,342] [INFO] [utils.py:828:see_memory_usage] MA 0.46 GB Max_MA 0.46 GB CA 0.58 GB Max_CA 1 GB 0: [2022-11-24 16:48:47,343] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.6 GB, percent = 6.5% 0: [2022-11-24 16:48:47,378] [INFO] [utils.py:827:see_memory_usage] end initialize_optimizer 0: [2022-11-24 16:48:47,379] [INFO] [utils.py:828:see_memory_usage] MA 0.47 GB Max_MA 0.47 GB CA 0.58 GB Max_CA 1 GB 0: [2022-11-24 16:48:47,379] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.6 GB, percent = 6.5% 0: [2022-11-24 16:48:47,409] [INFO] [utils.py:827:see_memory_usage] end bf16_optimizer 0: [2022-11-24 16:48:47,410] [INFO] [utils.py:828:see_memory_usage] MA 0.47 GB Max_MA 0.47 GB CA 0.58 GB Max_CA 1 GB 0: [2022-11-24 16:48:47,410] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 32.6 GB, percent = 6.5% 0: [2022-11-24 16:48:47,410] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = FusedAdam 0: [2022-11-24 16:48:47,410] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using client LR scheduler 0: [2022-11-24 16:48:47,410] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = 0: [2022-11-24 16:48:47,410] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0002, 0.0002, 0.0002], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 0: [2022-11-24 16:48:47,410] [INFO] [config.py:1007:print] DeepSpeedEngine configuration: 0: [2022-11-24 16:48:47,410] [INFO] [config.py:1011:print] activation_checkpointing_config { 0: "partition_activations": false, 0: "contiguous_memory_optimization": false, 0: "cpu_checkpointing": false, 0: "number_checkpoints": null, 0: "synchronize_checkpoint_boundary": false, 0: "profile": false 0: } 0: [2022-11-24 16:48:47,410] [INFO] [config.py:1011:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] amp_enabled .................. False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] amp_params ................... False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] autotuning_config ............ { 0: "enabled": false, 0: "start_step": null, 0: "end_step": null, 0: "metric_path": null, 0: "arg_mappings": null, 0: "metric": "throughput", 0: "model_info": null, 0: "results_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_results", 0: "exps_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_exps", 0: "overwrite": true, 0: "fast": true, 0: "start_profile_step": 3, 0: "end_profile_step": 5, 0: "tuner_type": "gridsearch", 0: "tuner_early_stopping": 5, 0: "tuner_num_trials": 50, 0: "model_info_path": null, 0: "mp_size": 1, 0: "max_train_batch_size": null, 0: "min_train_batch_size": 1, 0: "max_train_micro_batch_size_per_gpu": 1.024000e+03, 0: "min_train_micro_batch_size_per_gpu": 1, 0: "num_tuning_micro_batch_sizes": 3 0: } 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] bfloat16_enabled ............. True 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] checkpoint_parallel_write_pipeline False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] checkpoint_tag_validation_enabled True 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] checkpoint_tag_validation_fail False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] comms_config ................. 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] communication_data_type ...... None 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_pa 0: rameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] curriculum_enabled ........... False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] curriculum_params ............ False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] dataloader_drop_last ......... False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] disable_allgather ............ False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] dump_state ................... False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] dynamic_loss_scale_args ...... None 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_enabled ........... False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_gas_boundary_resolution 1 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_layer_name ........ bert.encoder.layer 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_layer_num ......... 0 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_max_iter .......... 100 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_stability ......... 1e-06 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_tol ............... 0.01 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] eigenvalue_verbose ........... False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] elasticity_enabled ........... False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] flops_profiler_config ........ { 0: "enabled": false, 0: "profile_step": 1, 0: "module_depth": -1, 0: "top_modules": 1, 0: "detailed": true, 0: "output_file": null 0: } 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] fp16_auto_cast ............... None 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] fp16_enabled ................. False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] fp16_master_weights_and_gradients False 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] global_rank .................. 0 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] gradient_accumulation_steps .. 1 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] gradient_clipping ............ 1.0 0: [2022-11-24 16:48:47,411] [INFO] [config.py:1011:print] gradient_predivide_factor .... 1.0 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] initial_dynamic_scale ........ 1 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] load_universal_checkpoint .... False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] loss_scale ................... 1.0 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] memory_breakdown ............. False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] monitor_config ............... 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] nebula_config ................ { 0: "enabled": false, 0: "persistent_storage_path": null, 0: "persistent_time_interval": 100, 0: "num_of_version_in_retention": 2, 0: "enable_nebula_load": true, 0: "load_path": null 0: } 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] optimizer_legacy_fusion ...... False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] optimizer_name ............... None 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] optimizer_params ............. None 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] pld_enabled .................. False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] pld_params ................... False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] prescale_gradients ........... False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] scheduler_name ............... None 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] scheduler_params ............. None 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] sparse_attention ............. None 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] sparse_gradients_enabled ..... False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] steps_per_print .............. 2000 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] train_batch_size ............. 256 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] train_micro_batch_size_per_gpu 1 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] use_node_local_storage ....... False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] wall_clock_breakdown ......... False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] world_size ................... 256 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] zero_allow_untested_optimizer False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] zero_enabled ................. False 0: [2022-11-24 16:48:47,412] [INFO] [config.py:1011:print] zero_optimization_stage ...... 0 0: [2022-11-24 16:48:47,412] [INFO] [config.py:996:print_user_config] json = { 0: "train_micro_batch_size_per_gpu": 1, 0: "train_batch_size": 256, 0: "gradient_clipping": 1.0, 0: "zero_optimization": { 0: "stage": 0 0: }, 0: "bf16": { 0: "enabled": true 0: }, 0: "steps_per_print": 2.000000e+03, 0: "wall_clock_breakdown": false 0: } 0: Time to load utils op: 0.00040435791015625 seconds 0: [2022-11-24 16:48:47,413] [INFO] [engine.py:87:__init__] CONFIG: micro_batches=1 micro_batch_size=1 11: Time to load utils op: 0.0010144710540771484 seconds 11: Time to load utils op: 0.0010755062103271484 seconds 8: Time to load utils op: 0.0009655952453613281 seconds 11: Time to load utils op: 0.0015628337860107422 seconds 11: Time to load utils op: 0.001543283462524414 seconds 11: Time to load utils op: 0.0015447139739990234 seconds 8: Time to load utils op: 0.0012848377227783203 seconds 11: Time to load utils op: 0.001585245132446289 seconds 11: Time to load utils op: 0.0015468597412109375 secondsTime to load utils op: 0.0015478134155273438 seconds 11: 27: Time to load utils op: 0.0006952285766601562 seconds 7: Time to load utils op: 0.0004968643188476562 seconds 27: Time to load utils op: 0.0007481575012207031 seconds 8: Time to load utils op: 0.0014843940734863281 seconds 8: Time to load utils op: 0.001421213150024414 secondsTime to load utils op: 0.0014312267303466797 seconds 8: Time to load utils op: 0.0014040470123291016 seconds 23: Time to load utils op: 0.0005233287811279297 secondsTime to load utils op: 0.0004851818084716797 seconds 23: 7: Time to load utils op: 0.0005857944488525391 seconds 7: Time to load utils op: 0.0005910396575927734 seconds 8: 12: Time to load utils op: 0.0009622573852539062 seconds 7: Time to load utils op: 0.000568389892578125 seconds 8: Time to load utils op: 0.0014216899871826172 seconds 27: Time to load utils op: 0.0007836818695068359 seconds 8: Time to load utils op: 0.0015048980712890625 seconds 27: Time to load utils op: 0.0005342960357666016 secondsTime to load utils op: 0.00040268898010253906 seconds 27: 27: Time to load utils op: 0.00047659873962402344 seconds 1: Time to load utils op: 0.0007529258728027344 seconds 23: Time to load utils op: 0.0005869865417480469 secondsTime to load utils op: 0.0005724430084228516 seconds 23: 7: Time to load utils op: 0.0005981922149658203 seconds 16: Time to load utils op: 0.0009100437164306641 seconds 22: Time to load utils op: 0.0009183883666992188 seconds 23: Time to load utils op: 0.0005908012390136719 seconds 7: Time to load utils op: 0.0006098747253417969 seconds 1: Time to load utils op: 0.0007910728454589844 seconds 27: Time to load utils op: 0.0005438327789306641 seconds 7: Time to load utils op: 0.0006539821624755859 seconds 7: Time to load utils op: 0.00066375732421875 seconds 22: Time to load utils op: 0.001085519790649414 seconds 23: Time to load utils op: 0.0007207393646240234 secondsTime to load utils op: 0.0007193088531494141 secondsTime to load utils op: 0.0006747245788574219 seconds 23: 23: 27: Time to load utils op: 0.0005168914794921875 seconds 12: Time to load utils op: 0.0014524459838867188 seconds 1: Time to load utils op: 0.001184225082397461 seconds 22: Time to load utils op: 0.0014460086822509766 seconds 22: Time to load utils op: 0.0014126300811767578 secondsTime to load utils op: 0.001402139663696289 seconds 22: 1: Time to load utils op: 0.0013415813446044922 secondsTime to load utils op: 0.0013861656188964844 seconds 1: 1: Time to load utils op: 0.001348733901977539 seconds 16: Time to load utils op: 0.0014717578887939453 seconds 22: Time to load utils op: 0.0014348030090332031 seconds 22: Time to load utils op: 0.0014390945434570312 seconds 12: Time to load utils op: 0.0015676021575927734 seconds 22: Time to load utils op: 0.0014925003051757812 seconds 12: Time to load utils op: 0.0015816688537597656 seconds 12: Time to load utils op: 0.0015494823455810547 seconds 1: Time to load utils op: 0.0013570785522460938 seconds 16: Time to load utils op: 0.0014388561248779297 seconds 9: Time to load utils op: 0.0011565685272216797 seconds 1: Time to load utils op: 0.0014584064483642578 seconds 16: Time to load utils op: 0.0014925003051757812 secondsTime to load utils op: 0.001434326171875 seconds 16: Time to load utils op: 0.0015106201171875 seconds 16: 4: Time to load utils op: 0.0005009174346923828 seconds 12: Time to load utils op: 0.0015642642974853516 seconds 16: Time to load utils op: 0.0014414787292480469 seconds 4: Time to load utils op: 0.0005171298980712891 seconds 4: Time to load utils op: 0.0005049705505371094 seconds 12: Time to load utils op: 0.001522064208984375 seconds 16: Time to load utils op: 0.0015637874603271484 seconds 12: Time to load utils op: 0.0016036033630371094 seconds 4: Time to load utils op: 0.0005426406860351562 secondsTime to load utils op: 0.0005297660827636719 secondsTime to load utils op: 0.0005717277526855469 secondsTime to load utils op: 0.0005619525909423828 seconds 4: 4: 4: 4: Time to load utils op: 0.0004467964172363281 seconds 9: Time to load utils op: 0.0013699531555175781 seconds 9: Time to load utils op: 0.0013728141784667969 seconds 9: Time to load utils op: 0.001461029052734375 seconds 9: Time to load utils op: 0.0013828277587890625 seconds 9: Time to load utils op: 0.0014736652374267578 secondsTime to load utils op: 0.0013895034790039062 seconds 9: 9: Time to load utils op: 0.001505136489868164 seconds 21: Time to load utils op: 0.0008704662322998047 seconds 21: Time to load utils op: 0.001149892807006836 seconds 21: Time to load utils op: 0.001386404037475586 secondsTime to load utils op: 0.0013637542724609375 seconds 21: Time to load utils op: 0.0013802051544189453 seconds 21: 21: Time to load utils op: 0.001371622085571289 seconds 21: Time to load utils op: 0.001356363296508789 seconds 21: Time to load utils op: 0.0013692378997802734 seconds 25: Time to load utils op: 0.0010380744934082031 seconds 25: Time to load utils op: 0.0010843276977539062 seconds 25: Time to load utils op: 0.0013189315795898438 seconds 25: Time to load utils op: 0.0012736320495605469 seconds 25: Time to load utils op: 0.0012936592102050781 seconds 25: Time to load utils op: 0.0013284683227539062 seconds 25: Time to load utils op: 0.0013344287872314453 seconds 25: Time to load utils op: 0.0013356208801269531 seconds 0: [2022-11-24 16:48:47,460] [INFO] [engine.py:145:__init__] RANK=0 STAGE=0 LAYERS=17 [0, 17) STAGE_PARAMS=82741760 (82.742M) TOTAL_PARAMS=82741760 (82.742M) UNIQUE_PARAMS=82741760 (82.742M) 0: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,506] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,507] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,508] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,509] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,510] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,511] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,529] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,529] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 31: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 30: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 26: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 13: [2022-11-24 16:48:47,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,531] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 24: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,532] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,534] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,535] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,536] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,536] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,536] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,538] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,539] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,540] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,540] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,540] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,540] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,540] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,541] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,542] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,543] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 29: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,544] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 2: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 18: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,545] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,546] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,546] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,533] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,569] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 6: [2022-11-24 16:48:47,651] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 27: [2022-11-24 16:48:47,570] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,570] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,570] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,681] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 27: [2022-11-24 16:48:47,570] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 3: [2022-11-24 16:48:47,687] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,570] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 28: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,570] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,726] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,590] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,726] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,688] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 19: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 28: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,688] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,688] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,688] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,688] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 25: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,591] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,744] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,689] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,653] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 15: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 17: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,684] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,689] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 20: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 12: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,620] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 0: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,656] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 21: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,686] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,686] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,689] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,760] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,656] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,686] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,693] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,734] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,656] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,728] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,687] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,726] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,694] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 31: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,734] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,656] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,687] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,694] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,733] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,733] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,735] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,624] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt... 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,656] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,656] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,656] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 10: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,738] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,687] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,694] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,732] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,735] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,747] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,678] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,738] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,745] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,687] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 30: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,694] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,694] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 20: [2022-11-24 16:48:47,734] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,571] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,735] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,621] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 19: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,679] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,740] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 17: [2022-11-24 16:48:47,732] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,708] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,694] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,763] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,734] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,572] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,735] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 26: [2022-11-24 16:48:47,740] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,740] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,747] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,733] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,710] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,760] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,760] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,714] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,734] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,572] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 12: [2022-11-24 16:48:47,755] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,755] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,683] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,592] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,740] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,747] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,733] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,710] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,760] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,723] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,734] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,572] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,763] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 5: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,731] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 6: [2022-11-24 16:48:47,684] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,684] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,748] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,733] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,711] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,726] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,734] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 27: [2022-11-24 16:48:47,572] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,763] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 5: [2022-11-24 16:48:47,771] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 0: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,689] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 8: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,748] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,733] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 28: [2022-11-24 16:48:47,714] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 14: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 23: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,572] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,756] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,756] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 5: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,689] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,770] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 15: [2022-11-24 16:48:47,748] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,714] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,714] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 23: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,756] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 5: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,625] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/mp_rank_00_model_states.pt. 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,692] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,692] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,771] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,756] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,714] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,756] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,760] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,738] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 5: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 5: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,693] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,593] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,718] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,756] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,763] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 3: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,738] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 2: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,693] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,722] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,722] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 18: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,727] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 24: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,738] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,622] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,693] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,770] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,723] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 29: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,737] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 13: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,739] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,694] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,748] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,770] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,723] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,783] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,738] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,739] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,739] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,767] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 4: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,623] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,723] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,738] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,739] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 28: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 30: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,743] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 23: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,626] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 15: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,764] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,726] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,760] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 14: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,743] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 24: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 13: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,763] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,782] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 18: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 29: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 3: [2022-11-24 16:48:47,743] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 2: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,750] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,795] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,743] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,774] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,743] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,750] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,771] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,770] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,744] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,768] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,750] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,771] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,749] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,744] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 1: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,761] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,769] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,751] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 10: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 11: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 21: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 17: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,741] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,750] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 23: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,744] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,762] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 19: [2022-11-24 16:48:47,770] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,742] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,772] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,750] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,744] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 7: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 0: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 21: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,743] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,751] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,751] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,765] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,725] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,777] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,779] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,751] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 22: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 26: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,751] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 9: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,778] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 10: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,746] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,782] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 7: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,729] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 8: [2022-11-24 16:48:47,784] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,812] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,747] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,780] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 11: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 22: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,748] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 27: [2022-11-24 16:48:47,773] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,730] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,775] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt... 16: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 20: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,776] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 28: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,812] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,795] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 9: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,752] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,757] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 13: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,812] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 26: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,800] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,787] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 12: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,753] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,736] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 18: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 29: [2022-11-24 16:48:47,781] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,758] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 2: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_01-model_00-model_states.pt. 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 28: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,783] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,815] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,790] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,783] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,783] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 3: [2022-11-24 16:48:47,759] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 24: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 13: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,791] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 7: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 15: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 30: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,785] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,801] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,794] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,798] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 31: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 17: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 18: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,786] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,803] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,788] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 14: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,766] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,799] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,808] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,796] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 29: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,838] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 31: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 5: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 2: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 11: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,797] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,789] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 3: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,838] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 19: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 10: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,792] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 23: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 31: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 7: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 21: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,802] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,793] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,804] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,838] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,805] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,838] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,819] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,806] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,838] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 8: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 27: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 31: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,838] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,812] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 10: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 23: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 31: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,813] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,807] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 31: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 16: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 6: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 26: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,809] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 31: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 16: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 9: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 8: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 4: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,845] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 12: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 12: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,867] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,874] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 12: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 4: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 4: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 4: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 12: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 12: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 12: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 4: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 12: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 4: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 12: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 4: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 4: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 12: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 4: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 12: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 4: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,263] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,263] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,263] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_96_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_97_mp_rank_00_optim_states.pt... 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_101_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_102_mp_rank_00_optim_states.pt... 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_103_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_100_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_98_mp_rank_00_optim_states.pt... 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_99_mp_rank_00_optim_states.pt... 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,349] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_97_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 97 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,349] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_99_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,350] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 99 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_98_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,351] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 98 4: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,358] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_103_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 12: [2022-11-24 16:48:48,358] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 103 4: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_100_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 100 4: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_101_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 12: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_102_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 101 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,368] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 102 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,381] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_96_mp_rank_00_optim_states.pt. 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,381] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 96 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 4: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 12: [2022-11-24 16:48:48,419] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 98 4: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 12: [2022-11-24 16:48:48,436] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 97 4: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 12: [2022-11-24 16:48:48,439] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 96 4: [2022-11-24 16:48:48,345] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 39 12: [2022-11-24 16:48:48,451] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 99 4: [2022-11-24 16:48:48,345] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 35 12: [2022-11-24 16:48:48,651] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 103 4: [2022-11-24 16:48:48,346] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 12: [2022-11-24 16:48:48,653] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 101 4: [2022-11-24 16:48:48,347] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 36 12: [2022-11-24 16:48:48,654] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 100 4: [2022-11-24 16:48:48,349] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 12: [2022-11-24 16:48:48,661] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 102 4: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 32 5: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,356] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:48,357] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 38 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,358] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:48,359] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 33 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,361] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 37 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 34 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,467] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 36 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,485] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 35 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,497] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 32 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:48,498] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 39 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:48,499] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 38 5: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 4: [2022-11-24 16:48:48,513] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 33 5: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,542] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 37 5: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 4: [2022-11-24 16:48:48,585] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 34 5: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,867] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,867] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,867] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:47,867] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,867] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 7: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 5: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 5: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 7: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 5: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 5: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 7: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 5: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,083] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 7: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 5: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,145] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,145] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,145] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,145] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 7: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 5: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 7: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 5: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 5: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,230] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 7: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,233] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,224] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,233] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 5: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 7: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 5: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 7: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,316] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 5: [2022-11-24 16:48:48,316] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,317] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,317] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,317] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 5: [2022-11-24 16:48:48,317] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,317] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,317] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,332] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:48,332] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,332] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,332] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 47 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,332] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 45 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,332] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 43 7: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 7: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,334] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 41 7: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 5: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 7: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 5: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 7: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 5: [2022-11-24 16:48:48,335] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 7: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 5: [2022-11-24 16:48:48,335] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 40 7: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 5: [2022-11-24 16:48:48,335] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 46 7: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 5: [2022-11-24 16:48:48,335] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 44 7: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 5: [2022-11-24 16:48:48,345] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 7: [2022-11-24 16:48:48,342] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:48,346] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 42 7: [2022-11-24 16:48:48,342] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:48,370] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 47 7: [2022-11-24 16:48:48,342] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:48,407] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 45 7: [2022-11-24 16:48:48,342] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 56 5: [2022-11-24 16:48:48,410] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 42 7: [2022-11-24 16:48:48,342] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 60 5: [2022-11-24 16:48:48,517] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 43 7: [2022-11-24 16:48:48,342] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 58 5: [2022-11-24 16:48:48,586] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 46 7: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:48,588] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 40 7: [2022-11-24 16:48:48,351] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 59 5: [2022-11-24 16:48:48,598] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 44 7: [2022-11-24 16:48:48,352] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 5: [2022-11-24 16:48:48,610] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 41 7: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 63 7: [2022-11-24 16:48:48,353] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:48,354] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 62 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:48,355] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:48,355] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 61 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 57 0: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:48,461] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 57 0: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:48,469] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 63 0: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:48,491] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 62 0: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:48,502] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 60 0: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:48,511] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 56 0: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 7: [2022-11-24 16:48:48,519] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 61 0: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:48,536] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 58 0: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 7: [2022-11-24 16:48:48,548] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 59 0: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 2: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 2: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 0: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 2: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,962] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,962] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 2: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,994] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,994] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,994] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:47,994] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:47,998] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 2: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 2: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 0: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 2: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,104] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,106] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,106] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,106] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 0: [2022-11-24 16:48:48,106] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,107] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 0: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,165] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,166] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,166] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,166] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,168] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,168] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,168] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,168] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,169] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 2: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 2: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,224] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 2: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 2: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 0: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 2: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,264] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,264] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,265] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,265] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: > using checkpoint value 0.0002 for learning rate 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: > using checkpoint value 2e-05 for minimum learning rate 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: > using checkpoint value 0 for warmup iterations 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: > using checkpoint value 9703701 for total number of iterations 0: > using checkpoint value cosine for decay style 2: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,347] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,348] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 4 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,358] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 4 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: [2022-11-24 16:48:48,365] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 6 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,369] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 3 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,370] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,371] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 0 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,372] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 2: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 0: [2022-11-24 16:48:48,372] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 1 2: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:48,373] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:48,373] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 2 2: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 18 0: [2022-11-24 16:48:48,392] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 6 2: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 19 0: [2022-11-24 16:48:48,398] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 20 0: [2022-11-24 16:48:48,398] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 7 2: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:48,412] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 2: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 16 0: [2022-11-24 16:48:48,412] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 5 2: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:48,439] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 0 2: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 0: checkpoint version 3.0 2: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 22 0: [2022-11-24 16:48:48,450] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 3 2: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 23 0: [2022-11-24 16:48:48,461] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 5 2: [2022-11-24 16:48:48,370] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:48,475] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 1 2: [2022-11-24 16:48:48,371] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 21 0: [2022-11-24 16:48:48,509] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 2 2: [2022-11-24 16:48:48,372] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 0: [2022-11-24 16:48:48,691] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 7 2: [2022-11-24 16:48:48,373] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 17 0: successfully loaded checkpoint from checkpoints_83m at iteration 36000 2: [2022-11-24 16:48:48,388] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 19 2: [2022-11-24 16:48:48,395] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 16 0: estimated model parameters: 0.08274176 2: [2022-11-24 16:48:48,406] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 20 1: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt... 0: estimated model parameters without embeddings: 0.04923648 2: [2022-11-24 16:48:48,406] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 18 0: [after model, optimizer, and learning rate scheduler are built] datetime: 2022-11-24 16:48:49 2: [2022-11-24 16:48:48,435] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 22 0: > building train, validation, and test datasets ... 2: [2022-11-24 16:48:48,441] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 23 0: > datasets target sizes (minimum size): 0: train: 9703701 2: [2022-11-24 16:48:48,605] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 21 0: validation: 9728 0: test: 256 2: [2022-11-24 16:48:48,607] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 17 0: > building train, validation, and test datasets for GPT ... 1: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: > building dataset index ... 1: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: reading sizes... 0: reading pointers... 1: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: reading document index... 0: creating numpy buffer of mmap... 1: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: creating memory view of numpy buffer... 1: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: > finished creating indexed dataset in 0.406208 seconds 1: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: number of documents: 210604984 1: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: > dataset split: 1: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 0: train: 1: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: document indices in [0, 199864130) total of 199864130 documents 1: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: validation: 0: document indices in [199864130, 210394379) total of 10530249 documents 1: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: test: 1: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: document indices in [210394379, 210604984) total of 210605 documents 1: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,859] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,859] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 1: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 1: [2022-11-24 16:48:47,880] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,882] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,882] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 1: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 16: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 1: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,914] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 16: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 16: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 1: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 16: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 16: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 1: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 16: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 1: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 16: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 1: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 16: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 1: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 16: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 1: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 16: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,147] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 1: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 16: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 1: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 16: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 1: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 16: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 1: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 16: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 16: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 1: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 16: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 1: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_128_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_131_mp_rank_00_optim_states.pt... 1: [2022-11-24 16:48:48,342] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_130_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_134_mp_rank_00_optim_states.pt... 1: [2022-11-24 16:48:48,343] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 10 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_132_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_133_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_135_mp_rank_00_optim_states.pt... 16: [2022-11-24 16:48:48,321] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_129_mp_rank_00_optim_states.pt... 1: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_131_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 11 16: [2022-11-24 16:48:48,330] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 131 1: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,332] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_135_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 15 16: [2022-11-24 16:48:48,332] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 135 1: [2022-11-24 16:48:48,365] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_130_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 13 16: [2022-11-24 16:48:48,337] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 130 1: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,349] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_134_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 14 16: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 134 1: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,351] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_128_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 128 1: [2022-11-24 16:48:48,369] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 12 16: [2022-11-24 16:48:48,352] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_132_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,369] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 8 16: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 132 1: [2022-11-24 16:48:48,404] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 16: [2022-11-24 16:48:48,355] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_129_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,405] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 9 16: [2022-11-24 16:48:48,355] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 129 1: [2022-11-24 16:48:48,432] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 10 16: [2022-11-24 16:48:48,355] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_133_mp_rank_00_optim_states.pt. 1: [2022-11-24 16:48:48,440] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 13 16: [2022-11-24 16:48:48,356] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 133 1: [2022-11-24 16:48:48,441] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 11 16: [2022-11-24 16:48:48,461] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 134 1: [2022-11-24 16:48:48,441] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 15 16: [2022-11-24 16:48:48,465] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 129 1: [2022-11-24 16:48:48,441] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 8 16: [2022-11-24 16:48:48,480] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 133 1: [2022-11-24 16:48:48,442] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 14 16: [2022-11-24 16:48:48,496] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 132 1: [2022-11-24 16:48:48,446] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 9 16: [2022-11-24 16:48:48,502] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 135 1: [2022-11-24 16:48:48,452] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 12 16: [2022-11-24 16:48:48,565] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 131 16: [2022-11-24 16:48:48,566] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 128 19: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 16: [2022-11-24 16:48:48,727] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 130 19: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 6: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 6: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 19: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 6: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,971] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 6: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 6: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 19: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 19: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,147] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,147] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,147] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,147] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 6: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 19: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 6: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 19: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 6: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 19: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,232] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 19: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 6: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,338] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 55 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,338] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 53 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 52 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 50 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,348] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 54 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 19: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,361] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 49 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,361] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 51 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,365] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 48 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,454] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 53 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,481] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 51 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,481] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 48 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,482] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 54 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 6: [2022-11-24 16:48:48,492] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 50 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,493] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 55 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,504] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 52 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 6: [2022-11-24 16:48:48,508] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 49 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 25: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 25: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 25: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 25: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_153_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_152_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_159_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_158_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_157_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_155_mp_rank_00_optim_states.pt... 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_156_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_154_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_159_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,337] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 159 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,347] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_153_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,347] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 153 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,349] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_152_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 152 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,349] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 159 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_154_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_158_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,350] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 154 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 19: [2022-11-24 16:48:48,350] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 158 25: [2022-11-24 16:48:47,880] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,352] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_157_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,882] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,882] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 157 25: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,363] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_156_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 156 25: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,380] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 158 25: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 19: [2022-11-24 16:48:48,415] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_155_mp_rank_00_optim_states.pt. 25: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:48,415] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 155 25: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:48,445] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 156 25: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:48,477] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 153 25: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:48,494] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 152 25: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:48,510] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 154 25: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 19: [2022-11-24 16:48:48,542] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 155 25: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 19: [2022-11-24 16:48:48,552] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 157 25: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 25: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,859] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,863] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,863] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,863] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,863] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,864] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,914] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 25: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,880] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,881] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,881] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,883] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 9: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 25: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 9: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 25: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,963] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,963] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,963] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,963] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 9: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 25: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:47,996] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 25: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 9: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 25: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,046] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,048] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 25: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 9: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 25: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,105] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,105] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,105] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,105] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,105] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 9: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 25: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,166] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,166] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 9: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 9: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 9: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,224] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,226] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,226] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,226] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_202_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_200_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_207_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_203_mp_rank_00_optim_states.pt... 9: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_201_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_206_mp_rank_00_optim_states.pt... 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_205_mp_rank_00_optim_states.pt... 9: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 9: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_204_mp_rank_00_optim_states.pt... 9: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_207_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 207 9: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,352] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_206_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 206 9: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,361] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_201_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,361] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_200_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 201 9: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 200 9: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_202_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_204_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_203_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 202 9: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 25: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 204 9: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 203 9: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_205_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,364] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 205 9: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 25: [2022-11-24 16:48:48,451] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 204 9: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 25: [2022-11-24 16:48:48,451] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 202 9: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 25: [2022-11-24 16:48:48,535] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 206 9: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,544] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 200 9: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 25: [2022-11-24 16:48:48,557] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 201 9: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 9: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 25: [2022-11-24 16:48:48,559] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 207 9: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 25: [2022-11-24 16:48:48,573] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 203 9: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 25: [2022-11-24 16:48:48,700] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 205 9: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 9: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:47,816] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:47,817] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,818] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,260] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,261] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,262] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 8: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_75_mp_rank_00_optim_states.pt... 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_73_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_74_mp_rank_00_optim_states.pt... 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_76_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_72_mp_rank_00_optim_states.pt... 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_77_mp_rank_00_optim_states.pt... 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_79_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_78_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,347] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_75_mp_rank_00_optim_states.pt. 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,347] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 75 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,348] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_74_mp_rank_00_optim_states.pt. 9: [2022-11-24 16:48:48,348] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_76_mp_rank_00_optim_states.pt. 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,348] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_77_mp_rank_00_optim_states.pt. 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,348] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_72_mp_rank_00_optim_states.pt. 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 74 8: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 76 8: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 77 8: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 72 8: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,351] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_73_mp_rank_00_optim_states.pt. 8: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 73 8: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_79_mp_rank_00_optim_states.pt. 8: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 9: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 79 8: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,365] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_78_mp_rank_00_optim_states.pt. 8: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,365] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 78 8: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,457] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 79 8: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,483] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 72 8: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,498] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 76 8: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:48,502] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 78 8: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,552] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 74 8: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 9: [2022-11-24 16:48:48,561] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 77 8: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:48,564] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 75 8: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 9: [2022-11-24 16:48:48,572] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 73 8: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 8: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,860] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 8: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 8: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 8: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 8: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 10: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 8: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 10: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 10: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 8: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 10: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 10: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 8: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 10: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 10: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 8: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 10: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_64_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_68_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_67_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_66_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_71_mp_rank_00_optim_states.pt... 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_69_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_65_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_70_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_70_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,338] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 70 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_67_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_69_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 8: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_71_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_68_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,339] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 67 10: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,339] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 69 10: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,339] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 71 10: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 10: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,339] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 68 10: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_66_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,344] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 66 10: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 8: [2022-11-24 16:48:48,353] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_65_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,353] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 65 10: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_64_mp_rank_00_optim_states.pt. 10: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 64 10: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,421] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 64 10: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,448] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 67 10: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,461] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 69 10: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,476] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 65 10: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:48,485] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 70 10: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,498] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 68 10: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 8: [2022-11-24 16:48:48,498] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 71 10: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 8: [2022-11-24 16:48:48,511] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 66 10: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 10: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,868] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,873] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 10: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,259] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_80_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_81_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_82_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_83_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_86_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_84_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,335] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_85_mp_rank_00_optim_states.pt... 10: [2022-11-24 16:48:48,335] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_87_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_85_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 10: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_82_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,345] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 85 11: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,345] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 82 11: [2022-11-24 16:48:47,913] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_84_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,914] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,351] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 84 11: [2022-11-24 16:48:47,914] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_86_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,914] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 86 11: [2022-11-24 16:48:47,914] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,371] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 84 11: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,372] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_83_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,373] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_87_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,373] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 83 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,373] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 87 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,381] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_81_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,381] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 81 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,382] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_80_mp_rank_00_optim_states.pt. 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,382] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 80 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,387] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 82 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,396] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 85 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,458] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 86 11: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,563] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 83 11: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,578] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 87 11: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 10: [2022-11-24 16:48:48,595] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 81 11: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 10: [2022-11-24 16:48:48,596] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 80 11: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 22: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_03-model_00-model_states.pt. 11: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 22: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,869] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,870] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 11: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,874] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,874] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 11: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 11: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,912] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 11: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,915] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 11: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 11: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,959] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,962] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 11: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 11: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,992] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 11: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:47,993] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 11: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 11: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 22: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,223] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 22: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,043] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,044] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 11: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 11: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 11: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 22: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 11: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_90_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_93_mp_rank_00_optim_states.pt... 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_89_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_91_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_94_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_95_mp_rank_00_optim_states.pt... 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_88_mp_rank_00_optim_states.pt... 22: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 11: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_92_mp_rank_00_optim_states.pt... 22: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,347] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_94_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,347] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_93_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,348] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 94 22: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,348] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 93 22: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,348] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_91_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,348] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_89_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 91 22: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,349] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 89 22: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 11: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_90_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_92_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 90 22: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 92 22: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_88_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_95_mp_rank_00_optim_states.pt. 22: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 88 22: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 95 22: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 11: [2022-11-24 16:48:48,438] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 91 22: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,442] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 93 22: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 11: [2022-11-24 16:48:48,461] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 94 22: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,477] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 88 22: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 11: [2022-11-24 16:48:48,477] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 92 22: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,480] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 95 22: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 11: [2022-11-24 16:48:48,481] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 89 22: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 11: [2022-11-24 16:48:48,487] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 90 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 22: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 22: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,834] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 22: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 22: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 26: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_181_mp_rank_00_optim_states.pt... 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_178_mp_rank_00_optim_states.pt... 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_179_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_176_mp_rank_00_optim_states.pt... 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_177_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_183_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_180_mp_rank_00_optim_states.pt... 22: [2022-11-24 16:48:48,324] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_182_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,345] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_183_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,345] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_179_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,969] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,345] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 183 26: [2022-11-24 16:48:47,969] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,345] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 179 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_177_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_178_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_180_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,350] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 177 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,350] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 178 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,350] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 180 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,352] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_182_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,971] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 182 26: [2022-11-24 16:48:47,971] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 22: [2022-11-24 16:48:48,352] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_181_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,353] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 181 26: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,356] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_176_mp_rank_00_optim_states.pt. 26: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,356] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 176 26: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,456] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 176 26: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,458] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 182 26: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,466] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 183 26: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:48,489] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 181 26: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:48,501] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 180 26: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 22: [2022-11-24 16:48:48,504] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 177 26: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:48,545] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 178 26: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 22: [2022-11-24 16:48:48,654] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 179 26: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 26: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 26: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 26: [2022-11-24 16:48:48,023] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,837] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 26: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,840] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,841] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,077] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,077] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,077] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,077] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,077] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 26: [2022-11-24 16:48:48,077] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,078] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 15: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 26: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,083] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,083] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,083] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,116] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,116] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,116] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 26: [2022-11-24 16:48:48,116] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,118] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 26: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,122] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,122] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,122] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,123] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,123] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,124] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,126] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 26: [2022-11-24 16:48:48,138] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,140] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,140] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,141] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,141] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,142] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:48,142] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,143] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,143] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,143] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,143] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,144] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:48,144] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,145] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,199] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,199] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,199] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 15: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 26: [2022-11-24 16:48:48,199] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 15: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 26: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 26: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,080] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 15: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 26: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,083] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,147] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,281] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_210_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:48,281] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_214_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:48,281] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_212_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,281] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_211_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:48,281] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_208_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,281] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_215_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_209_mp_rank_00_optim_states.pt... 26: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_213_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,293] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_212_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,293] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_209_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,293] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_211_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,293] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 212 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,293] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 209 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,293] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 211 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,303] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_213_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,303] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 213 15: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,304] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_208_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 26: [2022-11-24 16:48:48,304] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 208 15: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,306] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_214_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,306] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 214 15: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,309] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_210_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,309] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 210 15: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 15: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_215_mp_rank_00_optim_states.pt. 15: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,316] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 215 15: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,372] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 212 15: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 26: [2022-11-24 16:48:48,372] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 208 15: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,389] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 213 15: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,399] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 211 15: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,400] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 209 15: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,435] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 214 15: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 26: [2022-11-24 16:48:48,519] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 210 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 26: [2022-11-24 16:48:48,590] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 215 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,871] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:47,872] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 15: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:47,874] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 15: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,875] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,876] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 15: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,877] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,878] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,879] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,880] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,880] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:47,880] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,880] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 15: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 15: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 15: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 15: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_122_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_127_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_121_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_126_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_125_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_120_mp_rank_00_optim_states.pt... 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_123_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,322] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_124_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_127_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,331] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 127 21: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,340] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 127 21: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,341] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_125_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,342] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 125 21: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,345] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_120_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,346] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 120 21: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,351] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_124_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,351] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 124 21: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,356] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_122_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,356] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 122 21: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_126_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 126 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,376] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 124 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,382] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_121_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,382] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_123_mp_rank_00_optim_states.pt. 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,382] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 121 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,383] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 123 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,391] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 125 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,400] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 120 21: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 15: [2022-11-24 16:48:48,416] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 121 21: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,422] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 123 21: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,570] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 126 21: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 15: [2022-11-24 16:48:48,638] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 122 21: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 21: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 17: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 17: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 21: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,884] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 21: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 21: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,916] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 21: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,920] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,921] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 21: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 17: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,957] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,958] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,960] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 21: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,961] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:47,995] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,996] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:47,996] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:47,996] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,996] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:47,996] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,996] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 17: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:47,997] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:47,998] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:47,998] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:47,998] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,000] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,001] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,001] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,001] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:48,001] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 17: [2022-11-24 16:48:48,001] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,002] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 21: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,002] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,003] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,005] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:48,005] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,005] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,005] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:48,045] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,046] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,046] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,046] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:48,046] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:48,046] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 17: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,047] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,048] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,048] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,048] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,049] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 17: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,052] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,052] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,053] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 21: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,055] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,055] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,055] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,056] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 17: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: > loading doc-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_train_indexmap_9703701ns_2048sl_1234s_doc_idx.npy 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: > loading sample-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_train_indexmap_9703701ns_2048sl_1234s_sample_idx.npy 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 0: > loading shuffle-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_train_indexmap_9703701ns_2048sl_1234s_shuffle_idx.npy 21: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 21: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 0: loaded indexed file in 0.087 seconds 0: total number of samples: 173377817 0: total number of epochs: 1 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_169_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_173_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_168_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_172_mp_rank_00_optim_states.pt... 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_170_mp_rank_00_optim_states.pt... 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_174_mp_rank_00_optim_states.pt... 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_171_mp_rank_00_optim_states.pt... 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,331] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_175_mp_rank_00_optim_states.pt... 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_171_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 171 17: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,351] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_175_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 17: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,352] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 175 17: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_173_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 21: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 173 17: [2022-11-24 16:48:48,122] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,122] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_168_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,122] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_174_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,122] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_172_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,124] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 17: [2022-11-24 16:48:48,124] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 168 17: [2022-11-24 16:48:48,124] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 174 17: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 172 17: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,365] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_169_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,365] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 169 17: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 21: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_170_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 170 17: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,403] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 173 17: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,407] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 171 17: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,423] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 169 17: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,426] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 174 17: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:48,426] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 168 17: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,426] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 170 17: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 21: [2022-11-24 16:48:48,432] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 172 17: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 21: [2022-11-24 16:48:48,433] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 175 17: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 28: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 17: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 28: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 28: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 17: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 17: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 17: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_137_mp_rank_00_optim_states.pt... 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_136_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_138_mp_rank_00_optim_states.pt... 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_142_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_143_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_139_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_141_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 17: [2022-11-24 16:48:48,282] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_140_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,294] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_142_mp_rank_00_optim_states.pt. 28: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,294] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 142 28: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,294] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_141_mp_rank_00_optim_states.pt. 28: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,295] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_138_mp_rank_00_optim_states.pt. 28: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,295] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 141 28: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,295] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 138 28: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 0: > loading doc-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_valid_indexmap_9728ns_2048sl_1234s_doc_idx.npy 17: [2022-11-24 16:48:48,297] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_136_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,297] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 136 28: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 17: [2022-11-24 16:48:48,312] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_143_mp_rank_00_optim_states.pt. 28: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: > loading sample-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_valid_indexmap_9728ns_2048sl_1234s_sample_idx.npy 17: [2022-11-24 16:48:48,312] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 143 17: [2022-11-24 16:48:48,313] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_139_mp_rank_00_optim_states.pt. 28: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 17: [2022-11-24 16:48:48,314] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 139 28: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 0: > loading shuffle-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_valid_indexmap_9728ns_2048sl_1234s_shuffle_idx.npy 17: [2022-11-24 16:48:48,315] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_137_mp_rank_00_optim_states.pt. 17: [2022-11-24 16:48:48,315] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 137 28: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 17: [2022-11-24 16:48:48,344] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 142 28: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 17: [2022-11-24 16:48:48,357] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 141 28: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 17: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_140_mp_rank_00_optim_states.pt. 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: loaded indexed file in 0.084 seconds 0: total number of samples: 9118345 0: total number of epochs: 1 17: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 140 17: [2022-11-24 16:48:48,374] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 136 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:48,388] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 138 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:48,405] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 140 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 17: [2022-11-24 16:48:48,480] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 143 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 17: [2022-11-24 16:48:48,519] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 139 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 17: [2022-11-24 16:48:48,550] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 137 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: > loading doc-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_test_indexmap_256ns_2048sl_1234s_doc_idx.npy 28: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,843] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: > loading sample-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_test_indexmap_256ns_2048sl_1234s_sample_idx.npy 28: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 0: > loading shuffle-idx mapping from /scratch/project_462000119/data/pile/megatron_data/meg-gpt2_pile_text_document_test_indexmap_256ns_2048sl_1234s_shuffle_idx.npy 28: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 0: loaded indexed file in 0.052 seconds 0: total number of samples: 182928 0: total number of epochs: 1 28: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: > finished creating GPT datasets ... 28: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,844] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:47,848] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 28: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 28: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 28: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,136] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,137] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,139] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,141] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,141] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,141] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,141] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,142] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,142] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,142] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,142] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,144] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,144] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 28: [2022-11-24 16:48:48,144] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,145] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,145] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,146] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 30: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 30: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,164] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,166] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 28: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 30: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 28: [2022-11-24 16:48:48,167] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 30: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 28: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 28: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 30: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 30: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 30: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_230_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_229_mp_rank_00_optim_states.pt... 30: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_225_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_228_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_227_mp_rank_00_optim_states.pt... 30: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_231_mp_rank_00_optim_states.pt... 30: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_226_mp_rank_00_optim_states.pt... 28: [2022-11-24 16:48:48,280] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_224_mp_rank_00_optim_states.pt... 30: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 30: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,296] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_229_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,296] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 229 30: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,303] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_225_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 28: [2022-11-24 16:48:48,304] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 225 30: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,308] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_226_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,308] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_230_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 30: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,308] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 226 30: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,308] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 230 30: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,308] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_224_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,309] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_228_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,309] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 224 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,309] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 228 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,311] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_227_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,311] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 227 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,311] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_231_mp_rank_00_optim_states.pt. 30: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,311] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 231 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,432] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 224 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,474] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 229 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,511] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 225 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,511] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 230 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,511] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 228 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,518] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 227 30: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 28: [2022-11-24 16:48:48,519] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 226 30: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 28: [2022-11-24 16:48:48,520] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 231 30: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 30: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:47,846] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 30: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:47,847] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,850] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,885] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 30: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,888] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 30: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [after dataloaders are built] datetime: 2022-11-24 16:49:11 0: done with setup ... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 0: training ... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: Number of parameters: [tensor rank - pipeline rank] w/ and w/o embeddings: 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 0: [000-000] 0.0827B / 0.0492B 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 30: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:47,917] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_244_mp_rank_00_optim_states.pt... 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_241_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_240_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_243_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_246_mp_rank_00_optim_states.pt... 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_242_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:47,918] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_247_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_245_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,389] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_244_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,389] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_246_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,389] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 244 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 0: [before the start of training step] datetime: 2022-11-24 16:49:11 30: [2022-11-24 16:48:48,389] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 246 30: [2022-11-24 16:48:48,402] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_241_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,402] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 241 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,403] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_245_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,403] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 245 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,403] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_240_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 30: [2022-11-24 16:48:48,403] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 240 18: [2022-11-24 16:48:47,919] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,412] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_242_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,412] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 242 18: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,413] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_247_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:48,413] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 247 18: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,419] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 246 18: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,436] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 245 18: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,439] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_243_mp_rank_00_optim_states.pt. 18: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,439] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 243 18: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:48,451] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 247 18: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:48,471] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 244 18: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 30: [2022-11-24 16:48:48,472] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 240 18: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:48,485] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 242 18: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:48,721] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 243 18: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 30: [2022-11-24 16:48:48,767] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 241 18: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:47,963] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,964] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 18: [2022-11-24 16:48:47,965] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,966] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,966] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,966] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,968] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,968] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:47,969] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,970] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:47,971] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 18: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,013] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,013] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,014] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,014] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,014] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,014] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 29: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,015] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,016] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,016] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,016] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,018] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,018] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,018] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,020] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,021] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,021] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,021] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,021] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,022] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,022] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 18: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,049] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,049] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,049] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,049] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 29: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,050] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,050] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,050] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,050] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,050] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,051] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,054] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,054] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,054] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,056] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,056] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,057] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,057] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,057] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,058] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,058] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,060] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 29: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 18: [2022-11-24 16:48:48,060] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,061] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 0: [2022-11-24 16:49:12,159] [INFO] [checkpointing.py:553:forward] Activation Checkpointing Information 18: [2022-11-24 16:48:48,061] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:49:12,160] [INFO] [checkpointing.py:554:forward] ----Partition Activations False, CPU CHECKPOINTING False 18: [2022-11-24 16:48:48,061] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:49:12,160] [INFO] [checkpointing.py:557:forward] ----contiguous Memory Checkpointing False with None total layers 18: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 0: [2022-11-24 16:49:12,160] [INFO] [checkpointing.py:560:forward] ----Synchronization False 18: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 0: [2022-11-24 16:49:12,160] [INFO] [checkpointing.py:561:forward] ----Profiling time in checkpointing False 18: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 18: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 29: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,117] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 18: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,119] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,123] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,123] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 18: [2022-11-24 16:48:48,123] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,126] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,126] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,126] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 29: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,127] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,127] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,127] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,130] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,130] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 18: [2022-11-24 16:48:48,130] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,130] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,131] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,199] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,199] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,200] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,201] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 29: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 18: [2022-11-24 16:48:48,207] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,202] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,203] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,204] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,205] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,206] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 29: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_144_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 29: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_149_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_145_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,227] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_146_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_150_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_151_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_147_mp_rank_00_optim_states.pt... 18: [2022-11-24 16:48:48,279] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_148_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,287] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_144_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,288] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 144 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,294] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_145_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,294] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 145 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,300] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_148_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,228] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 18: [2022-11-24 16:48:48,300] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 148 29: [2022-11-24 16:48:48,230] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,306] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_151_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,306] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 151 29: [2022-11-24 16:48:48,231] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_149_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,233] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_146_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,233] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,314] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 149 29: [2022-11-24 16:48:48,233] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,314] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 146 29: [2022-11-24 16:48:48,233] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_147_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,233] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,315] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 147 29: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_150_mp_rank_00_optim_states.pt. 29: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 18: [2022-11-24 16:48:48,330] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 150 29: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:48,363] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 151 29: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 29: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:48,370] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 149 29: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:48,379] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 146 29: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:48,381] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 145 29: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 18: [2022-11-24 16:48:48,397] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 147 29: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:48,460] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 148 29: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:48,468] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 144 29: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 18: [2022-11-24 16:48:48,600] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 150 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 29: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,849] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,851] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_232_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_236_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_233_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_235_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_234_mp_rank_00_optim_states.pt... 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_237_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_238_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,305] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_239_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_239_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,314] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 239 14: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,316] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_235_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,316] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_238_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,316] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 235 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,316] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 238 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_233_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,338] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 233 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,338] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_237_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,338] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 237 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_232_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 232 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,341] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_236_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,341] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 236 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,353] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_234_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 29: [2022-11-24 16:48:48,353] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 234 14: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,382] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 238 14: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,409] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 235 14: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,421] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 232 14: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,423] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 236 14: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 29: [2022-11-24 16:48:48,423] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 233 14: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,425] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 237 14: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,426] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 234 14: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 29: [2022-11-24 16:48:48,429] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 239 14: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,821] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 14: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,831] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,832] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 14: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 3: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 14: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 3: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 14: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 3: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 14: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 3: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 14: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,102] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,103] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 3: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 14: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 3: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,225] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,229] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_118_mp_rank_00_optim_states.pt... 3: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_115_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_114_mp_rank_00_optim_states.pt... 3: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 3: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_112_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_113_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_119_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_116_mp_rank_00_optim_states.pt... 14: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_117_mp_rank_00_optim_states.pt... 3: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_117_mp_rank_00_optim_states.pt. 3: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,341] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 117 3: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_119_mp_rank_00_optim_states.pt. 3: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_115_mp_rank_00_optim_states.pt. 3: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 14: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_112_mp_rank_00_optim_states.pt. 3: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,343] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 119 3: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,343] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 115 3: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:48,344] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 112 3: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,350] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_114_mp_rank_00_optim_states.pt. 3: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:48,350] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 114 3: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,350] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 117 3: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:48,356] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_116_mp_rank_00_optim_states.pt. 3: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,357] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_118_mp_rank_00_optim_states.pt. 14: [2022-11-24 16:48:48,357] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 116 3: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,357] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 118 3: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 3: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,386] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 119 3: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,387] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 115 3: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 14: [2022-11-24 16:48:48,389] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 112 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,390] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 114 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,399] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 116 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,405] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_113_mp_rank_00_optim_states.pt. 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 14: [2022-11-24 16:48:48,406] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 113 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:48,430] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 118 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 14: [2022-11-24 16:48:48,439] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 113 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:48,257] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,859] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,258] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,337] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,337] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 28 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,337] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 29 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,340] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 26 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,357] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,358] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 25 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,358] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,359] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 24 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 30 24: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 3: [2022-11-24 16:48:48,391] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,392] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 31 24: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,396] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 29 24: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,400] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 28 24: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,402] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 24: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,402] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 24 24: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,402] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 30 24: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,402] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 27 24: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:48,403] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 25 24: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 3: [2022-11-24 16:48:48,471] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 26 24: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:48,609] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 27 24: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 3: [2022-11-24 16:48:48,615] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 31 24: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,852] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,853] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,908] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,854] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 24: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,855] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,856] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,857] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,944] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 13: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,886] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,949] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:47,887] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 24: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,889] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 13: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 13: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,922] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 13: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 24: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 13: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 24: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,972] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 13: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 24: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 13: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 24: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,039] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:48,024] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 13: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 24: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 13: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 24: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,120] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,121] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,123] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,124] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,124] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,125] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,126] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 13: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 24: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,128] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,317] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,320] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,335] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,335] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,335] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 13: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,089] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,336] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,339] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,340] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,342] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,342] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 24: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,344] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 13: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,345] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,358] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,358] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,359] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,363] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,363] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 13: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 24: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,237] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 13: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 24: [2022-11-24 16:48:48,368] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_198_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_193_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_192_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_194_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_196_mp_rank_00_optim_states.pt... 13: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_199_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_195_mp_rank_00_optim_states.pt... 24: [2022-11-24 16:48:48,435] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_197_mp_rank_00_optim_states.pt... 13: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,443] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_196_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,444] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 196 13: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,458] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 196 13: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,467] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_199_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,468] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 199 13: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 24: [2022-11-24 16:48:48,468] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_193_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,468] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_194_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,469] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 193 13: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,469] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 194 13: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,490] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_198_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,491] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 198 13: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,492] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_192_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,492] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 192 13: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,525] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 198 13: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,526] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 192 13: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,530] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_195_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,530] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 195 13: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,607] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 199 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,665] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 194 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,717] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_197_mp_rank_00_optim_states.pt. 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 24: [2022-11-24 16:48:48,718] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 197 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,726] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 195 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,726] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 193 13: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 24: [2022-11-24 16:48:48,934] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 197 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:47,842] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_104_mp_rank_00_optim_states.pt... 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_111_mp_rank_00_optim_states.pt... 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_105_mp_rank_00_optim_states.pt... 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_107_mp_rank_00_optim_states.pt... 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_108_mp_rank_00_optim_states.pt... 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_110_mp_rank_00_optim_states.pt... 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_106_mp_rank_00_optim_states.pt... 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,326] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_109_mp_rank_00_optim_states.pt... 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,345] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_110_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,345] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 110 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,351] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_106_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,351] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 106 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,358] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_111_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,358] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_105_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,358] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 111 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,359] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 105 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,359] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 110 23: [2022-11-24 16:48:47,858] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 13: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_107_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,361] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 107 23: [2022-11-24 16:48:47,861] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_104_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 104 23: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,369] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_108_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,370] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 108 23: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,380] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 106 23: [2022-11-24 16:48:47,862] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,381] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_109_mp_rank_00_optim_states.pt. 23: [2022-11-24 16:48:47,863] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 13: [2022-11-24 16:48:48,381] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 109 23: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:48,400] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 107 23: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:48,421] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 109 23: [2022-11-24 16:48:47,865] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:48,515] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 111 23: [2022-11-24 16:48:47,866] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:48,520] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 104 23: [2022-11-24 16:48:47,866] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:48,521] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 105 23: [2022-11-24 16:48:47,866] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 13: [2022-11-24 16:48:48,560] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 108 23: [2022-11-24 16:48:47,866] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,866] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,820] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 23: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:47,822] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:47,823] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 23: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,903] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,904] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,905] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,906] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,907] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,909] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,910] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,911] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,947] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,948] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,950] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 23: [2022-11-24 16:48:47,951] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,952] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,953] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,900] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,954] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,955] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,902] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,956] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 20: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 23: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,932] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 20: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,935] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 23: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,988] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 20: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,989] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 20: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,990] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:47,991] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,033] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,037] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,981] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 23: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,040] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,041] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,042] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:47,985] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 20: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,028] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 23: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 20: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,097] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 20: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 23: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,100] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,101] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 23: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,093] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,162] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 23: [2022-11-24 16:48:48,163] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,096] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,098] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,099] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 23: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,222] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,246] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 20: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,161] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,255] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 20: [2022-11-24 16:48:48,217] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,220] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,221] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,256] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_186_mp_rank_00_optim_states.pt... 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_188_mp_rank_00_optim_states.pt... 20: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_191_mp_rank_00_optim_states.pt... 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_185_mp_rank_00_optim_states.pt... 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_187_mp_rank_00_optim_states.pt... 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_189_mp_rank_00_optim_states.pt... 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_184_mp_rank_00_optim_states.pt... 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_190_mp_rank_00_optim_states.pt... 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_186_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_184_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,344] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 186 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,344] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 184 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,363] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_191_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,363] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 191 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_189_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,364] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 189 20: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 23: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_185_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,366] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 185 20: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,366] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_188_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_187_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 188 20: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 187 20: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,367] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_190_mp_rank_00_optim_states.pt. 20: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,367] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 190 20: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 23: [2022-11-24 16:48:48,394] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 186 20: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,408] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 184 20: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,422] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 187 20: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,423] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 189 20: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:48,423] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 191 20: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 20: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,425] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 188 20: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 23: [2022-11-24 16:48:48,434] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 190 20: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 23: [2022-11-24 16:48:48,457] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 185 20: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,810] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,811] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,814] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt... 20: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_161_mp_rank_00_optim_states.pt... 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_162_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:47,830] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_160_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_165_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_166_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_164_mp_rank_00_optim_states.pt... 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_163_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:47,833] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,330] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_167_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,835] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,341] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_164_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 20: [2022-11-24 16:48:48,341] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_166_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,341] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_161_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,836] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,341] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 164 20: [2022-11-24 16:48:48,341] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 166 27: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,341] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 161 27: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_163_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_160_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,839] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_162_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 163 27: [2022-11-24 16:48:47,890] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,360] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_165_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 160 27: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 165 27: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,360] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 162 27: [2022-11-24 16:48:47,891] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,389] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_167_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,389] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 167 27: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,410] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 161 27: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,411] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 162 27: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,411] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 160 27: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,411] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 163 27: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,419] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 165 27: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 20: [2022-11-24 16:48:48,447] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 164 27: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,447] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 167 27: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 20: [2022-11-24 16:48:48,448] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 166 27: [2022-11-24 16:48:47,894] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 31: [2022-11-24 16:48:47,824] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,896] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 31: [2022-11-24 16:48:47,825] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,826] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_04-model_00-model_states.pt. 27: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 31: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 31: [2022-11-24 16:48:47,827] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,828] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,829] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,923] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,892] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,924] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,925] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,926] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,927] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,893] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt... 27: [2022-11-24 16:48:47,928] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,895] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,929] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,897] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,930] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,931] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,898] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_05-model_00-model_states.pt. 27: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,933] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,899] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,934] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,901] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:47,973] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 31: [2022-11-24 16:48:47,936] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,974] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,975] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,937] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,976] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,938] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt... 27: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,939] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,940] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,979] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,941] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,942] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_06-model_00-model_states.pt. 27: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,943] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,945] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,946] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,025] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,977] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,026] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,027] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,978] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt... 27: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,980] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,982] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,031] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:47,983] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,984] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_07-model_00-model_states.pt. 27: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,986] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:47,987] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,081] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,029] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,030] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt... 27: [2022-11-24 16:48:48,082] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,084] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,032] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:48,034] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 31: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,087] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,035] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_08-model_00-model_states.pt. 27: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,036] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,038] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,085] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,148] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 31: [2022-11-24 16:48:48,086] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt... 27: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,088] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,090] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 31: [2022-11-24 16:48:48,091] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_09-model_00-model_states.pt. 27: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,092] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,094] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,154] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,095] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,149] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,150] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,208] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,151] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt... 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,152] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,153] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,155] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,156] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,157] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_10-model_00-model_states.pt. 27: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,213] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,158] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,159] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,160] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 31: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,209] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,234] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,210] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,211] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt... 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,212] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,214] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,235] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,236] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,215] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_11-model_00-model_states.pt. 27: [2022-11-24 16:48:48,238] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,216] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,239] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,218] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,219] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,240] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,241] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,242] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt... 27: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,243] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 31: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,247] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,248] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_12-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,249] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,250] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,251] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,252] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,244] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,245] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_221_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_222_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_218_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_219_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_216_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_217_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_220_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,314] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_223_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_221_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,328] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 221 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,328] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_222_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,328] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 222 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_217_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,253] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt... 27: [2022-11-24 16:48:48,329] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 217 31: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_219_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,329] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 219 31: [2022-11-24 16:48:48,254] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/layer_14-model_00-model_states.pt. 27: [2022-11-24 16:48:48,329] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_218_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_253_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_255_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,329] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 218 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_248_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_249_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,334] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_220_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_252_mp_rank_00_optim_states.pt... 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_250_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,334] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 220 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_254_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_223_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,333] [INFO] [torch_checkpoint_engine.py:21:load] [Torch] Loading checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_251_mp_rank_00_optim_states.pt... 27: [2022-11-24 16:48:48,344] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 223 31: [2022-11-24 16:48:48,343] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_248_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:48,347] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_216_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,343] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 248 27: [2022-11-24 16:48:48,347] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 216 31: [2022-11-24 16:48:48,355] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_252_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:48,395] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 216 31: [2022-11-24 16:48:48,355] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 252 27: [2022-11-24 16:48:48,579] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 223 31: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_249_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:48,687] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 222 31: [2022-11-24 16:48:48,362] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_250_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:48,697] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 217 31: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 249 27: [2022-11-24 16:48:48,715] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 218 31: [2022-11-24 16:48:48,362] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 250 27: [2022-11-24 16:48:48,724] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 220 31: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_253_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:48,744] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 219 31: [2022-11-24 16:48:48,364] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_251_mp_rank_00_optim_states.pt. 27: [2022-11-24 16:48:48,745] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 221 31: [2022-11-24 16:48:48,364] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 253 31: [2022-11-24 16:48:48,364] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 251 31: [2022-11-24 16:48:48,365] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_254_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,365] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 254 31: [2022-11-24 16:48:48,390] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 252 31: [2022-11-24 16:48:48,395] [INFO] [torch_checkpoint_engine.py:23:load] [Torch] Loaded checkpoint from checkpoints_83m/global_step36000/bf16_zero_pp_rank_255_mp_rank_00_optim_states.pt. 31: [2022-11-24 16:48:48,395] [INFO] [engine.py:2844:_get_all_zero_checkpoint_state_dicts] successfully read 256 ZeRO state_dicts for rank 255 31: [2022-11-24 16:48:48,396] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 249 31: [2022-11-24 16:48:48,399] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 250 31: [2022-11-24 16:48:48,404] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 253 31: [2022-11-24 16:48:48,404] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 254 31: [2022-11-24 16:48:48,409] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 251 31: [2022-11-24 16:48:48,413] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 248 31: [2022-11-24 16:48:48,611] [INFO] [engine.py:2784:_load_zero_checkpoint] loading 256 zero partition checkpoints for rank 255 31: time (ms) | load-checkpoint: 1442.18 31: time (ms) | model-and-optimizer-setup: 30580.61 | train/valid/test-data-iterators-setup: 20680.12 0: [Rank 0] (after 36010 iterations) memory (MB) | allocated: 1042.54638671875 | max allocated: 1703.8681640625 | reserved: 2544.0 | max reserved: 2544.0 31: iteration 36010/ 37905 | consumed samples: 9218560 | consumed tokens: 18879610880 | elapsed time per iteration (s): 1.61 | learning rate: 2.111E-05 | global batch size: 256 | lm loss: 2.875034E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 159.412 | TFLOPs: 1.02 | 31: iteration 36020/ 37905 | consumed samples: 9221120 | consumed tokens: 18884853760 | elapsed time per iteration (s): 0.30 | learning rate: 2.110E-05 | global batch size: 256 | lm loss: 2.886648E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 854.166 | TFLOPs: 5.44 | 31: iteration 36030/ 37905 | consumed samples: 9223680 | consumed tokens: 18890096640 | elapsed time per iteration (s): 0.31 | learning rate: 2.108E-05 | global batch size: 256 | lm loss: 2.865829E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 831.923 | TFLOPs: 5.30 | 31: iteration 36040/ 37905 | consumed samples: 9226240 | consumed tokens: 18895339520 | elapsed time per iteration (s): 0.27 | learning rate: 2.107E-05 | global batch size: 256 | lm loss: 2.881052E+00 | grad norm: 0.181 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 944.575 | TFLOPs: 6.02 | 31: iteration 36050/ 37905 | consumed samples: 9228800 | consumed tokens: 18900582400 | elapsed time per iteration (s): 0.33 | learning rate: 2.106E-05 | global batch size: 256 | lm loss: 2.905450E+00 | grad norm: 0.185 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 780.919 | TFLOPs: 4.97 | 31: iteration 36060/ 37905 | consumed samples: 9231360 | consumed tokens: 18905825280 | elapsed time per iteration (s): 0.31 | learning rate: 2.105E-05 | global batch size: 256 | lm loss: 2.913760E+00 | grad norm: 0.194 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 816.949 | TFLOPs: 5.20 | 31: iteration 36070/ 37905 | consumed samples: 9233920 | consumed tokens: 18911068160 | elapsed time per iteration (s): 0.27 | learning rate: 2.104E-05 | global batch size: 256 | lm loss: 2.899444E+00 | grad norm: 0.193 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 936.176 | TFLOPs: 5.96 | 31: iteration 36080/ 37905 | consumed samples: 9236480 | consumed tokens: 18916311040 | elapsed time per iteration (s): 0.25 | learning rate: 2.103E-05 | global batch size: 256 | lm loss: 2.876626E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1032.743 | TFLOPs: 6.58 | 31: iteration 36090/ 37905 | consumed samples: 9239040 | consumed tokens: 18921553920 | elapsed time per iteration (s): 0.26 | learning rate: 2.102E-05 | global batch size: 256 | lm loss: 2.869850E+00 | grad norm: 0.188 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1002.184 | TFLOPs: 6.38 | 31: iteration 36100/ 37905 | consumed samples: 9241600 | consumed tokens: 18926796800 | elapsed time per iteration (s): 0.25 | learning rate: 2.101E-05 | global batch size: 256 | lm loss: 2.925788E+00 | grad norm: 0.200 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1038.494 | TFLOPs: 6.61 | 31: iteration 36110/ 37905 | consumed samples: 9244160 | consumed tokens: 18932039680 | elapsed time per iteration (s): 0.25 | learning rate: 2.099E-05 | global batch size: 256 | lm loss: 2.901951E+00 | grad norm: 0.192 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1029.234 | TFLOPs: 6.55 | 31: iteration 36120/ 37905 | consumed samples: 9246720 | consumed tokens: 18937282560 | elapsed time per iteration (s): 0.29 | learning rate: 2.098E-05 | global batch size: 256 | lm loss: 2.886228E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 879.597 | TFLOPs: 5.60 | 31: iteration 36130/ 37905 | consumed samples: 9249280 | consumed tokens: 18942525440 | elapsed time per iteration (s): 0.28 | learning rate: 2.097E-05 | global batch size: 256 | lm loss: 2.864394E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 908.405 | TFLOPs: 5.79 | 31: iteration 36140/ 37905 | consumed samples: 9251840 | consumed tokens: 18947768320 | elapsed time per iteration (s): 0.29 | learning rate: 2.096E-05 | global batch size: 256 | lm loss: 2.870307E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 890.735 | TFLOPs: 5.67 | 31: iteration 36150/ 37905 | consumed samples: 9254400 | consumed tokens: 18953011200 | elapsed time per iteration (s): 0.26 | learning rate: 2.095E-05 | global batch size: 256 | lm loss: 2.926268E+00 | grad norm: 0.185 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 982.002 | TFLOPs: 6.25 | 31: iteration 36160/ 37905 | consumed samples: 9256960 | consumed tokens: 18958254080 | elapsed time per iteration (s): 0.27 | learning rate: 2.094E-05 | global batch size: 256 | lm loss: 2.925979E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 941.834 | TFLOPs: 6.00 | 31: iteration 36170/ 37905 | consumed samples: 9259520 | consumed tokens: 18963496960 | elapsed time per iteration (s): 0.24 | learning rate: 2.093E-05 | global batch size: 256 | lm loss: 2.908002E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1045.824 | TFLOPs: 6.66 | 31: iteration 36180/ 37905 | consumed samples: 9262080 | consumed tokens: 18968739840 | elapsed time per iteration (s): 0.21 | learning rate: 2.092E-05 | global batch size: 256 | lm loss: 2.891486E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1206.252 | TFLOPs: 7.68 | 31: iteration 36190/ 37905 | consumed samples: 9264640 | consumed tokens: 18973982720 | elapsed time per iteration (s): 0.21 | learning rate: 2.091E-05 | global batch size: 256 | lm loss: 2.885267E+00 | grad norm: 0.187 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1245.780 | TFLOPs: 7.93 | 31: iteration 36200/ 37905 | consumed samples: 9267200 | consumed tokens: 18979225600 | elapsed time per iteration (s): 0.23 | learning rate: 2.090E-05 | global batch size: 256 | lm loss: 2.914770E+00 | grad norm: 0.195 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1094.696 | TFLOPs: 6.97 | 31: iteration 36210/ 37905 | consumed samples: 9269760 | consumed tokens: 18984468480 | elapsed time per iteration (s): 0.21 | learning rate: 2.089E-05 | global batch size: 256 | lm loss: 2.879210E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1248.053 | TFLOPs: 7.95 | 31: iteration 36220/ 37905 | consumed samples: 9272320 | consumed tokens: 18989711360 | elapsed time per iteration (s): 0.21 | learning rate: 2.088E-05 | global batch size: 256 | lm loss: 2.916792E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1198.038 | TFLOPs: 7.63 | 31: iteration 36230/ 37905 | consumed samples: 9274880 | consumed tokens: 18994954240 | elapsed time per iteration (s): 0.25 | learning rate: 2.087E-05 | global batch size: 256 | lm loss: 2.907865E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1003.972 | TFLOPs: 6.39 | 31: iteration 36240/ 37905 | consumed samples: 9277440 | consumed tokens: 19000197120 | elapsed time per iteration (s): 0.25 | learning rate: 2.086E-05 | global batch size: 256 | lm loss: 2.906250E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1012.044 | TFLOPs: 6.45 | 31: iteration 36250/ 37905 | consumed samples: 9280000 | consumed tokens: 19005440000 | elapsed time per iteration (s): 0.22 | learning rate: 2.085E-05 | global batch size: 256 | lm loss: 2.873895E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1144.042 | TFLOPs: 7.29 | 31: iteration 36260/ 37905 | consumed samples: 9282560 | consumed tokens: 19010682880 | elapsed time per iteration (s): 0.21 | learning rate: 2.084E-05 | global batch size: 256 | lm loss: 2.863333E+00 | grad norm: 0.193 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1202.922 | TFLOPs: 7.66 | 31: iteration 36270/ 37905 | consumed samples: 9285120 | consumed tokens: 19015925760 | elapsed time per iteration (s): 0.22 | learning rate: 2.083E-05 | global batch size: 256 | lm loss: 2.908844E+00 | grad norm: 0.191 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1173.207 | TFLOPs: 7.47 | 31: iteration 36280/ 37905 | consumed samples: 9287680 | consumed tokens: 19021168640 | elapsed time per iteration (s): 0.23 | learning rate: 2.082E-05 | global batch size: 256 | lm loss: 2.878504E+00 | grad norm: 0.191 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1132.705 | TFLOPs: 7.21 | 31: iteration 36290/ 37905 | consumed samples: 9290240 | consumed tokens: 19026411520 | elapsed time per iteration (s): 0.22 | learning rate: 2.081E-05 | global batch size: 256 | lm loss: 2.909422E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1169.396 | TFLOPs: 7.45 | 31: iteration 36300/ 37905 | consumed samples: 9292800 | consumed tokens: 19031654400 | elapsed time per iteration (s): 0.22 | learning rate: 2.080E-05 | global batch size: 256 | lm loss: 2.912795E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1146.030 | TFLOPs: 7.30 | 31: iteration 36310/ 37905 | consumed samples: 9295360 | consumed tokens: 19036897280 | elapsed time per iteration (s): 0.23 | learning rate: 2.079E-05 | global batch size: 256 | lm loss: 2.908192E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1120.732 | TFLOPs: 7.14 | 31: iteration 36320/ 37905 | consumed samples: 9297920 | consumed tokens: 19042140160 | elapsed time per iteration (s): 0.26 | learning rate: 2.078E-05 | global batch size: 256 | lm loss: 2.877349E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 990.005 | TFLOPs: 6.31 | 31: iteration 36330/ 37905 | consumed samples: 9300480 | consumed tokens: 19047383040 | elapsed time per iteration (s): 0.24 | learning rate: 2.077E-05 | global batch size: 256 | lm loss: 2.855331E+00 | grad norm: 0.193 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1068.031 | TFLOPs: 6.80 | 31: iteration 36340/ 37905 | consumed samples: 9303040 | consumed tokens: 19052625920 | elapsed time per iteration (s): 0.27 | learning rate: 2.076E-05 | global batch size: 256 | lm loss: 2.895671E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 963.504 | TFLOPs: 6.14 | 31: iteration 36350/ 37905 | consumed samples: 9305600 | consumed tokens: 19057868800 | elapsed time per iteration (s): 0.25 | learning rate: 2.075E-05 | global batch size: 256 | lm loss: 2.861573E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1039.004 | TFLOPs: 6.62 | 31: iteration 36360/ 37905 | consumed samples: 9308160 | consumed tokens: 19063111680 | elapsed time per iteration (s): 0.24 | learning rate: 2.074E-05 | global batch size: 256 | lm loss: 2.903195E+00 | grad norm: 0.200 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1073.526 | TFLOPs: 6.84 | 31: iteration 36370/ 37905 | consumed samples: 9310720 | consumed tokens: 19068354560 | elapsed time per iteration (s): 0.27 | learning rate: 2.073E-05 | global batch size: 256 | lm loss: 2.890572E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 944.116 | TFLOPs: 6.01 | 31: iteration 36380/ 37905 | consumed samples: 9313280 | consumed tokens: 19073597440 | elapsed time per iteration (s): 0.24 | learning rate: 2.072E-05 | global batch size: 256 | lm loss: 2.911003E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1078.932 | TFLOPs: 6.87 | 31: iteration 36390/ 37905 | consumed samples: 9315840 | consumed tokens: 19078840320 | elapsed time per iteration (s): 0.25 | learning rate: 2.071E-05 | global batch size: 256 | lm loss: 2.882359E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1022.371 | TFLOPs: 6.51 | 31: iteration 36400/ 37905 | consumed samples: 9318400 | consumed tokens: 19084083200 | elapsed time per iteration (s): 0.25 | learning rate: 2.070E-05 | global batch size: 256 | lm loss: 2.912443E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1013.520 | TFLOPs: 6.45 | 31: iteration 36410/ 37905 | consumed samples: 9320960 | consumed tokens: 19089326080 | elapsed time per iteration (s): 0.25 | learning rate: 2.069E-05 | global batch size: 256 | lm loss: 2.871431E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1035.474 | TFLOPs: 6.59 | 31: iteration 36420/ 37905 | consumed samples: 9323520 | consumed tokens: 19094568960 | elapsed time per iteration (s): 0.23 | learning rate: 2.068E-05 | global batch size: 256 | lm loss: 2.874770E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1104.297 | TFLOPs: 7.03 | 31: iteration 36430/ 37905 | consumed samples: 9326080 | consumed tokens: 19099811840 | elapsed time per iteration (s): 0.28 | learning rate: 2.067E-05 | global batch size: 256 | lm loss: 2.890210E+00 | grad norm: 0.201 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 901.694 | TFLOPs: 5.74 | 31: iteration 36440/ 37905 | consumed samples: 9328640 | consumed tokens: 19105054720 | elapsed time per iteration (s): 0.26 | learning rate: 2.066E-05 | global batch size: 256 | lm loss: 2.869658E+00 | grad norm: 0.192 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 996.883 | TFLOPs: 6.35 | 31: iteration 36450/ 37905 | consumed samples: 9331200 | consumed tokens: 19110297600 | elapsed time per iteration (s): 0.25 | learning rate: 2.065E-05 | global batch size: 256 | lm loss: 2.908603E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1029.916 | TFLOPs: 6.56 | 31: iteration 36460/ 37905 | consumed samples: 9333760 | consumed tokens: 19115540480 | elapsed time per iteration (s): 0.32 | learning rate: 2.064E-05 | global batch size: 256 | lm loss: 2.884865E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 809.432 | TFLOPs: 5.16 | 31: iteration 36470/ 37905 | consumed samples: 9336320 | consumed tokens: 19120783360 | elapsed time per iteration (s): 0.27 | learning rate: 2.064E-05 | global batch size: 256 | lm loss: 2.872897E+00 | grad norm: 0.187 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 965.738 | TFLOPs: 6.15 | 31: iteration 36480/ 37905 | consumed samples: 9338880 | consumed tokens: 19126026240 | elapsed time per iteration (s): 0.28 | learning rate: 2.063E-05 | global batch size: 256 | lm loss: 2.898441E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 914.744 | TFLOPs: 5.83 | 31: iteration 36490/ 37905 | consumed samples: 9341440 | consumed tokens: 19131269120 | elapsed time per iteration (s): 0.24 | learning rate: 2.062E-05 | global batch size: 256 | lm loss: 2.908790E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1072.765 | TFLOPs: 6.83 | 31: iteration 36500/ 37905 | consumed samples: 9344000 | consumed tokens: 19136512000 | elapsed time per iteration (s): 0.24 | learning rate: 2.061E-05 | global batch size: 256 | lm loss: 2.859162E+00 | grad norm: 0.194 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1060.600 | TFLOPs: 6.75 | 31: iteration 36510/ 37905 | consumed samples: 9346560 | consumed tokens: 19141754880 | elapsed time per iteration (s): 0.26 | learning rate: 2.060E-05 | global batch size: 256 | lm loss: 2.901369E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 990.454 | TFLOPs: 6.31 | 31: iteration 36520/ 37905 | consumed samples: 9349120 | consumed tokens: 19146997760 | elapsed time per iteration (s): 0.23 | learning rate: 2.059E-05 | global batch size: 256 | lm loss: 2.859150E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1112.583 | TFLOPs: 7.09 | 31: iteration 36530/ 37905 | consumed samples: 9351680 | consumed tokens: 19152240640 | elapsed time per iteration (s): 0.27 | learning rate: 2.058E-05 | global batch size: 256 | lm loss: 2.892040E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 931.159 | TFLOPs: 5.93 | 31: iteration 36540/ 37905 | consumed samples: 9354240 | consumed tokens: 19157483520 | elapsed time per iteration (s): 0.25 | learning rate: 2.058E-05 | global batch size: 256 | lm loss: 2.892833E+00 | grad norm: 0.200 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1021.372 | TFLOPs: 6.50 | 31: iteration 36550/ 37905 | consumed samples: 9356800 | consumed tokens: 19162726400 | elapsed time per iteration (s): 0.23 | learning rate: 2.057E-05 | global batch size: 256 | lm loss: 2.890966E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1116.574 | TFLOPs: 7.11 | 31: iteration 36560/ 37905 | consumed samples: 9359360 | consumed tokens: 19167969280 | elapsed time per iteration (s): 0.24 | learning rate: 2.056E-05 | global batch size: 256 | lm loss: 2.902261E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1084.403 | TFLOPs: 6.91 | 31: iteration 36570/ 37905 | consumed samples: 9361920 | consumed tokens: 19173212160 | elapsed time per iteration (s): 0.24 | learning rate: 2.055E-05 | global batch size: 256 | lm loss: 2.873650E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1045.243 | TFLOPs: 6.66 | 31: iteration 36580/ 37905 | consumed samples: 9364480 | consumed tokens: 19178455040 | elapsed time per iteration (s): 0.23 | learning rate: 2.054E-05 | global batch size: 256 | lm loss: 2.861152E+00 | grad norm: 0.188 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1105.778 | TFLOPs: 7.04 | 31: iteration 36590/ 37905 | consumed samples: 9367040 | consumed tokens: 19183697920 | elapsed time per iteration (s): 0.22 | learning rate: 2.053E-05 | global batch size: 256 | lm loss: 2.879736E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1165.230 | TFLOPs: 7.42 | 31: iteration 36600/ 37905 | consumed samples: 9369600 | consumed tokens: 19188940800 | elapsed time per iteration (s): 0.26 | learning rate: 2.053E-05 | global batch size: 256 | lm loss: 2.841673E+00 | grad norm: 0.192 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 999.871 | TFLOPs: 6.37 | 31: iteration 36610/ 37905 | consumed samples: 9372160 | consumed tokens: 19194183680 | elapsed time per iteration (s): 0.25 | learning rate: 2.052E-05 | global batch size: 256 | lm loss: 2.921107E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1013.586 | TFLOPs: 6.46 | 31: iteration 36620/ 37905 | consumed samples: 9374720 | consumed tokens: 19199426560 | elapsed time per iteration (s): 0.23 | learning rate: 2.051E-05 | global batch size: 256 | lm loss: 2.891282E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1109.348 | TFLOPs: 7.07 | 31: iteration 36630/ 37905 | consumed samples: 9377280 | consumed tokens: 19204669440 | elapsed time per iteration (s): 0.24 | learning rate: 2.050E-05 | global batch size: 256 | lm loss: 2.837898E+00 | grad norm: 0.191 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1082.420 | TFLOPs: 6.89 | 31: iteration 36640/ 37905 | consumed samples: 9379840 | consumed tokens: 19209912320 | elapsed time per iteration (s): 0.23 | learning rate: 2.049E-05 | global batch size: 256 | lm loss: 2.883154E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1094.045 | TFLOPs: 6.97 | 31: iteration 36650/ 37905 | consumed samples: 9382400 | consumed tokens: 19215155200 | elapsed time per iteration (s): 0.25 | learning rate: 2.049E-05 | global batch size: 256 | lm loss: 2.869710E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1030.292 | TFLOPs: 6.56 | 31: iteration 36660/ 37905 | consumed samples: 9384960 | consumed tokens: 19220398080 | elapsed time per iteration (s): 0.26 | learning rate: 2.048E-05 | global batch size: 256 | lm loss: 2.890990E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 988.640 | TFLOPs: 6.30 | 31: iteration 36670/ 37905 | consumed samples: 9387520 | consumed tokens: 19225640960 | elapsed time per iteration (s): 0.24 | learning rate: 2.047E-05 | global batch size: 256 | lm loss: 2.905231E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1080.889 | TFLOPs: 6.88 | 31: iteration 36680/ 37905 | consumed samples: 9390080 | consumed tokens: 19230883840 | elapsed time per iteration (s): 0.26 | learning rate: 2.046E-05 | global batch size: 256 | lm loss: 2.890004E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 971.351 | TFLOPs: 6.19 | 31: iteration 36690/ 37905 | consumed samples: 9392640 | consumed tokens: 19236126720 | elapsed time per iteration (s): 0.25 | learning rate: 2.046E-05 | global batch size: 256 | lm loss: 2.886239E+00 | grad norm: 0.187 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1036.348 | TFLOPs: 6.60 | 31: iteration 36700/ 37905 | consumed samples: 9395200 | consumed tokens: 19241369600 | elapsed time per iteration (s): 0.23 | learning rate: 2.045E-05 | global batch size: 256 | lm loss: 2.884292E+00 | grad norm: 0.184 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1121.173 | TFLOPs: 7.14 | 31: iteration 36710/ 37905 | consumed samples: 9397760 | consumed tokens: 19246612480 | elapsed time per iteration (s): 0.24 | learning rate: 2.044E-05 | global batch size: 256 | lm loss: 2.872428E+00 | grad norm: 0.200 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1048.399 | TFLOPs: 6.68 | 31: iteration 36720/ 37905 | consumed samples: 9400320 | consumed tokens: 19251855360 | elapsed time per iteration (s): 0.23 | learning rate: 2.043E-05 | global batch size: 256 | lm loss: 2.869320E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1125.764 | TFLOPs: 7.17 | 31: iteration 36730/ 37905 | consumed samples: 9402880 | consumed tokens: 19257098240 | elapsed time per iteration (s): 0.23 | learning rate: 2.043E-05 | global batch size: 256 | lm loss: 2.892738E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1110.243 | TFLOPs: 7.07 | 31: iteration 36740/ 37905 | consumed samples: 9405440 | consumed tokens: 19262341120 | elapsed time per iteration (s): 0.22 | learning rate: 2.042E-05 | global batch size: 256 | lm loss: 2.901523E+00 | grad norm: 0.193 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1148.306 | TFLOPs: 7.31 | 31: iteration 36750/ 37905 | consumed samples: 9408000 | consumed tokens: 19267584000 | elapsed time per iteration (s): 0.22 | learning rate: 2.041E-05 | global batch size: 256 | lm loss: 2.876865E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1153.903 | TFLOPs: 7.35 | 31: iteration 36760/ 37905 | consumed samples: 9410560 | consumed tokens: 19272826880 | elapsed time per iteration (s): 0.22 | learning rate: 2.041E-05 | global batch size: 256 | lm loss: 2.895394E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1148.247 | TFLOPs: 7.31 | 31: iteration 36770/ 37905 | consumed samples: 9413120 | consumed tokens: 19278069760 | elapsed time per iteration (s): 0.24 | learning rate: 2.040E-05 | global batch size: 256 | lm loss: 2.878531E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1068.867 | TFLOPs: 6.81 | 31: iteration 36780/ 37905 | consumed samples: 9415680 | consumed tokens: 19283312640 | elapsed time per iteration (s): 0.25 | learning rate: 2.039E-05 | global batch size: 256 | lm loss: 2.917804E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1018.818 | TFLOPs: 6.49 | 31: iteration 36790/ 37905 | consumed samples: 9418240 | consumed tokens: 19288555520 | elapsed time per iteration (s): 0.25 | learning rate: 2.038E-05 | global batch size: 256 | lm loss: 2.904743E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1021.078 | TFLOPs: 6.50 | 31: iteration 36800/ 37905 | consumed samples: 9420800 | consumed tokens: 19293798400 | elapsed time per iteration (s): 0.24 | learning rate: 2.038E-05 | global batch size: 256 | lm loss: 2.870599E+00 | grad norm: 0.186 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1051.867 | TFLOPs: 6.70 | 31: iteration 36810/ 37905 | consumed samples: 9423360 | consumed tokens: 19299041280 | elapsed time per iteration (s): 0.27 | learning rate: 2.037E-05 | global batch size: 256 | lm loss: 2.895813E+00 | grad norm: 0.187 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 954.421 | TFLOPs: 6.08 | 31: iteration 36820/ 37905 | consumed samples: 9425920 | consumed tokens: 19304284160 | elapsed time per iteration (s): 0.26 | learning rate: 2.036E-05 | global batch size: 256 | lm loss: 2.901664E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 993.885 | TFLOPs: 6.33 | 31: iteration 36830/ 37905 | consumed samples: 9428480 | consumed tokens: 19309527040 | elapsed time per iteration (s): 0.22 | learning rate: 2.036E-05 | global batch size: 256 | lm loss: 2.899883E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1171.277 | TFLOPs: 7.46 | 31: iteration 36840/ 37905 | consumed samples: 9431040 | consumed tokens: 19314769920 | elapsed time per iteration (s): 0.23 | learning rate: 2.035E-05 | global batch size: 256 | lm loss: 2.873335E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1097.688 | TFLOPs: 6.99 | 31: iteration 36850/ 37905 | consumed samples: 9433600 | consumed tokens: 19320012800 | elapsed time per iteration (s): 0.24 | learning rate: 2.034E-05 | global batch size: 256 | lm loss: 2.909697E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1063.966 | TFLOPs: 6.78 | 31: iteration 36860/ 37905 | consumed samples: 9436160 | consumed tokens: 19325255680 | elapsed time per iteration (s): 0.21 | learning rate: 2.034E-05 | global batch size: 256 | lm loss: 2.896464E+00 | grad norm: 0.200 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1190.869 | TFLOPs: 7.58 | 31: iteration 36870/ 37905 | consumed samples: 9438720 | consumed tokens: 19330498560 | elapsed time per iteration (s): 0.27 | learning rate: 2.033E-05 | global batch size: 256 | lm loss: 2.861786E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 959.130 | TFLOPs: 6.11 | 31: iteration 36880/ 37905 | consumed samples: 9441280 | consumed tokens: 19335741440 | elapsed time per iteration (s): 0.25 | learning rate: 2.032E-05 | global batch size: 256 | lm loss: 2.854629E+00 | grad norm: 0.194 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1034.520 | TFLOPs: 6.59 | 31: iteration 36890/ 37905 | consumed samples: 9443840 | consumed tokens: 19340984320 | elapsed time per iteration (s): 0.23 | learning rate: 2.032E-05 | global batch size: 256 | lm loss: 2.875157E+00 | grad norm: 0.194 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1119.291 | TFLOPs: 7.13 | 31: iteration 36900/ 37905 | consumed samples: 9446400 | consumed tokens: 19346227200 | elapsed time per iteration (s): 0.23 | learning rate: 2.031E-05 | global batch size: 256 | lm loss: 2.921170E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1124.864 | TFLOPs: 7.16 | 31: iteration 36910/ 37905 | consumed samples: 9448960 | consumed tokens: 19351470080 | elapsed time per iteration (s): 0.26 | learning rate: 2.031E-05 | global batch size: 256 | lm loss: 2.924317E+00 | grad norm: 0.180 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 969.081 | TFLOPs: 6.17 | 31: iteration 36920/ 37905 | consumed samples: 9451520 | consumed tokens: 19356712960 | elapsed time per iteration (s): 0.23 | learning rate: 2.030E-05 | global batch size: 256 | lm loss: 2.893167E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1130.938 | TFLOPs: 7.20 | 31: iteration 36930/ 37905 | consumed samples: 9454080 | consumed tokens: 19361955840 | elapsed time per iteration (s): 0.26 | learning rate: 2.029E-05 | global batch size: 256 | lm loss: 2.908124E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 967.755 | TFLOPs: 6.16 | 31: iteration 36940/ 37905 | consumed samples: 9456640 | consumed tokens: 19367198720 | elapsed time per iteration (s): 0.25 | learning rate: 2.029E-05 | global batch size: 256 | lm loss: 2.926399E+00 | grad norm: 0.182 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1033.160 | TFLOPs: 6.58 | 31: iteration 36950/ 37905 | consumed samples: 9459200 | consumed tokens: 19372441600 | elapsed time per iteration (s): 0.25 | learning rate: 2.028E-05 | global batch size: 256 | lm loss: 2.885104E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1014.637 | TFLOPs: 6.46 | 31: iteration 36960/ 37905 | consumed samples: 9461760 | consumed tokens: 19377684480 | elapsed time per iteration (s): 0.22 | learning rate: 2.028E-05 | global batch size: 256 | lm loss: 2.866648E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1178.484 | TFLOPs: 7.51 | 31: iteration 36970/ 37905 | consumed samples: 9464320 | consumed tokens: 19382927360 | elapsed time per iteration (s): 0.28 | learning rate: 2.027E-05 | global batch size: 256 | lm loss: 2.908297E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 908.678 | TFLOPs: 5.79 | 31: iteration 36980/ 37905 | consumed samples: 9466880 | consumed tokens: 19388170240 | elapsed time per iteration (s): 0.24 | learning rate: 2.026E-05 | global batch size: 256 | lm loss: 2.902516E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1070.869 | TFLOPs: 6.82 | 31: iteration 36990/ 37905 | consumed samples: 9469440 | consumed tokens: 19393413120 | elapsed time per iteration (s): 0.27 | learning rate: 2.026E-05 | global batch size: 256 | lm loss: 2.872168E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 934.016 | TFLOPs: 5.95 | 31: iteration 37000/ 37905 | consumed samples: 9472000 | consumed tokens: 19398656000 | elapsed time per iteration (s): 0.24 | learning rate: 2.025E-05 | global batch size: 256 | lm loss: 2.880073E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1049.924 | TFLOPs: 6.69 |