2023-04-14 08:32:07.120550: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[2023-04-14 08:32:07,881] [WARNING] [runner.py:190:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2023-04-14 08:32:07,893] [INFO] [runner.py:540:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMF19 --master_addr=127.0.0.1 --master_port=29500 --enable_each_rank_log=None main.py --data_path c-s-ale/alpaca-gpt4-data --data_split 10,0,0 --model_name_or_path Hikerell/shine-RLHF-20230414-on-opt-1.3b --gradient_accumulation_steps 2 --lora_dim 128 --zero_stage 0 --per_device_train_batch_size 8 --per_device_eval_batch_size 8 --deepspeed --output_dir ./output
2023-04-14 08:32:12.237885: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[2023-04-14 08:32:12,991] [INFO] [launch.py:222:main] 0 NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.16.2-1+cuda11.8
[2023-04-14 08:32:12,991] [INFO] [launch.py:222:main] 0 NV_LIBNCCL_DEV_PACKAGE_VERSION=2.16.2-1
[2023-04-14 08:32:12,991] [INFO] [launch.py:222:main] 0 NCCL_VERSION=2.16.2-1
[2023-04-14 08:32:12,991] [INFO] [launch.py:222:main] 0 NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
[2023-04-14 08:32:12,991] [INFO] [launch.py:222:main] 0 NV_LIBNCCL_PACKAGE=libnccl2=2.16.2-1+cuda11.8
[2023-04-14 08:32:12,991] [INFO] [launch.py:222:main] 0 NV_LIBNCCL_PACKAGE_NAME=libnccl2
[2023-04-14 08:32:12,992] [INFO] [launch.py:222:main] 0 NV_LIBNCCL_PACKAGE_VERSION=2.16.2-1
[2023-04-14 08:32:12,992] [INFO] [launch.py:229:main] WORLD INFO DICT: {'localhost': [0]}
[2023-04-14 08:32:12,992] [INFO] [launch.py:235:main] nnodes=1, num_local_procs=1, node_rank=0
[2023-04-14 08:32:12,992] [INFO] [launch.py:246:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0]})
[2023-04-14 08:32:12,992] [INFO] [launch.py:247:main] dist_world_size=1
[2023-04-14 08:32:12,992] [INFO] [launch.py:249:main] Setting CUDA_VISIBLE_DEVICES=0
2023-04-14 08:32:16.231797: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
[2023-04-14 08:32:18,468] [INFO] [comm.py:586:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl

Downloading (…)lve/main/config.json:   0%|          | 0.00/802 [00:00<?, ?B/s]
Downloading (…)lve/main/config.json: 100%|██████████| 802/802 [00:00<00:00, 123kB/s]

Downloading (…)olve/main/vocab.json:   0%|          | 0.00/798k [00:00<?, ?B/s]
Downloading (…)olve/main/vocab.json: 100%|██████████| 798k/798k [00:00<00:00, 1.88MB/s]
Downloading (…)olve/main/vocab.json: 100%|██████████| 798k/798k [00:00<00:00, 1.87MB/s]

Downloading (…)olve/main/merges.txt:   0%|          | 0.00/456k [00:00<?, ?B/s]
Downloading (…)olve/main/merges.txt: 100%|██████████| 456k/456k [00:00<00:00, 36.7MB/s]

Downloading pytorch_model.bin:   0%|          | 0.00/2.86G [00:00<?, ?B/s]
Downloading pytorch_model.bin:   0%|          | 10.5M/2.86G [00:01<08:42, 5.45MB/s]
Downloading pytorch_model.bin:   1%|          | 21.0M/2.86G [00:02<04:43, 10.0MB/s]
Downloading pytorch_model.bin:   1%|          | 31.5M/2.86G [00:02<03:52, 12.2MB/s]
Downloading pytorch_model.bin:   1%|▏         | 41.9M/2.86G [00:03<03:27, 13.6MB/s]
Downloading pytorch_model.bin:   2%|▏         | 52.4M/2.86G [00:04<02:55, 16.0MB/s]
Downloading pytorch_model.bin:   2%|▏         | 62.9M/2.86G [00:04<02:52, 16.2MB/s]
Downloading pytorch_model.bin:   3%|▎         | 73.4M/2.86G [00:05<02:34, 18.0MB/s]
Downloading pytorch_model.bin:   3%|▎         | 83.9M/2.86G [00:05<02:38, 17.5MB/s]
Downloading pytorch_model.bin:   3%|▎         | 94.4M/2.86G [00:06<02:40, 17.2MB/s]
Downloading pytorch_model.bin:   4%|▎         | 105M/2.86G [00:06<02:26, 18.8MB/s] 
Downloading pytorch_model.bin:   4%|▍         | 115M/2.86G [00:07<02:32, 18.0MB/s]
Downloading pytorch_model.bin:   4%|▍         | 126M/2.86G [00:07<02:21, 19.4MB/s]
Downloading pytorch_model.bin:   5%|▍         | 136M/2.86G [00:08<02:27, 18.5MB/s]
Downloading pytorch_model.bin:   5%|▌         | 147M/2.86G [00:09<02:33, 17.7MB/s]
Downloading pytorch_model.bin:   6%|▌         | 157M/2.86G [00:09<02:36, 17.2MB/s]
Downloading pytorch_model.bin:   6%|▌         | 168M/2.86G [00:10<02:22, 18.8MB/s]
Downloading pytorch_model.bin:   6%|▌         | 178M/2.86G [00:10<02:28, 18.0MB/s]
Downloading pytorch_model.bin:   7%|▋         | 189M/2.86G [00:11<02:33, 17.4MB/s]
Downloading pytorch_model.bin:   7%|▋         | 199M/2.86G [00:12<02:35, 17.1MB/s]
Downloading pytorch_model.bin:   7%|▋         | 210M/2.86G [00:12<02:24, 18.3MB/s]
Downloading pytorch_model.bin:   8%|▊         | 220M/2.86G [00:13<02:26, 18.0MB/s]
Downloading pytorch_model.bin:   8%|▊         | 231M/2.86G [00:13<02:30, 17.5MB/s]
Downloading pytorch_model.bin:   8%|▊         | 241M/2.86G [00:14<02:48, 15.5MB/s]
Downloading pytorch_model.bin:   9%|▉         | 252M/2.86G [00:15<02:45, 15.8MB/s]
Downloading pytorch_model.bin:   9%|▉         | 262M/2.86G [00:16<02:43, 15.9MB/s]
Downloading pytorch_model.bin:  10%|▉         | 273M/2.86G [00:16<02:41, 16.0MB/s]
Downloading pytorch_model.bin:  10%|▉         | 283M/2.86G [00:17<02:53, 14.8MB/s]
Downloading pytorch_model.bin:  10%|█         | 294M/2.86G [00:18<02:49, 15.1MB/s]
Downloading pytorch_model.bin:  11%|█         | 304M/2.86G [00:18<02:45, 15.5MB/s]
Downloading pytorch_model.bin:  11%|█         | 315M/2.86G [00:19<02:41, 15.7MB/s]
Downloading pytorch_model.bin:  11%|█▏        | 325M/2.86G [00:20<02:39, 15.9MB/s]
Downloading pytorch_model.bin:  12%|█▏        | 336M/2.86G [00:20<02:37, 16.0MB/s]
Downloading pytorch_model.bin:  12%|█▏        | 346M/2.86G [00:21<02:36, 16.1MB/s]
Downloading pytorch_model.bin:  12%|█▏        | 357M/2.86G [00:22<02:34, 16.1MB/s]
Downloading pytorch_model.bin:  13%|█▎        | 367M/2.86G [00:22<02:33, 16.2MB/s]
Downloading pytorch_model.bin:  13%|█▎        | 377M/2.86G [00:23<02:32, 16.3MB/s]
Downloading pytorch_model.bin:  14%|█▎        | 388M/2.86G [00:24<02:31, 16.3MB/s]
Downloading pytorch_model.bin:  14%|█▍        | 398M/2.86G [00:24<02:30, 16.3MB/s]
Downloading pytorch_model.bin:  14%|█▍        | 409M/2.86G [00:25<02:29, 16.3MB/s]
Downloading pytorch_model.bin:  15%|█▍        | 419M/2.86G [00:25<02:29, 16.3MB/s]
Downloading pytorch_model.bin:  15%|█▌        | 430M/2.86G [00:26<02:28, 16.3MB/s]
Downloading pytorch_model.bin:  15%|█▌        | 440M/2.86G [00:27<02:28, 16.3MB/s]
Downloading pytorch_model.bin:  16%|█▌        | 451M/2.86G [00:27<02:27, 16.3MB/s]
Downloading pytorch_model.bin:  16%|█▌        | 461M/2.86G [00:28<02:27, 16.3MB/s]
Downloading pytorch_model.bin:  17%|█▋        | 472M/2.86G [00:29<02:26, 16.3MB/s]
Downloading pytorch_model.bin:  17%|█▋        | 482M/2.86G [00:29<02:25, 16.3MB/s]
Downloading pytorch_model.bin:  17%|█▋        | 493M/2.86G [00:30<02:39, 14.9MB/s]
Downloading pytorch_model.bin:  18%|█▊        | 503M/2.86G [00:31<02:48, 14.0MB/s]
Downloading pytorch_model.bin:  18%|█▊        | 514M/2.86G [00:32<02:53, 13.5MB/s]
Downloading pytorch_model.bin:  18%|█▊        | 524M/2.86G [00:33<02:58, 13.1MB/s]
Downloading pytorch_model.bin:  19%|█▊        | 535M/2.86G [00:34<03:14, 11.9MB/s]
Downloading pytorch_model.bin:  19%|█▉        | 545M/2.86G [00:35<03:11, 12.0MB/s]
Downloading pytorch_model.bin:  19%|█▉        | 556M/2.86G [00:36<03:22, 11.4MB/s]
Downloading pytorch_model.bin:  20%|█▉        | 566M/2.86G [00:37<03:17, 11.6MB/s]
Downloading pytorch_model.bin:  20%|██        | 577M/2.86G [00:37<03:12, 11.8MB/s]
Downloading pytorch_model.bin:  21%|██        | 587M/2.86G [00:38<03:09, 12.0MB/s]
Downloading pytorch_model.bin:  21%|██        | 598M/2.86G [00:39<03:07, 12.1MB/s]
Downloading pytorch_model.bin:  21%|██▏       | 608M/2.86G [00:40<03:04, 12.2MB/s]
Downloading pytorch_model.bin:  22%|██▏       | 619M/2.86G [00:41<03:15, 11.5MB/s]
Downloading pytorch_model.bin:  22%|██▏       | 629M/2.86G [00:42<03:10, 11.7MB/s]
Downloading pytorch_model.bin:  22%|██▏       | 640M/2.86G [00:43<03:06, 11.9MB/s]
Downloading pytorch_model.bin:  23%|██▎       | 650M/2.86G [00:43<03:03, 12.0MB/s]
Downloading pytorch_model.bin:  23%|██▎       | 661M/2.86G [00:44<03:01, 12.1MB/s]
Downloading pytorch_model.bin:  23%|██▎       | 671M/2.86G [00:45<02:59, 12.2MB/s]
Downloading pytorch_model.bin:  24%|██▍       | 682M/2.86G [00:46<02:57, 12.3MB/s]
Downloading pytorch_model.bin:  24%|██▍       | 692M/2.86G [00:47<02:56, 12.3MB/s]
Downloading pytorch_model.bin:  25%|██▍       | 703M/2.86G [00:48<02:54, 12.3MB/s]
Downloading pytorch_model.bin:  25%|██▍       | 713M/2.86G [00:49<02:51, 12.5MB/s]
Downloading pytorch_model.bin:  25%|██▌       | 724M/2.86G [00:49<02:41, 13.2MB/s]
Downloading pytorch_model.bin:  26%|██▌       | 734M/2.86G [00:50<02:44, 13.0MB/s]
Downloading pytorch_model.bin:  26%|██▌       | 744M/2.86G [00:51<02:35, 13.6MB/s]
Downloading pytorch_model.bin:  26%|██▋       | 755M/2.86G [00:52<02:38, 13.2MB/s]
Downloading pytorch_model.bin:  27%|██▋       | 765M/2.86G [00:52<02:29, 14.0MB/s]
Downloading pytorch_model.bin:  27%|██▋       | 776M/2.86G [00:53<02:22, 14.6MB/s]
Downloading pytorch_model.bin:  28%|██▊       | 786M/2.86G [00:54<02:17, 15.1MB/s]
Downloading pytorch_model.bin:  28%|██▊       | 797M/2.86G [00:54<02:13, 15.4MB/s]
Downloading pytorch_model.bin:  28%|██▊       | 807M/2.86G [00:55<02:10, 15.7MB/s]
Downloading pytorch_model.bin:  29%|██▊       | 818M/2.86G [00:55<02:08, 15.9MB/s]
Downloading pytorch_model.bin:  29%|██▉       | 828M/2.86G [00:56<01:55, 17.6MB/s]
Downloading pytorch_model.bin:  29%|██▉       | 839M/2.86G [00:57<02:09, 15.6MB/s]
Downloading pytorch_model.bin:  30%|██▉       | 849M/2.86G [00:57<02:07, 15.8MB/s]
Downloading pytorch_model.bin:  30%|███       | 860M/2.86G [00:58<02:05, 15.9MB/s]
Downloading pytorch_model.bin:  30%|███       | 870M/2.86G [00:59<02:03, 16.1MB/s]
Downloading pytorch_model.bin:  31%|███       | 881M/2.86G [00:59<02:02, 16.1MB/s]
Downloading pytorch_model.bin:  31%|███       | 891M/2.86G [01:00<02:01, 16.2MB/s]
Downloading pytorch_model.bin:  32%|███▏      | 902M/2.86G [01:01<02:00, 16.3MB/s]
Downloading pytorch_model.bin:  32%|███▏      | 912M/2.86G [01:01<01:48, 18.0MB/s]
Downloading pytorch_model.bin:  32%|███▏      | 923M/2.86G [01:02<01:51, 17.4MB/s]
Downloading pytorch_model.bin:  33%|███▎      | 933M/2.86G [01:02<01:52, 17.2MB/s]
Downloading pytorch_model.bin:  33%|███▎      | 944M/2.86G [01:03<01:42, 18.6MB/s]
Downloading pytorch_model.bin:  33%|███▎      | 954M/2.86G [01:03<01:46, 17.9MB/s]
Downloading pytorch_model.bin:  34%|███▎      | 965M/2.86G [01:04<01:48, 17.5MB/s]
Downloading pytorch_model.bin:  34%|███▍      | 975M/2.86G [01:05<01:39, 18.9MB/s]
Downloading pytorch_model.bin:  34%|███▍      | 986M/2.86G [01:05<01:43, 18.1MB/s]
Downloading pytorch_model.bin:  35%|███▍      | 996M/2.86G [01:06<01:46, 17.6MB/s]
Downloading pytorch_model.bin:  35%|███▌      | 1.01G/2.86G [01:06<01:37, 19.0MB/s]
Downloading pytorch_model.bin:  36%|███▌      | 1.02G/2.86G [01:07<01:41, 18.2MB/s]
Downloading pytorch_model.bin:  36%|███▌      | 1.03G/2.86G [01:07<01:43, 17.6MB/s]
Downloading pytorch_model.bin:  36%|███▋      | 1.04G/2.86G [01:08<01:46, 17.1MB/s]
Downloading pytorch_model.bin:  37%|███▋      | 1.05G/2.86G [01:09<01:47, 16.9MB/s]
Downloading pytorch_model.bin:  37%|███▋      | 1.06G/2.86G [01:10<01:58, 15.2MB/s]
Downloading pytorch_model.bin:  37%|███▋      | 1.07G/2.86G [01:10<01:55, 15.5MB/s]
Downloading pytorch_model.bin:  38%|███▊      | 1.08G/2.86G [01:11<01:52, 15.8MB/s]
Downloading pytorch_model.bin:  38%|███▊      | 1.09G/2.86G [01:12<01:50, 15.9MB/s]
Downloading pytorch_model.bin:  39%|███▊      | 1.10G/2.86G [01:12<01:49, 16.1MB/s]
Downloading pytorch_model.bin:  39%|███▉      | 1.11G/2.86G [01:13<01:48, 16.1MB/s]
Downloading pytorch_model.bin:  39%|███▉      | 1.12G/2.86G [01:14<01:57, 14.7MB/s]
Downloading pytorch_model.bin:  40%|███▉      | 1.13G/2.86G [01:14<01:53, 15.2MB/s]
Downloading pytorch_model.bin:  40%|███▉      | 1.14G/2.86G [01:15<02:00, 14.2MB/s]
Downloading pytorch_model.bin:  40%|████      | 1.15G/2.86G [01:16<02:05, 13.6MB/s]
Downloading pytorch_model.bin:  41%|████      | 1.16G/2.86G [01:17<02:07, 13.2MB/s]
Downloading pytorch_model.bin:  41%|████      | 1.17G/2.86G [01:18<02:00, 14.0MB/s]
Downloading pytorch_model.bin:  41%|████▏     | 1.18G/2.86G [01:18<02:04, 13.4MB/s]
Downloading pytorch_model.bin:  42%|████▏     | 1.20G/2.86G [01:19<02:05, 13.2MB/s]
Downloading pytorch_model.bin:  42%|████▏     | 1.21G/2.86G [01:20<01:59, 13.9MB/s]
Downloading pytorch_model.bin:  43%|████▎     | 1.22G/2.86G [01:21<02:01, 13.5MB/s]
Downloading pytorch_model.bin:  43%|████▎     | 1.23G/2.86G [01:22<02:05, 13.0MB/s]
Downloading pytorch_model.bin:  43%|████▎     | 1.24G/2.86G [01:23<02:16, 11.9MB/s]
Downloading pytorch_model.bin:  44%|████▎     | 1.25G/2.86G [01:23<02:14, 12.0MB/s]
Downloading pytorch_model.bin:  44%|████▍     | 1.26G/2.86G [01:24<02:12, 12.1MB/s]
Downloading pytorch_model.bin:  44%|████▍     | 1.27G/2.86G [01:25<02:16, 11.7MB/s]
Downloading pytorch_model.bin:  45%|████▍     | 1.28G/2.86G [01:26<02:16, 11.6MB/s]
Downloading pytorch_model.bin:  45%|████▌     | 1.29G/2.86G [01:27<02:13, 11.8MB/s]
Downloading pytorch_model.bin:  45%|████▌     | 1.30G/2.86G [01:28<02:10, 12.0MB/s]
Downloading pytorch_model.bin:  46%|████▌     | 1.31G/2.86G [01:29<02:08, 12.1MB/s]
Downloading pytorch_model.bin:  46%|████▌     | 1.32G/2.86G [01:30<02:07, 12.1MB/s]
Downloading pytorch_model.bin:  47%|████▋     | 1.33G/2.86G [01:30<02:05, 12.2MB/s]
Downloading pytorch_model.bin:  47%|████▋     | 1.34G/2.86G [01:32<02:13, 11.4MB/s]
Downloading pytorch_model.bin:  47%|████▋     | 1.35G/2.86G [01:33<02:18, 10.9MB/s]
Downloading pytorch_model.bin:  48%|████▊     | 1.36G/2.86G [01:34<02:21, 10.6MB/s]
Downloading pytorch_model.bin:  48%|████▊     | 1.37G/2.86G [01:35<02:23, 10.4MB/s]
Downloading pytorch_model.bin:  48%|████▊     | 1.38G/2.86G [01:36<02:15, 10.9MB/s]
Downloading pytorch_model.bin:  49%|████▉     | 1.39G/2.86G [01:36<02:10, 11.2MB/s]
Downloading pytorch_model.bin:  49%|████▉     | 1.41G/2.86G [01:38<02:23, 10.2MB/s]
Downloading pytorch_model.bin:  50%|████▉     | 1.42G/2.86G [01:39<02:31, 9.52MB/s]
Downloading pytorch_model.bin:  50%|████▉     | 1.43G/2.86G [01:40<02:28, 9.62MB/s]
Downloading pytorch_model.bin:  50%|█████     | 1.44G/2.86G [01:41<02:26, 9.69MB/s]
Downloading pytorch_model.bin:  51%|█████     | 1.45G/2.86G [01:42<02:24, 9.76MB/s]
Downloading pytorch_model.bin:  51%|█████     | 1.46G/2.86G [01:43<02:22, 9.80MB/s]
Downloading pytorch_model.bin:  51%|█████▏    | 1.47G/2.86G [01:44<02:21, 9.84MB/s]
Downloading pytorch_model.bin:  52%|█████▏    | 1.48G/2.86G [01:45<02:20, 9.86MB/s]
Downloading pytorch_model.bin:  52%|█████▏    | 1.49G/2.86G [01:46<02:18, 9.86MB/s]
Downloading pytorch_model.bin:  52%|█████▏    | 1.50G/2.86G [01:47<02:17, 9.89MB/s]
Downloading pytorch_model.bin:  53%|█████▎    | 1.51G/2.86G [01:49<02:16, 9.89MB/s]
Downloading pytorch_model.bin:  53%|█████▎    | 1.52G/2.86G [01:50<02:14, 9.91MB/s]
Downloading pytorch_model.bin:  54%|█████▎    | 1.53G/2.86G [01:50<02:06, 10.5MB/s]
Downloading pytorch_model.bin:  54%|█████▍    | 1.54G/2.86G [01:52<02:08, 10.2MB/s]
Downloading pytorch_model.bin:  54%|█████▍    | 1.55G/2.86G [01:53<02:07, 10.2MB/s]
Downloading pytorch_model.bin:  55%|█████▍    | 1.56G/2.86G [01:53<02:00, 10.7MB/s]
Downloading pytorch_model.bin:  55%|█████▌    | 1.57G/2.86G [01:54<01:55, 11.2MB/s]
Downloading pytorch_model.bin:  55%|█████▌    | 1.58G/2.86G [01:55<01:50, 11.5MB/s]
Downloading pytorch_model.bin:  56%|█████▌    | 1.59G/2.86G [01:56<01:47, 11.8MB/s]
Downloading pytorch_model.bin:  56%|█████▌    | 1.60G/2.86G [01:57<01:44, 12.0MB/s]
Downloading pytorch_model.bin:  56%|█████▋    | 1.61G/2.86G [01:57<01:35, 13.0MB/s]
Downloading pytorch_model.bin:  57%|█████▋    | 1.63G/2.86G [01:58<01:30, 13.7MB/s]
Downloading pytorch_model.bin:  57%|█████▋    | 1.64G/2.86G [01:59<01:31, 13.4MB/s]
Downloading pytorch_model.bin:  58%|█████▊    | 1.65G/2.86G [02:00<01:25, 14.1MB/s]
Downloading pytorch_model.bin:  58%|█████▊    | 1.66G/2.86G [02:00<01:21, 14.7MB/s]
Downloading pytorch_model.bin:  58%|█████▊    | 1.67G/2.86G [02:01<01:11, 16.6MB/s]
Downloading pytorch_model.bin:  59%|█████▊    | 1.68G/2.86G [02:01<01:11, 16.6MB/s]
Downloading pytorch_model.bin:  59%|█████▉    | 1.69G/2.86G [02:02<01:10, 16.5MB/s]
Downloading pytorch_model.bin:  59%|█████▉    | 1.70G/2.86G [02:02<01:03, 18.1MB/s]
Downloading pytorch_model.bin:  60%|█████▉    | 1.71G/2.86G [02:03<01:05, 17.6MB/s]
Downloading pytorch_model.bin:  60%|██████    | 1.72G/2.86G [02:03<00:59, 19.1MB/s]
Downloading pytorch_model.bin:  61%|██████    | 1.73G/2.86G [02:04<01:02, 18.2MB/s]
Downloading pytorch_model.bin:  61%|██████    | 1.74G/2.86G [02:05<01:03, 17.7MB/s]
Downloading pytorch_model.bin:  61%|██████▏   | 1.75G/2.86G [02:05<00:57, 19.1MB/s]
Downloading pytorch_model.bin:  62%|██████▏   | 1.76G/2.86G [02:06<01:00, 18.2MB/s]
Downloading pytorch_model.bin:  62%|██████▏   | 1.77G/2.86G [02:06<00:55, 19.6MB/s]
Downloading pytorch_model.bin:  62%|██████▏   | 1.78G/2.86G [02:07<00:58, 18.5MB/s]
Downloading pytorch_model.bin:  63%|██████▎   | 1.79G/2.86G [02:08<00:59, 17.8MB/s]
Downloading pytorch_model.bin:  63%|██████▎   | 1.80G/2.86G [02:08<01:01, 17.2MB/s]
Downloading pytorch_model.bin:  63%|██████▎   | 1.81G/2.86G [02:09<01:07, 15.4MB/s]
Downloading pytorch_model.bin:  64%|██████▍   | 1.82G/2.86G [02:10<01:06, 15.6MB/s]
Downloading pytorch_model.bin:  64%|██████▍   | 1.84G/2.86G [02:10<01:04, 15.8MB/s]
Downloading pytorch_model.bin:  65%|██████▍   | 1.85G/2.86G [02:11<01:03, 15.9MB/s]
Downloading pytorch_model.bin:  65%|██████▍   | 1.86G/2.86G [02:12<01:08, 14.7MB/s]
Downloading pytorch_model.bin:  65%|██████▌   | 1.87G/2.86G [02:12<01:06, 15.0MB/s]
Downloading pytorch_model.bin:  66%|██████▌   | 1.88G/2.86G [02:13<01:09, 14.2MB/s]
Downloading pytorch_model.bin:  66%|██████▌   | 1.89G/2.86G [02:14<01:05, 14.8MB/s]
Downloading pytorch_model.bin:  66%|██████▋   | 1.90G/2.86G [02:15<01:08, 14.0MB/s]
Downloading pytorch_model.bin:  67%|██████▋   | 1.91G/2.86G [02:16<01:10, 13.5MB/s]
Downloading pytorch_model.bin:  67%|██████▋   | 1.92G/2.86G [02:16<01:06, 14.2MB/s]
Downloading pytorch_model.bin:  67%|██████▋   | 1.93G/2.86G [02:17<01:03, 14.7MB/s]
Downloading pytorch_model.bin:  68%|██████▊   | 1.94G/2.86G [02:18<01:00, 15.2MB/s]
Downloading pytorch_model.bin:  68%|██████▊   | 1.95G/2.86G [02:18<00:58, 15.5MB/s]
Downloading pytorch_model.bin:  69%|██████▊   | 1.96G/2.86G [02:19<01:02, 14.4MB/s]
Downloading pytorch_model.bin:  69%|██████▉   | 1.97G/2.86G [02:20<00:59, 14.9MB/s]
Downloading pytorch_model.bin:  69%|██████▉   | 1.98G/2.86G [02:20<00:57, 15.3MB/s]
Downloading pytorch_model.bin:  70%|██████▉   | 1.99G/2.86G [02:21<00:55, 15.6MB/s]
Downloading pytorch_model.bin:  70%|███████   | 2.00G/2.86G [02:22<00:54, 15.8MB/s]
Downloading pytorch_model.bin:  70%|███████   | 2.01G/2.86G [02:22<00:53, 15.9MB/s]
Downloading pytorch_model.bin:  71%|███████   | 2.02G/2.86G [02:23<00:52, 16.0MB/s]
Downloading pytorch_model.bin:  71%|███████   | 2.03G/2.86G [02:24<00:53, 15.4MB/s]
Downloading pytorch_model.bin:  72%|███████▏  | 2.04G/2.86G [02:24<00:55, 14.7MB/s]
Downloading pytorch_model.bin:  72%|███████▏  | 2.06G/2.86G [02:25<00:53, 15.1MB/s]
Downloading pytorch_model.bin:  72%|███████▏  | 2.07G/2.86G [02:26<00:51, 15.5MB/s]
Downloading pytorch_model.bin:  73%|███████▎  | 2.08G/2.86G [02:26<00:49, 15.7MB/s]
Downloading pytorch_model.bin:  73%|███████▎  | 2.09G/2.86G [02:27<00:48, 15.9MB/s]
Downloading pytorch_model.bin:  73%|███████▎  | 2.10G/2.86G [02:28<00:55, 13.7MB/s]
Downloading pytorch_model.bin:  74%|███████▎  | 2.11G/2.86G [02:29<00:56, 13.3MB/s]
Downloading pytorch_model.bin:  74%|███████▍  | 2.12G/2.86G [02:30<00:52, 14.0MB/s]
Downloading pytorch_model.bin:  74%|███████▍  | 2.13G/2.86G [02:30<00:54, 13.5MB/s]
Downloading pytorch_model.bin:  75%|███████▍  | 2.14G/2.86G [02:31<00:54, 13.2MB/s]
Downloading pytorch_model.bin:  75%|███████▌  | 2.15G/2.86G [02:32<00:50, 14.0MB/s]
Downloading pytorch_model.bin:  76%|███████▌  | 2.16G/2.86G [02:33<00:47, 14.6MB/s]
Downloading pytorch_model.bin:  76%|███████▌  | 2.17G/2.86G [02:33<00:49, 13.9MB/s]
Downloading pytorch_model.bin:  76%|███████▋  | 2.18G/2.86G [02:34<00:46, 14.5MB/s]
Downloading pytorch_model.bin:  77%|███████▋  | 2.19G/2.86G [02:35<00:44, 15.0MB/s]
Downloading pytorch_model.bin:  77%|███████▋  | 2.20G/2.86G [02:35<00:43, 15.2MB/s]
Downloading pytorch_model.bin:  77%|███████▋  | 2.21G/2.86G [02:36<00:44, 14.4MB/s]
Downloading pytorch_model.bin:  78%|███████▊  | 2.22G/2.86G [02:37<00:54, 11.7MB/s]
Downloading pytorch_model.bin:  78%|███████▊  | 2.23G/2.86G [02:39<00:59, 10.4MB/s]
Downloading pytorch_model.bin:  79%|███████▊  | 2.24G/2.86G [02:40<01:00, 10.2MB/s]
Downloading pytorch_model.bin:  79%|███████▉  | 2.25G/2.86G [02:41<00:59, 10.1MB/s]
Downloading pytorch_model.bin:  79%|███████▉  | 2.26G/2.86G [02:42<00:59, 10.0MB/s]
Downloading pytorch_model.bin:  80%|███████▉  | 2.28G/2.86G [02:43<00:58, 10.0MB/s]
Downloading pytorch_model.bin:  80%|███████▉  | 2.29G/2.86G [02:44<00:57, 9.97MB/s]
Downloading pytorch_model.bin:  80%|████████  | 2.30G/2.86G [02:45<00:56, 9.95MB/s]
Downloading pytorch_model.bin:  81%|████████  | 2.31G/2.86G [02:46<00:56, 9.80MB/s]
Downloading pytorch_model.bin:  81%|████████  | 2.32G/2.86G [02:47<00:54, 9.87MB/s]
Downloading pytorch_model.bin:  81%|████████▏ | 2.33G/2.86G [02:48<00:53, 9.88MB/s]
Downloading pytorch_model.bin:  82%|████████▏ | 2.34G/2.86G [02:49<00:52, 9.87MB/s]
Downloading pytorch_model.bin:  82%|████████▏ | 2.35G/2.86G [02:50<00:51, 9.89MB/s]
Downloading pytorch_model.bin:  83%|████████▎ | 2.36G/2.86G [02:51<00:50, 9.90MB/s]
Downloading pytorch_model.bin:  83%|████████▎ | 2.37G/2.86G [02:52<00:48, 10.1MB/s]
Downloading pytorch_model.bin:  83%|████████▎ | 2.38G/2.86G [02:53<00:45, 10.5MB/s]
Downloading pytorch_model.bin:  84%|████████▎ | 2.39G/2.86G [02:54<00:42, 11.0MB/s]
Downloading pytorch_model.bin:  84%|████████▍ | 2.40G/2.86G [02:55<00:40, 11.2MB/s]
Downloading pytorch_model.bin:  84%|████████▍ | 2.41G/2.86G [02:56<00:38, 11.7MB/s]
Downloading pytorch_model.bin:  85%|████████▍ | 2.42G/2.86G [02:57<00:36, 11.9MB/s]
Downloading pytorch_model.bin:  85%|████████▌ | 2.43G/2.86G [02:58<00:35, 12.1MB/s]
Downloading pytorch_model.bin:  85%|████████▌ | 2.44G/2.86G [02:58<00:31, 13.1MB/s]
Downloading pytorch_model.bin:  86%|████████▌ | 2.45G/2.86G [02:59<00:31, 12.9MB/s]
Downloading pytorch_model.bin:  86%|████████▌ | 2.46G/2.86G [03:00<00:28, 13.8MB/s]
Downloading pytorch_model.bin:  87%|████████▋ | 2.47G/2.86G [03:00<00:26, 14.4MB/s]
Downloading pytorch_model.bin:  87%|████████▋ | 2.49G/2.86G [03:01<00:23, 16.2MB/s]
Downloading pytorch_model.bin:  87%|████████▋ | 2.50G/2.86G [03:01<00:22, 16.4MB/s]
Downloading pytorch_model.bin:  88%|████████▊ | 2.51G/2.86G [03:02<00:21, 16.3MB/s]
Downloading pytorch_model.bin:  88%|████████▊ | 2.52G/2.86G [03:03<00:19, 17.9MB/s]
Downloading pytorch_model.bin:  88%|████████▊ | 2.53G/2.86G [03:03<00:18, 17.5MB/s]
Downloading pytorch_model.bin:  89%|████████▉ | 2.54G/2.86G [03:04<00:16, 18.9MB/s]
Downloading pytorch_model.bin:  89%|████████▉ | 2.55G/2.86G [03:04<00:17, 18.2MB/s]
Downloading pytorch_model.bin:  90%|████████▉ | 2.56G/2.86G [03:05<00:17, 17.6MB/s]
Downloading pytorch_model.bin:  90%|████████▉ | 2.57G/2.86G [03:05<00:15, 18.9MB/s]
Downloading pytorch_model.bin:  90%|█████████ | 2.58G/2.86G [03:06<00:15, 17.9MB/s]
Downloading pytorch_model.bin:  91%|█████████ | 2.59G/2.86G [03:07<00:18, 14.5MB/s]
Downloading pytorch_model.bin:  91%|█████████ | 2.60G/2.86G [03:09<00:26, 9.69MB/s]
Downloading pytorch_model.bin:  91%|█████████▏| 2.61G/2.86G [03:12<00:37, 6.61MB/s]
Downloading pytorch_model.bin:  92%|█████████▏| 2.62G/2.86G [03:15<00:45, 5.23MB/s]
Downloading pytorch_model.bin:  92%|█████████▏| 2.63G/2.86G [03:18<00:49, 4.58MB/s]
Downloading pytorch_model.bin:  92%|█████████▏| 2.64G/2.86G [03:21<00:52, 4.10MB/s]
Downloading pytorch_model.bin:  93%|█████████▎| 2.65G/2.86G [03:24<00:53, 3.82MB/s]
Downloading pytorch_model.bin:  93%|█████████▎| 2.66G/2.86G [03:27<00:51, 3.81MB/s]
Downloading pytorch_model.bin:  94%|█████████▎| 2.67G/2.86G [03:30<00:49, 3.73MB/s]
Downloading pytorch_model.bin:  94%|█████████▍| 2.68G/2.86G [03:32<00:45, 3.82MB/s]
Downloading pytorch_model.bin:  94%|█████████▍| 2.69G/2.86G [03:35<00:41, 3.93MB/s]
Downloading pytorch_model.bin:  95%|█████████▍| 2.71G/2.86G [03:37<00:35, 4.29MB/s]
Downloading pytorch_model.bin:  95%|█████████▌| 2.72G/2.86G [03:38<00:30, 4.72MB/s]
Downloading pytorch_model.bin:  95%|█████████▌| 2.73G/2.86G [03:40<00:26, 4.94MB/s]
Downloading pytorch_model.bin:  96%|█████████▌| 2.74G/2.86G [03:42<00:22, 5.34MB/s]
Downloading pytorch_model.bin:  96%|█████████▌| 2.75G/2.86G [03:44<00:19, 5.66MB/s]
Downloading pytorch_model.bin:  96%|█████████▋| 2.76G/2.86G [03:45<00:16, 6.00MB/s]
Downloading pytorch_model.bin:  97%|█████████▋| 2.77G/2.86G [03:47<00:14, 6.28MB/s]
Downloading pytorch_model.bin:  97%|█████████▋| 2.78G/2.86G [03:48<00:12, 6.47MB/s]
Downloading pytorch_model.bin:  98%|█████████▊| 2.79G/2.86G [03:50<00:10, 6.63MB/s]
Downloading pytorch_model.bin:  98%|█████████▊| 2.80G/2.86G [03:51<00:08, 6.72MB/s]
Downloading pytorch_model.bin:  98%|█████████▊| 2.81G/2.86G [03:52<00:07, 6.86MB/s]
Downloading pytorch_model.bin:  99%|█████████▊| 2.82G/2.86G [03:54<00:05, 6.98MB/s]
Downloading pytorch_model.bin:  99%|█████████▉| 2.83G/2.86G [03:55<00:03, 7.31MB/s]
Downloading pytorch_model.bin:  99%|█████████▉| 2.84G/2.86G [03:56<00:02, 7.58MB/s]
Downloading pytorch_model.bin: 100%|█████████▉| 2.85G/2.86G [03:58<00:00, 8.14MB/s]
Downloading pytorch_model.bin: 100%|██████████| 2.86G/2.86G [03:58<00:00, 8.44MB/s]
Downloading pytorch_model.bin: 100%|██████████| 2.86G/2.86G [03:58<00:00, 12.0MB/s]

Downloading readme:   0%|          | 0.00/1.39k [00:00<?, ?B/s]
Downloading readme: 100%|██████████| 1.39k/1.39k [00:00<00:00, 1.25MB/s]
Downloading and preparing dataset None/None to /root/.cache/huggingface/datasets/c-s-ale___json/c-s-ale--alpaca-gpt4-data-35876470b5e8cc77/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e...

Downloading data files:   0%|          | 0/1 [00:00<?, ?it/s]

Downloading data:   0%|          | 0.00/43.4M [00:00<?, ?B/s]

Downloading data:   0%|          | 9.22k/43.4M [00:00<16:17, 44.4kB/s]

Downloading data:   0%|          | 44.0k/43.4M [00:00<06:12, 116kB/s] 

Downloading data:   0%|          | 131k/43.4M [00:00<02:51, 252kB/s] 

Downloading data:   1%|          | 288k/43.4M [00:00<01:35, 451kB/s]

Downloading data:   1%|▏         | 617k/43.4M [00:01<00:49, 859kB/s]

Downloading data:   3%|▎         | 1.25M/43.4M [00:01<00:26, 1.58MB/s]

Downloading data:   6%|▌         | 2.52M/43.4M [00:01<00:13, 3.07MB/s]

Downloading data:  12%|█▏        | 5.10M/43.4M [00:01<00:06, 6.02MB/s]

Downloading data:  21%|██        | 9.13M/43.4M [00:01<00:03, 10.1MB/s]

Downloading data:  30%|███       | 13.2M/43.4M [00:02<00:02, 13.0MB/s]

Downloading data:  40%|███▉      | 17.3M/43.4M [00:02<00:01, 14.8MB/s]

Downloading data:  49%|████▉     | 21.3M/43.4M [00:02<00:01, 16.1MB/s]

Downloading data:  59%|█████▊    | 25.5M/43.4M [00:02<00:01, 17.0MB/s]

Downloading data:  68%|██████▊   | 29.6M/43.4M [00:02<00:00, 17.7MB/s]

Downloading data:  77%|███████▋  | 33.4M/43.4M [00:03<00:00, 17.8MB/s]

Downloading data:  86%|████████▌ | 37.4M/43.4M [00:03<00:00, 18.2MB/s]

Downloading data:  96%|█████████▌| 41.5M/43.4M [00:03<00:00, 18.4MB/s]
Downloading data: 100%|██████████| 43.4M/43.4M [00:03<00:00, 12.0MB/s]

Downloading data files: 100%|██████████| 1/1 [00:06<00:00,  6.01s/it]
Downloading data files: 100%|██████████| 1/1 [00:06<00:00,  6.01s/it]

Extracting data files:   0%|          | 0/1 [00:00<?, ?it/s]
Extracting data files: 100%|██████████| 1/1 [00:00<00:00, 1628.86it/s]

Generating train split:   0%|          | 0/52002 [00:00<?, ? examples/s]
Generating train split: 100%|██████████| 52002/52002 [00:00<00:00, 140619.84 examples/s]
                                                                                        
Dataset json downloaded and prepared to /root/.cache/huggingface/datasets/c-s-ale___json/c-s-ale--alpaca-gpt4-data-35876470b5e8cc77/0.0.0/fe5dd6ea2639a6df622901539cb550cf8797e5a6b2dd7af1cf934bed8e233e6e. Subsequent calls will reuse this data.

  0%|          | 0/1 [00:00<?, ?it/s]
100%|██████████| 1/1 [00:00<00:00, 236.93it/s]
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Using /root/.cache/torch_extensions/py39_cu118 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py39_cu118/fused_adam...
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Detected CUDA files, patching ldflags
Emitting ninja build file /root/.cache/torch_extensions/py39_cu118/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[1/3] /usr/local/cuda/bin/nvcc  -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/lib/python3.9/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/lib/python3.9/dist-packages/deepspeed/ops/csrc/adam -isystem /usr/local/lib/python3.9/dist-packages/torch/include -isystem /usr/local/lib/python3.9/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.9/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.9/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_80,code=compute_80 -gencode=arch=compute_80,code=sm_80 --compiler-options '-fPIC' -O3 -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -lineinfo --use_fast_math -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_80,code=compute_80 -std=c++17 -c /usr/local/lib/python3.9/dist-packages/deepspeed/ops/csrc/adam/multi_tensor_adam.cu -o multi_tensor_adam.cuda.o 
[2/3] c++ -MMD -MF fused_adam_frontend.o.d -DTORCH_EXTENSION_NAME=fused_adam -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/usr/local/lib/python3.9/dist-packages/deepspeed/ops/csrc/includes -I/usr/local/lib/python3.9/dist-packages/deepspeed/ops/csrc/adam -isystem /usr/local/lib/python3.9/dist-packages/torch/include -isystem /usr/local/lib/python3.9/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.9/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.9/dist-packages/torch/include/THC -isystem /usr/local/cuda/include -isystem /usr/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -O3 -std=c++14 -g -Wno-reorder -DVERSION_GE_1_1 -DVERSION_GE_1_3 -DVERSION_GE_1_5 -c /usr/local/lib/python3.9/dist-packages/deepspeed/ops/csrc/adam/fused_adam_frontend.cpp -o fused_adam_frontend.o 
[3/3] c++ fused_adam_frontend.o multi_tensor_adam.cuda.o -shared -L/usr/local/lib/python3.9/dist-packages/torch/lib -lc10 -lc10_cuda -ltorch_cpu -ltorch_cuda -ltorch -ltorch_python -L/usr/local/cuda/lib64 -lcudart -o fused_adam.so
Loading extension module fused_adam...
Time to load fused_adam op: 32.629486322402954 seconds
[2023-04-14 08:38:22,835] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.9.1+717c3020, git-hash=717c3020, git-branch=master
[2023-04-14 08:38:22,841] [INFO] [comm.py:580:init_distributed] Distributed backend already initialized
[2023-04-14 08:38:24,168] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2023-04-14 08:38:24,169] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer
[2023-04-14 08:38:24,169] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2023-04-14 08:38:24,210] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
[2023-04-14 08:38:24,210] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 optimizer with dynamic loss scale
[2023-04-14 08:38:24,277] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = FusedAdam
[2023-04-14 08:38:24,277] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using client LR scheduler
[2023-04-14 08:38:24,277] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = <torch.optim.lr_scheduler.LambdaLR object at 0x7f9db8637fa0>
[2023-04-14 08:38:24,277] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.001, 0.001], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:38:24,278] [INFO] [config.py:953:print] DeepSpeedEngine configuration:
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   activation_checkpointing_config  {
    "partition_activations": false, 
    "contiguous_memory_optimization": false, 
    "cpu_checkpointing": false, 
    "number_checkpoints": null, 
    "synchronize_checkpoint_boundary": false, 
    "profile": false
}
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   amp_enabled .................. False
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   amp_params ................... False
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   autotuning_config ............ {
    "enabled": false, 
    "start_step": null, 
    "end_step": null, 
    "metric_path": null, 
    "arg_mappings": null, 
    "metric": "throughput", 
    "model_info": null, 
    "results_dir": "autotuning_results", 
    "exps_dir": "autotuning_exps", 
    "overwrite": true, 
    "fast": true, 
    "start_profile_step": 3, 
    "end_profile_step": 5, 
    "tuner_type": "gridsearch", 
    "tuner_early_stopping": 5, 
    "tuner_num_trials": 50, 
    "model_info_path": null, 
    "mp_size": 1, 
    "max_train_batch_size": null, 
    "min_train_batch_size": 1, 
    "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
    "min_train_micro_batch_size_per_gpu": 1, 
    "num_tuning_micro_batch_sizes": 3
}
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   bfloat16_enabled ............. False
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   checkpoint_parallel_write_pipeline  False
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   checkpoint_tag_validation_enabled  True
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   checkpoint_tag_validation_fail  False
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f9dab10b4c0>
[2023-04-14 08:38:24,279] [INFO] [config.py:957:print]   communication_data_type ...... None
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   curriculum_enabled_legacy .... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   curriculum_params_legacy ..... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   data_efficiency_enabled ...... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   dataloader_drop_last ......... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   disable_allgather ............ False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   dump_state ................... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   dynamic_loss_scale_args ...... {'init_scale': 65536, 'scale_window': 100, 'delayed_shift': 2, 'min_scale': 1}
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_enabled ........... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_gas_boundary_resolution  1
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_layer_name ........ bert.encoder.layer
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_layer_num ......... 0
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_max_iter .......... 100
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_stability ......... 1e-06
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_tol ............... 0.01
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   eigenvalue_verbose ........... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   elasticity_enabled ........... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   flops_profiler_config ........ {
    "enabled": false, 
    "profile_step": 1, 
    "module_depth": -1, 
    "top_modules": 1, 
    "detailed": true, 
    "output_file": null
}
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   fp16_auto_cast ............... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   fp16_enabled ................. True
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   fp16_master_weights_and_gradients  False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   global_rank .................. 0
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   grad_accum_dtype ............. None
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   gradient_accumulation_steps .. 2
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   gradient_clipping ............ 1.0
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   gradient_predivide_factor .... 1.0
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   initial_dynamic_scale ........ 65536
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   load_universal_checkpoint .... False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   loss_scale ................... 0
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   memory_breakdown ............. False
[2023-04-14 08:38:24,280] [INFO] [config.py:957:print]   monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') enabled=False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   nebula_config ................ {
    "enabled": false, 
    "persistent_storage_path": null, 
    "persistent_time_interval": 100, 
    "num_of_version_in_retention": 2, 
    "enable_nebula_load": true, 
    "load_path": null
}
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   optimizer_legacy_fusion ...... False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   optimizer_name ............... None
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   optimizer_params ............. None
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   pld_enabled .................. False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   pld_params ................... False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   prescale_gradients ........... False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   scheduler_name ............... None
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   scheduler_params ............. None
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   sparse_attention ............. None
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   sparse_gradients_enabled ..... False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   steps_per_print .............. 10
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   train_batch_size ............. 16
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   train_micro_batch_size_per_gpu  8
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   use_node_local_storage ....... False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   wall_clock_breakdown ......... False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   world_size ................... 1
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   zero_allow_untested_optimizer  False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500,000,000 allgather_partitions=True allgather_bucket_size=500,000,000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100,000,000, max_in_cpu=1,000,000,000, pin_memory=False) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=False, pipeline=False, pipeline_read=False, pipeline_write=False, fast_init=False) sub_group_size=1,000,000,000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=30000000 param_persistence_threshold=10000 model_persistence_threshold=sys.maxsize max_live_parameters=30000000 max_reuse_distance=1,000,000,000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False memory_efficient_linear=False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   zero_enabled ................. False
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   zero_force_ds_cpu_optimizer .. True
[2023-04-14 08:38:24,281] [INFO] [config.py:957:print]   zero_optimization_stage ...... 0
[2023-04-14 08:38:24,281] [INFO] [config.py:943:print_user_config]   json = {
    "train_batch_size": 16, 
    "train_micro_batch_size_per_gpu": 8, 
    "steps_per_print": 10, 
    "zero_optimization": {
        "stage": 0, 
        "offload_param": {
            "device": "none"
        }, 
        "offload_optimizer": {
            "device": "none"
        }, 
        "stage3_param_persistence_threshold": 1.000000e+04, 
        "stage3_max_live_parameters": 3.000000e+07, 
        "stage3_prefetch_bucket_size": 3.000000e+07, 
        "memory_efficient_linear": false
    }, 
    "fp16": {
        "enabled": true, 
        "loss_scale_window": 100
    }, 
    "gradient_clipping": 1.0, 
    "prescale_gradients": false, 
    "wall_clock_breakdown": false, 
    "hybrid_engine": {
        "enabled": false, 
        "inference_tp_size": 1, 
        "release_inference_cache": false, 
        "pin_parameters": true, 
        "tp_gather_partition_size": 8
    }
}
Using /root/.cache/torch_extensions/py39_cu118 as PyTorch extensions root...
Creating extension directory /root/.cache/torch_extensions/py39_cu118/utils...
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Emitting ninja build file /root/.cache/torch_extensions/py39_cu118/utils/build.ninja...
Building extension module utils...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[1/2] c++ -MMD -MF flatten_unflatten.o.d -DTORCH_EXTENSION_NAME=utils -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /usr/local/lib/python3.9/dist-packages/torch/include -isystem /usr/local/lib/python3.9/dist-packages/torch/include/torch/csrc/api/include -isystem /usr/local/lib/python3.9/dist-packages/torch/include/TH -isystem /usr/local/lib/python3.9/dist-packages/torch/include/THC -isystem /usr/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++17 -c /usr/local/lib/python3.9/dist-packages/deepspeed/ops/csrc/utils/flatten_unflatten.cpp -o flatten_unflatten.o 
[2/2] c++ flatten_unflatten.o -shared -L/usr/local/lib/python3.9/dist-packages/torch/lib -lc10 -ltorch_cpu -ltorch -ltorch_python -o utils.so
Loading extension module utils...
Time to load utils op: 18.022384881973267 seconds
***** Running training *****
***** Evaluating perplexity, Epoch 0/1 *****
ppl: 63993.62890625
Beginning of Epoch 1/1, Total Micro Batches 5851
[2023-04-14 08:39:57,840] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 0
[2023-04-14 08:39:57,840] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 65536 to 32768.0
[2023-04-14 08:39:57,841] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 65536, reducing to 32768.0
[2023-04-14 08:39:58,452] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1
[2023-04-14 08:39:58,453] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 32768.0 to 16384.0
[2023-04-14 08:39:58,453] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 32768.0, reducing to 16384.0
[2023-04-14 08:40:03,724] [INFO] [logging.py:96:log_dist] [Rank 0] step=10, skipped=2, lr=[0.0009999815554351224, 0.0009999815554351224], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:03,727] [INFO] [timer.py:199:stop] epoch=0/micro_step=20/global_step=10, RunningAvgSamplesPerSec=24.33916149335125, CurrSamplesPerSec=24.679771328541456, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:10,225] [INFO] [logging.py:96:log_dist] [Rank 0] step=20, skipped=2, lr=[0.0009999066267225446, 0.0009999066267225446], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:10,227] [INFO] [timer.py:199:stop] epoch=0/micro_step=40/global_step=20, RunningAvgSamplesPerSec=24.5256045530467, CurrSamplesPerSec=24.680960363420514, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:16,732] [INFO] [logging.py:96:log_dist] [Rank 0] step=30, skipped=2, lr=[0.0009997740697079594, 0.0009997740697079594], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:16,735] [INFO] [timer.py:199:stop] epoch=0/micro_step=60/global_step=30, RunningAvgSamplesPerSec=24.56991418598948, CurrSamplesPerSec=24.51010515674169, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:23,234] [INFO] [logging.py:96:log_dist] [Rank 0] step=40, skipped=2, lr=[0.0009995838996722914, 0.0009995838996722914], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:23,237] [INFO] [timer.py:199:stop] epoch=0/micro_step=80/global_step=40, RunningAvgSamplesPerSec=24.595546797567366, CurrSamplesPerSec=24.69015883130534, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:29,740] [INFO] [logging.py:96:log_dist] [Rank 0] step=50, skipped=2, lr=[0.0009993361385379875, 0.0009993361385379875], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:29,743] [INFO] [timer.py:199:stop] epoch=0/micro_step=100/global_step=50, RunningAvgSamplesPerSec=24.60845630690527, CurrSamplesPerSec=24.60473155053863, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:36,246] [INFO] [logging.py:96:log_dist] [Rank 0] step=60, skipped=2, lr=[0.0009990308148664882, 0.0009990308148664882], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:36,248] [INFO] [timer.py:199:stop] epoch=0/micro_step=120/global_step=60, RunningAvgSamplesPerSec=24.616924725495373, CurrSamplesPerSec=24.658705394215264, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:42,753] [INFO] [logging.py:96:log_dist] [Rank 0] step=70, skipped=2, lr=[0.000998667963854936, 0.000998667963854936], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:42,755] [INFO] [timer.py:199:stop] epoch=0/micro_step=140/global_step=70, RunningAvgSamplesPerSec=24.621950858999046, CurrSamplesPerSec=24.45333928006635, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:49,258] [INFO] [logging.py:96:log_dist] [Rank 0] step=80, skipped=2, lr=[0.0009982476273321175, 0.0009982476273321175], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:49,261] [INFO] [timer.py:199:stop] epoch=0/micro_step=160/global_step=80, RunningAvgSamplesPerSec=24.62638273108732, CurrSamplesPerSec=24.668422267607106, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:40:55,770] [INFO] [logging.py:96:log_dist] [Rank 0] step=90, skipped=2, lr=[0.000997769853753642, 0.000997769853753642], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:40:55,773] [INFO] [timer.py:199:stop] epoch=0/micro_step=180/global_step=90, RunningAvgSamplesPerSec=24.626765716840758, CurrSamplesPerSec=24.671723395557176, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:02,274] [INFO] [logging.py:96:log_dist] [Rank 0] step=100, skipped=2, lr=[0.0009972346981963545, 0.0009972346981963545], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:02,276] [INFO] [timer.py:199:stop] epoch=0/micro_step=200/global_step=100, RunningAvgSamplesPerSec=24.63078623853849, CurrSamplesPerSec=24.63874277502806, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:04,210] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:41:04,211] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 16384.0 to 32768.0
[2023-04-14 08:41:08,777] [INFO] [logging.py:96:log_dist] [Rank 0] step=110, skipped=2, lr=[0.0009966422223519887, 0.0009966422223519887], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:08,780] [INFO] [timer.py:199:stop] epoch=0/micro_step=220/global_step=110, RunningAvgSamplesPerSec=24.633901847968517, CurrSamplesPerSec=24.654040028317173, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:15,280] [INFO] [logging.py:96:log_dist] [Rank 0] step=120, skipped=2, lr=[0.0009959924945200525, 0.0009959924945200525], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:15,283] [INFO] [timer.py:199:stop] epoch=0/micro_step=240/global_step=120, RunningAvgSamplesPerSec=24.636623507185316, CurrSamplesPerSec=24.64968426125361, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:21,796] [INFO] [logging.py:96:log_dist] [Rank 0] step=130, skipped=2, lr=[0.0009952855895999568, 0.0009952855895999568], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:21,797] [INFO] [timer.py:199:stop] epoch=0/micro_step=260/global_step=130, RunningAvgSamplesPerSec=24.635703336345212, CurrSamplesPerSec=24.37553266761686, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:28,296] [INFO] [logging.py:96:log_dist] [Rank 0] step=140, skipped=2, lr=[0.00099452158908238, 0.00099452158908238], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:28,298] [INFO] [timer.py:199:stop] epoch=0/micro_step=280/global_step=140, RunningAvgSamplesPerSec=24.63824453333836, CurrSamplesPerSec=24.64838960026768, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:34,804] [INFO] [logging.py:96:log_dist] [Rank 0] step=150, skipped=2, lr=[0.0009937005810398746, 0.0009937005810398746], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:34,806] [INFO] [timer.py:199:stop] epoch=0/micro_step=300/global_step=150, RunningAvgSamplesPerSec=24.638963243002955, CurrSamplesPerSec=24.644542634201372, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:41,320] [INFO] [logging.py:96:log_dist] [Rank 0] step=160, skipped=2, lr=[0.0009928226601167139, 0.0009928226601167139], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:41,323] [INFO] [timer.py:199:stop] epoch=0/micro_step=320/global_step=160, RunningAvgSamplesPerSec=24.637419780812994, CurrSamplesPerSec=24.647493376172438, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:47,843] [INFO] [logging.py:96:log_dist] [Rank 0] step=170, skipped=2, lr=[0.0009918879275179818, 0.0009918879275179818], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:47,846] [INFO] [timer.py:199:stop] epoch=0/micro_step=340/global_step=170, RunningAvgSamplesPerSec=24.634721563346023, CurrSamplesPerSec=24.487906907158937, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:41:54,354] [INFO] [logging.py:96:log_dist] [Rank 0] step=180, skipped=2, lr=[0.000990896490997906, 0.000990896490997906], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:41:54,357] [INFO] [timer.py:199:stop] epoch=0/micro_step=360/global_step=180, RunningAvgSamplesPerSec=24.634756466148893, CurrSamplesPerSec=24.687987992406963, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:00,864] [INFO] [logging.py:96:log_dist] [Rank 0] step=190, skipped=2, lr=[0.0009898484648474361, 0.0009898484648474361], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:00,866] [INFO] [timer.py:199:stop] epoch=0/micro_step=380/global_step=190, RunningAvgSamplesPerSec=24.635014745920756, CurrSamplesPerSec=24.647611058447524, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:07,382] [INFO] [logging.py:96:log_dist] [Rank 0] step=200, skipped=2, lr=[0.0009887439698810694, 0.0009887439698810694], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:07,385] [INFO] [timer.py:199:stop] epoch=0/micro_step=400/global_step=200, RunningAvgSamplesPerSec=24.633825740844294, CurrSamplesPerSec=24.67899080304979, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:09,323] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:42:09,324] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 32768.0 to 65536.0
[2023-04-14 08:42:13,890] [INFO] [logging.py:96:log_dist] [Rank 0] step=210, skipped=2, lr=[0.0009875831334229218, 0.0009875831334229218], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:13,893] [INFO] [timer.py:199:stop] epoch=0/micro_step=420/global_step=210, RunningAvgSamplesPerSec=24.634414887755113, CurrSamplesPerSec=24.684255768676913, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:20,400] [INFO] [logging.py:96:log_dist] [Rank 0] step=220, skipped=2, lr=[0.0009863660892920514, 0.0009863660892920514], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:20,403] [INFO] [timer.py:199:stop] epoch=0/micro_step=440/global_step=220, RunningAvgSamplesPerSec=24.63478810300769, CurrSamplesPerSec=24.694528915644916, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:26,897] [INFO] [logging.py:96:log_dist] [Rank 0] step=230, skipped=2, lr=[0.0009850929777870322, 0.0009850929777870322], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:26,899] [INFO] [timer.py:199:stop] epoch=0/micro_step=460/global_step=230, RunningAvgSamplesPerSec=24.63712685843032, CurrSamplesPerSec=24.68708888642827, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:33,414] [INFO] [logging.py:96:log_dist] [Rank 0] step=240, skipped=2, lr=[0.0009837639456697802, 0.0009837639456697802], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:33,416] [INFO] [timer.py:199:stop] epoch=0/micro_step=480/global_step=240, RunningAvgSamplesPerSec=24.636211101804335, CurrSamplesPerSec=24.662973711860587, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:39,922] [INFO] [logging.py:96:log_dist] [Rank 0] step=250, skipped=2, lr=[0.0009823791461486347, 0.0009823791461486347], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:39,925] [INFO] [timer.py:199:stop] epoch=0/micro_step=500/global_step=250, RunningAvgSamplesPerSec=24.636530785781275, CurrSamplesPerSec=24.42322367262209, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:46,427] [INFO] [logging.py:96:log_dist] [Rank 0] step=260, skipped=2, lr=[0.0009809387388606977, 0.0009809387388606977], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:46,429] [INFO] [timer.py:199:stop] epoch=0/micro_step=520/global_step=260, RunningAvgSamplesPerSec=24.63744761535955, CurrSamplesPerSec=24.679961929475382, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:52,926] [INFO] [logging.py:96:log_dist] [Rank 0] step=270, skipped=2, lr=[0.00097944288985343, 0.00097944288985343], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:52,929] [INFO] [timer.py:199:stop] epoch=0/micro_step=540/global_step=270, RunningAvgSamplesPerSec=24.63896123142433, CurrSamplesPerSec=24.69295696081992, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:42:59,429] [INFO] [logging.py:96:log_dist] [Rank 0] step=280, skipped=2, lr=[0.000977891771565511, 0.000977891771565511], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:42:59,431] [INFO] [timer.py:199:stop] epoch=0/micro_step=560/global_step=280, RunningAvgSamplesPerSec=24.639926753255008, CurrSamplesPerSec=24.676549710612825, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:05,932] [INFO] [logging.py:96:log_dist] [Rank 0] step=290, skipped=2, lr=[0.0009762855628069589, 0.0009762855628069589], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:05,935] [INFO] [timer.py:199:stop] epoch=0/micro_step=580/global_step=290, RunningAvgSamplesPerSec=24.640777604938364, CurrSamplesPerSec=24.671859450004174, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:12,437] [INFO] [logging.py:96:log_dist] [Rank 0] step=300, skipped=2, lr=[0.0009746244487385189, 0.0009746244487385189], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:12,440] [INFO] [timer.py:199:stop] epoch=0/micro_step=600/global_step=300, RunningAvgSamplesPerSec=24.641424572232626, CurrSamplesPerSec=24.647276119540027, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:14,371] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:43:14,371] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:43:14,998] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 303
[2023-04-14 08:43:14,998] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:43:14,998] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:43:18,900] [INFO] [logging.py:96:log_dist] [Rank 0] step=310, skipped=3, lr=[0.000973082660173131, 0.000973082660173131], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:18,902] [INFO] [timer.py:199:stop] epoch=0/micro_step=620/global_step=310, RunningAvgSamplesPerSec=24.647143951084253, CurrSamplesPerSec=24.57948949539589, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:25,405] [INFO] [logging.py:96:log_dist] [Rank 0] step=320, skipped=3, lr=[0.0009713177587846593, 0.0009713177587846593], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:25,408] [INFO] [timer.py:199:stop] epoch=0/micro_step=640/global_step=320, RunningAvgSamplesPerSec=24.64749858559626, CurrSamplesPerSec=24.679317528643168, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:31,910] [INFO] [logging.py:96:log_dist] [Rank 0] step=330, skipped=3, lr=[0.0009694985247654666, 0.0009694985247654666], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:31,912] [INFO] [timer.py:199:stop] epoch=0/micro_step=660/global_step=330, RunningAvgSamplesPerSec=24.64782630621037, CurrSamplesPerSec=24.577959156168088, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:38,421] [INFO] [logging.py:96:log_dist] [Rank 0] step=340, skipped=3, lr=[0.0009676251678334517, 0.0009676251678334517], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:38,423] [INFO] [timer.py:199:stop] epoch=0/micro_step=680/global_step=340, RunningAvgSamplesPerSec=24.64753553168755, CurrSamplesPerSec=24.491597863855652, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:44,921] [INFO] [logging.py:96:log_dist] [Rank 0] step=350, skipped=3, lr=[0.0009656979039457022, 0.0009656979039457022], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:44,923] [INFO] [timer.py:199:stop] epoch=0/micro_step=700/global_step=350, RunningAvgSamplesPerSec=24.648338585646798, CurrSamplesPerSec=24.61109303928123, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:51,428] [INFO] [logging.py:96:log_dist] [Rank 0] step=360, skipped=3, lr=[0.0009637169552735985, 0.0009637169552735985], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:51,430] [INFO] [timer.py:199:stop] epoch=0/micro_step=720/global_step=360, RunningAvgSamplesPerSec=24.648441289012748, CurrSamplesPerSec=24.700700545956927, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:43:57,927] [INFO] [logging.py:96:log_dist] [Rank 0] step=370, skipped=3, lr=[0.0009616825501772026, 0.0009616825501772026], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:43:57,929] [INFO] [timer.py:199:stop] epoch=0/micro_step=740/global_step=370, RunningAvgSamplesPerSec=24.649288300596588, CurrSamplesPerSec=24.649476019888876, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:04,431] [INFO] [logging.py:96:log_dist] [Rank 0] step=380, skipped=3, lr=[0.0009595949231789334, 0.0009595949231789334], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:04,433] [INFO] [timer.py:199:stop] epoch=0/micro_step=760/global_step=380, RunningAvgSamplesPerSec=24.649653626031096, CurrSamplesPerSec=24.649448858231132, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:10,957] [INFO] [logging.py:96:log_dist] [Rank 0] step=390, skipped=3, lr=[0.0009574543149365311, 0.0009574543149365311], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:10,959] [INFO] [timer.py:199:stop] epoch=0/micro_step=780/global_step=390, RunningAvgSamplesPerSec=24.64794836297907, CurrSamplesPerSec=24.113172497079674, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:17,463] [INFO] [logging.py:96:log_dist] [Rank 0] step=400, skipped=3, lr=[0.0009552609722153152, 0.0009552609722153152], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:17,465] [INFO] [timer.py:199:stop] epoch=0/micro_step=800/global_step=400, RunningAvgSamplesPerSec=24.648052661311077, CurrSamplesPerSec=24.67546997619922, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:20,699] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:44:20,700] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:44:21,327] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 405
[2023-04-14 08:44:21,328] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:44:21,328] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:44:23,931] [INFO] [logging.py:96:log_dist] [Rank 0] step=410, skipped=4, lr=[0.0009532420846376315, 0.0009532420846376315], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:23,933] [INFO] [timer.py:199:stop] epoch=0/micro_step=820/global_step=410, RunningAvgSamplesPerSec=24.65180526592401, CurrSamplesPerSec=24.67538831945664, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:30,446] [INFO] [logging.py:96:log_dist] [Rank 0] step=420, skipped=4, lr=[0.0009509492479940584, 0.0009509492479940584], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:30,449] [INFO] [timer.py:199:stop] epoch=0/micro_step=840/global_step=420, RunningAvgSamplesPerSec=24.650990756103813, CurrSamplesPerSec=24.69955506187507, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:36,962] [INFO] [logging.py:96:log_dist] [Rank 0] step=430, skipped=4, lr=[0.0009486044267636357, 0.0009486044267636357], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:36,965] [INFO] [timer.py:199:stop] epoch=0/micro_step=860/global_step=430, RunningAvgSamplesPerSec=24.650260687638422, CurrSamplesPerSec=24.702264394162203, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:43,468] [INFO] [logging.py:96:log_dist] [Rank 0] step=440, skipped=4, lr=[0.0009462078912529748, 0.0009462078912529748], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:43,470] [INFO] [timer.py:199:stop] epoch=0/micro_step=880/global_step=440, RunningAvgSamplesPerSec=24.650438188915707, CurrSamplesPerSec=24.68972281628747, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:49,987] [INFO] [logging.py:96:log_dist] [Rank 0] step=450, skipped=4, lr=[0.0009437599177302116, 0.0009437599177302116], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:49,990] [INFO] [timer.py:199:stop] epoch=0/micro_step=900/global_step=450, RunningAvgSamplesPerSec=24.649405044016525, CurrSamplesPerSec=24.679980082098822, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:44:56,496] [INFO] [logging.py:96:log_dist] [Rank 0] step=460, skipped=4, lr=[0.0009412607883931606, 0.0009412607883931606], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:44:56,497] [INFO] [timer.py:199:stop] epoch=0/micro_step=920/global_step=460, RunningAvgSamplesPerSec=24.64935703487508, CurrSamplesPerSec=24.449677660281807, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:02,999] [INFO] [logging.py:96:log_dist] [Rank 0] step=470, skipped=4, lr=[0.0009387107913367831, 0.0009387107913367831], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:03,001] [INFO] [timer.py:199:stop] epoch=0/micro_step=940/global_step=470, RunningAvgSamplesPerSec=24.649570525126055, CurrSamplesPerSec=24.613241348451425, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:09,504] [INFO] [logging.py:96:log_dist] [Rank 0] step=480, skipped=4, lr=[0.0009361102205199761, 0.0009361102205199761], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:09,506] [INFO] [timer.py:199:stop] epoch=0/micro_step=960/global_step=480, RunningAvgSamplesPerSec=24.649787966538355, CurrSamplesPerSec=24.672866299551536, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:16,021] [INFO] [logging.py:96:log_dist] [Rank 0] step=490, skipped=4, lr=[0.0009334593757316845, 0.0009334593757316845], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:16,023] [INFO] [timer.py:199:stop] epoch=0/micro_step=980/global_step=490, RunningAvgSamplesPerSec=24.649018305713003, CurrSamplesPerSec=24.67156920233125, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:22,523] [INFO] [logging.py:96:log_dist] [Rank 0] step=500, skipped=4, lr=[0.0009307585625563433, 0.0009307585625563433], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:22,525] [INFO] [timer.py:199:stop] epoch=0/micro_step=1000/global_step=500, RunningAvgSamplesPerSec=24.64943198676302, CurrSamplesPerSec=24.675560706547056, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:27,073] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:45:27,073] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:45:27,700] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 507
[2023-04-14 08:45:27,700] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:45:27,700] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:45:29,000] [INFO] [logging.py:96:log_dist] [Rank 0] step=510, skipped=5, lr=[0.0009282853649355565, 0.0009282853649355565], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:29,003] [INFO] [timer.py:199:stop] epoch=0/micro_step=1020/global_step=510, RunningAvgSamplesPerSec=24.651683140266123, CurrSamplesPerSec=24.657156115765304, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:35,508] [INFO] [logging.py:96:log_dist] [Rank 0] step=520, skipped=5, lr=[0.0009254904743114072, 0.0009254904743114072], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:35,511] [INFO] [timer.py:199:stop] epoch=0/micro_step=1040/global_step=520, RunningAvgSamplesPerSec=24.651589219536255, CurrSamplesPerSec=24.66172296893805, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:42,018] [INFO] [logging.py:96:log_dist] [Rank 0] step=530, skipped=5, lr=[0.0009226465339402767, 0.0009226465339402767], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:42,021] [INFO] [timer.py:199:stop] epoch=0/micro_step=1060/global_step=530, RunningAvgSamplesPerSec=24.6513551037351, CurrSamplesPerSec=24.678609634124502, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:48,520] [INFO] [logging.py:96:log_dist] [Rank 0] step=540, skipped=5, lr=[0.0009197538716662982, 0.0009197538716662982], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:48,522] [INFO] [timer.py:199:stop] epoch=0/micro_step=1080/global_step=540, RunningAvgSamplesPerSec=24.651705802340025, CurrSamplesPerSec=24.63615588698821, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:45:55,022] [INFO] [logging.py:96:log_dist] [Rank 0] step=550, skipped=5, lr=[0.0009168128209501743, 0.0009168128209501743], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:45:55,024] [INFO] [timer.py:199:stop] epoch=0/micro_step=1100/global_step=550, RunningAvgSamplesPerSec=24.652021831885328, CurrSamplesPerSec=24.698773284536085, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:01,518] [INFO] [logging.py:96:log_dist] [Rank 0] step=560, skipped=5, lr=[0.0009138237208307371, 0.0009138237208307371], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:01,521] [INFO] [timer.py:199:stop] epoch=0/micro_step=1120/global_step=560, RunningAvgSamplesPerSec=24.652660449518525, CurrSamplesPerSec=24.69124893622736, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:08,038] [INFO] [logging.py:96:log_dist] [Rank 0] step=570, skipped=5, lr=[0.0009107869158858643, 0.0009107869158858643], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:08,040] [INFO] [timer.py:199:stop] epoch=0/micro_step=1140/global_step=570, RunningAvgSamplesPerSec=24.6518168655344, CurrSamplesPerSec=24.65824330833908, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:14,551] [INFO] [logging.py:96:log_dist] [Rank 0] step=580, skipped=5, lr=[0.0009077027561927568, 0.0009077027561927568], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:14,554] [INFO] [timer.py:199:stop] epoch=0/micro_step=1160/global_step=580, RunningAvgSamplesPerSec=24.6513520108345, CurrSamplesPerSec=24.688705507797252, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:21,056] [INFO] [logging.py:96:log_dist] [Rank 0] step=590, skipped=5, lr=[0.0009045715972875818, 0.0009045715972875818], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:21,059] [INFO] [timer.py:199:stop] epoch=0/micro_step=1180/global_step=590, RunningAvgSamplesPerSec=24.6514581509846, CurrSamplesPerSec=24.69240273545383, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:27,555] [INFO] [logging.py:96:log_dist] [Rank 0] step=600, skipped=5, lr=[0.0009013938001244885, 0.0009013938001244885], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:27,558] [INFO] [timer.py:199:stop] epoch=0/micro_step=1200/global_step=600, RunningAvgSamplesPerSec=24.651887892751148, CurrSamplesPerSec=24.694474393630184, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:33,393] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:46:33,393] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:46:34,021] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 609
[2023-04-14 08:46:34,021] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:46:34,021] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:46:34,022] [INFO] [logging.py:96:log_dist] [Rank 0] step=610, skipped=6, lr=[0.0008984942096289488, 0.0008984942096289488], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:34,023] [INFO] [timer.py:199:stop] epoch=0/micro_step=1220/global_step=610, RunningAvgSamplesPerSec=24.65450007960378, CurrSamplesPerSec=26.258218401958267, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:40,539] [INFO] [logging.py:96:log_dist] [Rank 0] step=620, skipped=6, lr=[0.0008952288134259443, 0.0008952288134259443], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:40,542] [INFO] [timer.py:199:stop] epoch=0/micro_step=1240/global_step=620, RunningAvgSamplesPerSec=24.653710571624405, CurrSamplesPerSec=24.633289236637747, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:47,076] [INFO] [logging.py:96:log_dist] [Rank 0] step=630, skipped=6, lr=[0.0008919178559836909, 0.0008919178559836909], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:47,078] [INFO] [timer.py:199:stop] epoch=0/micro_step=1260/global_step=630, RunningAvgSamplesPerSec=24.65189811734757, CurrSamplesPerSec=24.702728131402626, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:46:53,582] [INFO] [logging.py:96:log_dist] [Rank 0] step=640, skipped=6, lr=[0.0008885617189831772, 0.0008885617189831772], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:46:53,584] [INFO] [timer.py:199:stop] epoch=0/micro_step=1280/global_step=640, RunningAvgSamplesPerSec=24.651944063824548, CurrSamplesPerSec=24.671941083392646, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:00,091] [INFO] [logging.py:96:log_dist] [Rank 0] step=650, skipped=6, lr=[0.0008851607893136064, 0.0008851607893136064], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:00,094] [INFO] [timer.py:199:stop] epoch=0/micro_step=1300/global_step=650, RunningAvgSamplesPerSec=24.651725405691558, CurrSamplesPerSec=24.695510353080753, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:06,594] [INFO] [logging.py:96:log_dist] [Rank 0] step=660, skipped=6, lr=[0.0008817154590277976, 0.0008817154590277976], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:06,596] [INFO] [timer.py:199:stop] epoch=0/micro_step=1320/global_step=660, RunningAvgSamplesPerSec=24.651980515887857, CurrSamplesPerSec=24.662874010307792, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:13,100] [INFO] [logging.py:96:log_dist] [Rank 0] step=670, skipped=6, lr=[0.0008782261252969894, 0.0008782261252969894], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:13,103] [INFO] [timer.py:199:stop] epoch=0/micro_step=1340/global_step=670, RunningAvgSamplesPerSec=24.651950001253887, CurrSamplesPerSec=24.67452642013293, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:19,617] [INFO] [logging.py:96:log_dist] [Rank 0] step=680, skipped=6, lr=[0.0008746931903650557, 0.0008746931903650557], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:19,619] [INFO] [timer.py:199:stop] epoch=0/micro_step=1360/global_step=680, RunningAvgSamplesPerSec=24.651437492849734, CurrSamplesPerSec=24.51611328820675, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:26,117] [INFO] [logging.py:96:log_dist] [Rank 0] step=690, skipped=6, lr=[0.000871117061502135, 0.000871117061502135], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:26,119] [INFO] [timer.py:199:stop] epoch=0/micro_step=1380/global_step=690, RunningAvgSamplesPerSec=24.651805249406713, CurrSamplesPerSec=24.687061641814875, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:32,627] [INFO] [logging.py:96:log_dist] [Rank 0] step=700, skipped=6, lr=[0.0008674981509576818, 0.0008674981509576818], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:32,629] [INFO] [timer.py:199:stop] epoch=0/micro_step=1400/global_step=700, RunningAvgSamplesPerSec=24.65158697768912, CurrSamplesPerSec=24.606644163055602, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:39,140] [INFO] [logging.py:96:log_dist] [Rank 0] step=710, skipped=6, lr=[0.0008638368759129432, 0.0008638368759129432], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:39,142] [INFO] [timer.py:199:stop] epoch=0/micro_step=1420/global_step=710, RunningAvgSamplesPerSec=24.651258960690207, CurrSamplesPerSec=24.623319937228, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:39,775] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:47:39,775] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:47:40,402] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 711
[2023-04-14 08:47:40,403] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:47:40,403] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:47:45,606] [INFO] [logging.py:96:log_dist] [Rank 0] step=720, skipped=7, lr=[0.0008605058554614858, 0.0008605058554614858], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:45,609] [INFO] [timer.py:199:stop] epoch=0/micro_step=1440/global_step=720, RunningAvgSamplesPerSec=24.65340096640429, CurrSamplesPerSec=24.666971501832688, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:52,110] [INFO] [logging.py:96:log_dist] [Rank 0] step=730, skipped=7, lr=[0.0008567652546523133, 0.0008567652546523133], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:52,113] [INFO] [timer.py:199:stop] epoch=0/micro_step=1460/global_step=730, RunningAvgSamplesPerSec=24.653543270614605, CurrSamplesPerSec=24.654936728578754, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:47:58,610] [INFO] [logging.py:96:log_dist] [Rank 0] step=740, skipped=7, lr=[0.0008529835266110901, 0.0008529835266110901], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:47:58,613] [INFO] [timer.py:199:stop] epoch=0/micro_step=1480/global_step=740, RunningAvgSamplesPerSec=24.653835161920437, CurrSamplesPerSec=24.685699487811803, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:05,110] [INFO] [logging.py:96:log_dist] [Rank 0] step=750, skipped=7, lr=[0.0008491611072883598, 0.0008491611072883598], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:05,112] [INFO] [timer.py:199:stop] epoch=0/micro_step=1500/global_step=750, RunningAvgSamplesPerSec=24.654149175915716, CurrSamplesPerSec=24.687315927211834, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:11,621] [INFO] [logging.py:96:log_dist] [Rank 0] step=760, skipped=7, lr=[0.0008452984373254808, 0.0008452984373254808], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:11,622] [INFO] [timer.py:199:stop] epoch=0/micro_step=1520/global_step=760, RunningAvgSamplesPerSec=24.653989197898042, CurrSamplesPerSec=24.549933676377304, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:18,116] [INFO] [logging.py:96:log_dist] [Rank 0] step=770, skipped=7, lr=[0.0008413959620038305, 0.0008413959620038305], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:18,118] [INFO] [timer.py:199:stop] epoch=0/micro_step=1540/global_step=770, RunningAvgSamplesPerSec=24.654421181644175, CurrSamplesPerSec=24.68378364560387, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:24,626] [INFO] [logging.py:96:log_dist] [Rank 0] step=780, skipped=7, lr=[0.0008374541311934732, 0.0008374541311934732], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:24,628] [INFO] [timer.py:199:stop] epoch=0/micro_step=1560/global_step=780, RunningAvgSamplesPerSec=24.654289850264774, CurrSamplesPerSec=24.617521031260857, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:31,121] [INFO] [logging.py:96:log_dist] [Rank 0] step=790, skipped=7, lr=[0.0008334733993013006, 0.0008334733993013006], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:31,123] [INFO] [timer.py:199:stop] epoch=0/micro_step=1580/global_step=790, RunningAvgSamplesPerSec=24.654745019679982, CurrSamplesPerSec=24.698909637485563, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:37,631] [INFO] [logging.py:96:log_dist] [Rank 0] step=800, skipped=7, lr=[0.0008294542252186486, 0.0008294542252186486], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:37,634] [INFO] [timer.py:199:stop] epoch=0/micro_step=1600/global_step=800, RunningAvgSamplesPerSec=24.654536688824397, CurrSamplesPerSec=24.689277733507275, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:44,154] [INFO] [logging.py:96:log_dist] [Rank 0] step=810, skipped=7, lr=[0.0008253970722683967, 0.0008253970722683967], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:44,156] [INFO] [timer.py:199:stop] epoch=0/micro_step=1620/global_step=810, RunningAvgSamplesPerSec=24.653759268005736, CurrSamplesPerSec=24.695864780215594, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:46,087] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:48:46,087] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:48:46,715] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 813
[2023-04-14 08:48:46,715] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:48:46,715] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:48:50,624] [INFO] [logging.py:96:log_dist] [Rank 0] step=820, skipped=8, lr=[0.0008217135491466636, 0.0008217135491466636], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:50,627] [INFO] [timer.py:199:stop] epoch=0/micro_step=1640/global_step=820, RunningAvgSamplesPerSec=24.655370120545186, CurrSamplesPerSec=24.68273959140198, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:48:57,496] [INFO] [logging.py:96:log_dist] [Rank 0] step=830, skipped=8, lr=[0.0008175855284393494, 0.0008175855284393494], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:48:57,498] [INFO] [timer.py:199:stop] epoch=0/micro_step=1660/global_step=830, RunningAvgSamplesPerSec=24.638592478845037, CurrSamplesPerSec=24.671233611972387, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:03,995] [INFO] [logging.py:96:log_dist] [Rank 0] step=840, skipped=8, lr=[0.0008134208970656271, 0.0008134208970656271], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:03,997] [INFO] [timer.py:199:stop] epoch=0/micro_step=1680/global_step=840, RunningAvgSamplesPerSec=24.639088309497943, CurrSamplesPerSec=24.689005241765777, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:10,498] [INFO] [logging.py:96:log_dist] [Rank 0] step=850, skipped=8, lr=[0.0008092201351164212, 0.0008092201351164212], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:10,500] [INFO] [timer.py:199:stop] epoch=0/micro_step=1700/global_step=850, RunningAvgSamplesPerSec=24.639364501417088, CurrSamplesPerSec=24.656105254069466, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:16,998] [INFO] [logging.py:96:log_dist] [Rank 0] step=860, skipped=8, lr=[0.0008049837268477212, 0.0008049837268477212], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:17,001] [INFO] [timer.py:199:stop] epoch=0/micro_step=1720/global_step=860, RunningAvgSamplesPerSec=24.639770162369015, CurrSamplesPerSec=24.692693473527115, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:23,516] [INFO] [logging.py:96:log_dist] [Rank 0] step=870, skipped=8, lr=[0.0008007121606247583, 0.0008007121606247583], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:23,518] [INFO] [timer.py:199:stop] epoch=0/micro_step=1740/global_step=870, RunningAvgSamplesPerSec=24.639433907711002, CurrSamplesPerSec=24.42727747905938, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:30,017] [INFO] [logging.py:96:log_dist] [Rank 0] step=880, skipped=8, lr=[0.0007964059288657061, 0.0007964059288657061], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:30,020] [INFO] [timer.py:199:stop] epoch=0/micro_step=1760/global_step=880, RunningAvgSamplesPerSec=24.63978525362981, CurrSamplesPerSec=24.64057020515063, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:36,528] [INFO] [logging.py:96:log_dist] [Rank 0] step=890, skipped=8, lr=[0.0007920655279849172, 0.0007920655279849172], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:36,531] [INFO] [timer.py:199:stop] epoch=0/micro_step=1780/global_step=890, RunningAvgSamplesPerSec=24.63971343618263, CurrSamplesPerSec=24.61463163072219, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:43,052] [INFO] [logging.py:96:log_dist] [Rank 0] step=900, skipped=8, lr=[0.0007876914583356964, 0.0007876914583356964], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:43,055] [INFO] [timer.py:199:stop] epoch=0/micro_step=1800/global_step=900, RunningAvgSamplesPerSec=24.639123569700974, CurrSamplesPerSec=24.67592363461071, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:49,558] [INFO] [logging.py:96:log_dist] [Rank 0] step=910, skipped=8, lr=[0.0007832842241526212, 0.0007832842241526212], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:49,560] [INFO] [timer.py:199:stop] epoch=0/micro_step=1820/global_step=910, RunningAvgSamplesPerSec=24.63932734264525, CurrSamplesPerSec=24.66465969654347, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:49:52,797] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:49:52,797] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:49:53,430] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 915
[2023-04-14 08:49:53,431] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:49:53,431] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:49:56,031] [INFO] [logging.py:96:log_dist] [Rank 0] step=920, skipped=9, lr=[0.0007792897775430632, 0.0007792897775430632], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:49:56,034] [INFO] [timer.py:199:stop] epoch=0/micro_step=1840/global_step=920, RunningAvgSamplesPerSec=24.64084426026159, CurrSamplesPerSec=24.677166744132684, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:02,539] [INFO] [logging.py:96:log_dist] [Rank 0] step=930, skipped=9, lr=[0.0007748209335573322, 0.0007748209335573322], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:02,541] [INFO] [timer.py:199:stop] epoch=0/micro_step=1860/global_step=930, RunningAvgSamplesPerSec=24.64091561810867, CurrSamplesPerSec=24.673528506661892, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:09,050] [INFO] [logging.py:96:log_dist] [Rank 0] step=940, skipped=9, lr=[0.0007703204087277988, 0.0007703204087277988], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:09,053] [INFO] [timer.py:199:stop] epoch=0/micro_step=1880/global_step=940, RunningAvgSamplesPerSec=24.64085398477739, CurrSamplesPerSec=24.670281310668102, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:15,563] [INFO] [logging.py:96:log_dist] [Rank 0] step=950, skipped=9, lr=[0.0007657887218665573, 0.0007657887218665573], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:15,566] [INFO] [timer.py:199:stop] epoch=0/micro_step=1900/global_step=950, RunningAvgSamplesPerSec=24.64073750600058, CurrSamplesPerSec=24.706274933106748, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:22,076] [INFO] [logging.py:96:log_dist] [Rank 0] step=960, skipped=9, lr=[0.0007612263953780034, 0.0007612263953780034], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:22,078] [INFO] [timer.py:199:stop] epoch=0/micro_step=1920/global_step=960, RunningAvgSamplesPerSec=24.640582879009013, CurrSamplesPerSec=24.6557429087362, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:28,589] [INFO] [logging.py:96:log_dist] [Rank 0] step=970, skipped=9, lr=[0.0007566339551986109, 0.0007566339551986109], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:28,591] [INFO] [timer.py:199:stop] epoch=0/micro_step=1940/global_step=970, RunningAvgSamplesPerSec=24.640467329305533, CurrSamplesPerSec=24.694892401895846, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:35,087] [INFO] [logging.py:96:log_dist] [Rank 0] step=980, skipped=9, lr=[0.0007520119307363045, 0.0007520119307363045], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:35,089] [INFO] [timer.py:199:stop] epoch=0/micro_step=1960/global_step=980, RunningAvgSamplesPerSec=24.64088208381276, CurrSamplesPerSec=24.701891598417227, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:41,601] [INFO] [logging.py:96:log_dist] [Rank 0] step=990, skipped=9, lr=[0.000747360854809429, 0.000747360854809429], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:41,603] [INFO] [timer.py:199:stop] epoch=0/micro_step=1980/global_step=990, RunningAvgSamplesPerSec=24.64071690503538, CurrSamplesPerSec=24.67571494966966, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:48,097] [INFO] [logging.py:96:log_dist] [Rank 0] step=1000, skipped=9, lr=[0.0007426812635853285, 0.0007426812635853285], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:48,099] [INFO] [timer.py:199:stop] epoch=0/micro_step=2000/global_step=1000, RunningAvgSamplesPerSec=24.641205258792617, CurrSamplesPerSec=24.688968909988038, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:54,603] [INFO] [logging.py:96:log_dist] [Rank 0] step=1010, skipped=9, lr=[0.0007379736965185368, 0.0007379736965185368], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:50:54,605] [INFO] [timer.py:199:stop] epoch=0/micro_step=2020/global_step=1010, RunningAvgSamplesPerSec=24.641320610939218, CurrSamplesPerSec=24.686607573777824, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:50:59,139] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:50:59,139] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:50:59,766] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1017
[2023-04-14 08:50:59,766] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:50:59,766] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:51:01,065] [INFO] [logging.py:96:log_dist] [Rank 0] step=1020, skipped=10, lr=[0.0007337134152719311, 0.0007337134152719311], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:01,068] [INFO] [timer.py:199:stop] epoch=0/micro_step=2040/global_step=1020, RunningAvgSamplesPerSec=24.643054469938516, CurrSamplesPerSec=24.665466512690195, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:07,568] [INFO] [logging.py:96:log_dist] [Rank 0] step=1030, skipped=10, lr=[0.0007289541918012085, 0.0007289541918012085], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:07,571] [INFO] [timer.py:199:stop] epoch=0/micro_step=2060/global_step=1030, RunningAvgSamplesPerSec=24.643259780889323, CurrSamplesPerSec=24.69184853496599, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:14,069] [INFO] [logging.py:96:log_dist] [Rank 0] step=1040, skipped=10, lr=[0.000724168574918985, 0.000724168574918985], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:14,071] [INFO] [timer.py:199:stop] epoch=0/micro_step=2080/global_step=1040, RunningAvgSamplesPerSec=24.643566568751176, CurrSamplesPerSec=24.693302228542937, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:20,580] [INFO] [logging.py:96:log_dist] [Rank 0] step=1050, skipped=10, lr=[0.0007193571163022348, 0.0007193571163022348], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:20,582] [INFO] [timer.py:199:stop] epoch=0/micro_step=2100/global_step=1050, RunningAvgSamplesPerSec=24.643485162188377, CurrSamplesPerSec=24.644624086919766, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:27,090] [INFO] [logging.py:96:log_dist] [Rank 0] step=1060, skipped=10, lr=[0.0007145203706069183, 0.0007145203706069183], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:27,093] [INFO] [timer.py:199:stop] epoch=0/micro_step=2120/global_step=1060, RunningAvgSamplesPerSec=24.643417492097715, CurrSamplesPerSec=24.66811396560153, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:33,598] [INFO] [logging.py:96:log_dist] [Rank 0] step=1070, skipped=10, lr=[0.000709658895404043, 0.000709658895404043], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:33,601] [INFO] [timer.py:199:stop] epoch=0/micro_step=2140/global_step=1070, RunningAvgSamplesPerSec=24.64346598697257, CurrSamplesPerSec=24.69227553969964, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:40,105] [INFO] [logging.py:96:log_dist] [Rank 0] step=1080, skipped=10, lr=[0.0007047732511153885, 0.0007047732511153885], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:40,108] [INFO] [timer.py:199:stop] epoch=0/micro_step=2160/global_step=1080, RunningAvgSamplesPerSec=24.64350684974393, CurrSamplesPerSec=24.609017298477486, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:46,614] [INFO] [logging.py:96:log_dist] [Rank 0] step=1090, skipped=10, lr=[0.0006998640009489009, 0.0006998640009489009], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:46,616] [INFO] [timer.py:199:stop] epoch=0/micro_step=2180/global_step=1090, RunningAvgSamplesPerSec=24.643530808738763, CurrSamplesPerSec=24.649657099136935, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:53,111] [INFO] [logging.py:96:log_dist] [Rank 0] step=1100, skipped=10, lr=[0.000694931710833768, 0.000694931710833768], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:53,113] [INFO] [timer.py:199:stop] epoch=0/micro_step=2200/global_step=1100, RunningAvgSamplesPerSec=24.64391953294995, CurrSamplesPerSec=24.652744909752013, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:51:59,616] [INFO] [logging.py:96:log_dist] [Rank 0] step=1110, skipped=10, lr=[0.00068997694935518, 0.00068997694935518], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:51:59,619] [INFO] [timer.py:199:stop] epoch=0/micro_step=2220/global_step=1110, RunningAvgSamplesPerSec=24.64403959473146, CurrSamplesPerSec=24.707912262047476, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:05,460] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:52:05,461] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:52:06,089] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1119
[2023-04-14 08:52:06,089] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:52:06,089] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:52:06,090] [INFO] [logging.py:96:log_dist] [Rank 0] step=1120, skipped=11, lr=[0.0006854989230329036, 0.0006854989230329036], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:06,091] [INFO] [timer.py:199:stop] epoch=0/micro_step=2240/global_step=1120, RunningAvgSamplesPerSec=24.645277573833134, CurrSamplesPerSec=26.2436062633767, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:12,594] [INFO] [logging.py:96:log_dist] [Rank 0] step=1130, skipped=11, lr=[0.0006805030416406152, 0.0006805030416406152], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:12,596] [INFO] [timer.py:199:stop] epoch=0/micro_step=2260/global_step=1130, RunningAvgSamplesPerSec=24.645377333446586, CurrSamplesPerSec=24.617286242148545, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:19,100] [INFO] [logging.py:96:log_dist] [Rank 0] step=1140, skipped=11, lr=[0.0006754863521949005, 0.0006754863521949005], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:19,103] [INFO] [timer.py:199:stop] epoch=0/micro_step=2280/global_step=1140, RunningAvgSamplesPerSec=24.645437570013403, CurrSamplesPerSec=24.635007338133406, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:25,604] [INFO] [logging.py:96:log_dist] [Rank 0] step=1150, skipped=11, lr=[0.0006704494330103469, 0.0006704494330103469], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:25,607] [INFO] [timer.py:199:stop] epoch=0/micro_step=2300/global_step=1150, RunningAvgSamplesPerSec=24.645564647817476, CurrSamplesPerSec=24.651685368494018, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:32,110] [INFO] [logging.py:96:log_dist] [Rank 0] step=1160, skipped=11, lr=[0.0006653928647335877, 0.0006653928647335877], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:32,112] [INFO] [timer.py:199:stop] epoch=0/micro_step=2320/global_step=1160, RunningAvgSamplesPerSec=24.64566530985825, CurrSamplesPerSec=24.635152031305637, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:38,610] [INFO] [logging.py:96:log_dist] [Rank 0] step=1170, skipped=11, lr=[0.0006603172302763677, 0.0006603172302763677], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:38,613] [INFO] [timer.py:199:stop] epoch=0/micro_step=2340/global_step=1170, RunningAvgSamplesPerSec=24.645913671728916, CurrSamplesPerSec=24.68116005990367, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:45,128] [INFO] [logging.py:96:log_dist] [Rank 0] step=1180, skipped=11, lr=[0.0006552231147483447, 0.0006552231147483447], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:45,130] [INFO] [timer.py:199:stop] epoch=0/micro_step=2360/global_step=1180, RunningAvgSamplesPerSec=24.645656705452616, CurrSamplesPerSec=24.452163165841743, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:51,624] [INFO] [logging.py:96:log_dist] [Rank 0] step=1190, skipped=11, lr=[0.0006501111053896399, 0.0006501111053896399], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:51,627] [INFO] [timer.py:199:stop] epoch=0/micro_step=2380/global_step=1190, RunningAvgSamplesPerSec=24.645998717037106, CurrSamplesPerSec=24.6983824144246, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:52:58,134] [INFO] [logging.py:96:log_dist] [Rank 0] step=1200, skipped=11, lr=[0.0006449817915031419, 0.0006449817915031419], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:52:58,137] [INFO] [timer.py:199:stop] epoch=0/micro_step=2400/global_step=1200, RunningAvgSamplesPerSec=24.645932624379306, CurrSamplesPerSec=24.652264935334006, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:04,652] [INFO] [logging.py:96:log_dist] [Rank 0] step=1210, skipped=11, lr=[0.0006398357643865731, 0.0006398357643865731], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:04,655] [INFO] [timer.py:199:stop] epoch=0/micro_step=2420/global_step=1210, RunningAvgSamplesPerSec=24.645619873444783, CurrSamplesPerSec=24.715037168936824, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:11,154] [INFO] [logging.py:96:log_dist] [Rank 0] step=1220, skipped=11, lr=[0.0006346736172643257, 0.0006346736172643257], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:11,157] [INFO] [timer.py:199:stop] epoch=0/micro_step=2440/global_step=1220, RunningAvgSamplesPerSec=24.645815206809925, CurrSamplesPerSec=24.679943776878645, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:11,788] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:53:11,788] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:53:12,416] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1221
[2023-04-14 08:53:12,417] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:53:12,417] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:53:17,624] [INFO] [logging.py:96:log_dist] [Rank 0] step=1230, skipped=12, lr=[0.0006300143940479881, 0.0006300143940479881], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:17,627] [INFO] [timer.py:199:stop] epoch=0/micro_step=2460/global_step=1230, RunningAvgSamplesPerSec=24.646966864219827, CurrSamplesPerSec=24.676549710612825, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:24,142] [INFO] [logging.py:96:log_dist] [Rank 0] step=1240, skipped=12, lr=[0.0006248232598483196, 0.0006248232598483196], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:24,145] [INFO] [timer.py:199:stop] epoch=0/micro_step=2480/global_step=1240, RunningAvgSamplesPerSec=24.64666142072051, CurrSamplesPerSec=24.68116005990367, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:30,644] [INFO] [logging.py:96:log_dist] [Rank 0] step=1250, skipped=12, lr=[0.0006196177362564495, 0.0006196177362564495], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:30,647] [INFO] [timer.py:199:stop] epoch=0/micro_step=2500/global_step=1250, RunningAvgSamplesPerSec=24.64684625254397, CurrSamplesPerSec=24.70809420088157, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:37,146] [INFO] [logging.py:96:log_dist] [Rank 0] step=1260, skipped=12, lr=[0.0006143984233554125, 0.0006143984233554125], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:37,149] [INFO] [timer.py:199:stop] epoch=0/micro_step=2520/global_step=1260, RunningAvgSamplesPerSec=24.647024003891147, CurrSamplesPerSec=24.671415011032657, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:43,653] [INFO] [logging.py:96:log_dist] [Rank 0] step=1270, skipped=12, lr=[0.0006091659228178492, 0.0006091659228178492], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:43,654] [INFO] [timer.py:199:stop] epoch=0/micro_step=2540/global_step=1270, RunningAvgSamplesPerSec=24.647126957717152, CurrSamplesPerSec=24.6308481361939, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:50,155] [INFO] [logging.py:96:log_dist] [Rank 0] step=1280, skipped=12, lr=[0.0006039208378366462, 0.0006039208378366462], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:50,157] [INFO] [timer.py:199:stop] epoch=0/micro_step=2560/global_step=1280, RunningAvgSamplesPerSec=24.647242665815536, CurrSamplesPerSec=24.682149512876975, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:53:56,665] [INFO] [logging.py:96:log_dist] [Rank 0] step=1290, skipped=12, lr=[0.0005986637730554013, 0.0005986637730554013], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:53:56,668] [INFO] [timer.py:199:stop] epoch=0/micro_step=2580/global_step=1290, RunningAvgSamplesPerSec=24.6471405539605, CurrSamplesPerSec=24.653541889688686, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:03,168] [INFO] [logging.py:96:log_dist] [Rank 0] step=1300, skipped=12, lr=[0.0005933953344987214, 0.0005933953344987214], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:03,170] [INFO] [timer.py:199:stop] epoch=0/micro_step=2600/global_step=1300, RunningAvgSamplesPerSec=24.647279759979778, CurrSamplesPerSec=24.704201294608936, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:09,685] [INFO] [logging.py:96:log_dist] [Rank 0] step=1310, skipped=12, lr=[0.000588116129502361, 0.000588116129502361], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:09,688] [INFO] [timer.py:199:stop] epoch=0/micro_step=2620/global_step=1310, RunningAvgSamplesPerSec=24.647013613139745, CurrSamplesPerSec=24.680651747939017, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:16,203] [INFO] [logging.py:96:log_dist] [Rank 0] step=1320, skipped=12, lr=[0.000582826766643209, 0.000582826766643209], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:16,206] [INFO] [timer.py:199:stop] epoch=0/micro_step=2640/global_step=1320, RunningAvgSamplesPerSec=24.646725941392546, CurrSamplesPerSec=24.691875790146042, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:18,136] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:54:18,136] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:54:18,763] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1323
[2023-04-14 08:54:18,763] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:54:18,764] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:54:22,660] [INFO] [logging.py:96:log_dist] [Rank 0] step=1330, skipped=13, lr=[0.0005780581590308367, 0.0005780581590308367], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:22,663] [INFO] [timer.py:199:stop] epoch=0/micro_step=2660/global_step=1330, RunningAvgSamplesPerSec=24.64815493093632, CurrSamplesPerSec=24.706320411535998, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:29,176] [INFO] [logging.py:96:log_dist] [Rank 0] step=1340, skipped=13, lr=[0.0005727511769989338, 0.0005727511769989338], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:29,178] [INFO] [timer.py:199:stop] epoch=0/micro_step=2680/global_step=1340, RunningAvgSamplesPerSec=24.647952894144726, CurrSamplesPerSec=24.68316628114275, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:35,691] [INFO] [logging.py:96:log_dist] [Rank 0] step=1350, skipped=13, lr=[0.0005674358083472597, 0.0005674358083472597], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:35,692] [INFO] [timer.py:199:stop] epoch=0/micro_step=2700/global_step=1350, RunningAvgSamplesPerSec=24.647756994356687, CurrSamplesPerSec=24.531402078716027, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:42,210] [INFO] [logging.py:96:log_dist] [Rank 0] step=1360, skipped=13, lr=[0.0005621126658215823, 0.0005621126658215823], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:42,212] [INFO] [timer.py:199:stop] epoch=0/micro_step=2720/global_step=1360, RunningAvgSamplesPerSec=24.64741378428855, CurrSamplesPerSec=24.65183025764923, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:48,728] [INFO] [logging.py:96:log_dist] [Rank 0] step=1370, skipped=13, lr=[0.0005567823630638272, 0.0005567823630638272], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:48,731] [INFO] [timer.py:199:stop] epoch=0/micro_step=2740/global_step=1370, RunningAvgSamplesPerSec=24.64712206631567, CurrSamplesPerSec=24.69048585267503, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:54:55,228] [INFO] [logging.py:96:log_dist] [Rank 0] step=1380, skipped=13, lr=[0.0005514455145413385, 0.0005514455145413385], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:54:55,230] [INFO] [timer.py:199:stop] epoch=0/micro_step=2760/global_step=1380, RunningAvgSamplesPerSec=24.647332075632313, CurrSamplesPerSec=24.687924417105332, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:01,735] [INFO] [logging.py:96:log_dist] [Rank 0] step=1390, skipped=13, lr=[0.0005461027354760436, 0.0005461027354760436], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:01,738] [INFO] [timer.py:199:stop] epoch=0/micro_step=2780/global_step=1390, RunningAvgSamplesPerSec=24.647347781456915, CurrSamplesPerSec=24.670771056442284, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:08,236] [INFO] [logging.py:96:log_dist] [Rank 0] step=1400, skipped=13, lr=[0.0005407546417735316, 0.0005407546417735316], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:08,238] [INFO] [timer.py:199:stop] epoch=0/micro_step=2800/global_step=1400, RunningAvgSamplesPerSec=24.647544278442602, CurrSamplesPerSec=24.684709750202398, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:14,765] [INFO] [logging.py:96:log_dist] [Rank 0] step=1410, skipped=13, lr=[0.0005354018499520535, 0.0005354018499520535], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:14,768] [INFO] [timer.py:199:stop] epoch=0/micro_step=2820/global_step=1410, RunningAvgSamplesPerSec=24.64697444057518, CurrSamplesPerSec=24.641619743004597, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:21,261] [INFO] [logging.py:96:log_dist] [Rank 0] step=1420, skipped=13, lr=[0.0005300449770714501, 0.0005300449770714501], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:21,263] [INFO] [timer.py:199:stop] epoch=0/micro_step=2840/global_step=1420, RunningAvgSamplesPerSec=24.64727307445048, CurrSamplesPerSec=24.692066578091133, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:24,495] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:55:24,496] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:55:25,123] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1425
[2023-04-14 08:55:25,123] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:55:25,123] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:55:27,728] [INFO] [logging.py:96:log_dist] [Rank 0] step=1430, skipped=14, lr=[0.0005252208125535731, 0.0005252208125535731], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:27,730] [INFO] [timer.py:199:stop] epoch=0/micro_step=2860/global_step=1430, RunningAvgSamplesPerSec=24.648358776663674, CurrSamplesPerSec=24.667270708305598, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:34,240] [INFO] [logging.py:96:log_dist] [Rank 0] step=1440, skipped=14, lr=[0.0005198578872880709, 0.0005198578872880709], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:34,243] [INFO] [timer.py:199:stop] epoch=0/micro_step=2880/global_step=1440, RunningAvgSamplesPerSec=24.648227558436865, CurrSamplesPerSec=24.711706241293587, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:40,744] [INFO] [logging.py:96:log_dist] [Rank 0] step=1450, skipped=14, lr=[0.0005144926728424205, 0.0005144926728424205], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:40,747] [INFO] [timer.py:199:stop] epoch=0/micro_step=2900/global_step=1450, RunningAvgSamplesPerSec=24.648324459399706, CurrSamplesPerSec=24.651558591880313, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:47,250] [INFO] [logging.py:96:log_dist] [Rank 0] step=1460, skipped=14, lr=[0.0005091257877085201, 0.0005091257877085201], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:47,252] [INFO] [timer.py:199:stop] epoch=0/micro_step=2920/global_step=1460, RunningAvgSamplesPerSec=24.64837541833112, CurrSamplesPerSec=24.675488122215413, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:55:53,754] [INFO] [logging.py:96:log_dist] [Rank 0] step=1470, skipped=14, lr=[0.000503757850570861, 0.000503757850570861], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:55:53,757] [INFO] [timer.py:199:stop] epoch=0/micro_step=2940/global_step=1470, RunningAvgSamplesPerSec=24.648462481620747, CurrSamplesPerSec=24.685590521985297, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:00,257] [INFO] [logging.py:96:log_dist] [Rank 0] step=1480, skipped=14, lr=[0.0004983894802352086, 0.0004983894802352086], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:00,260] [INFO] [timer.py:199:stop] epoch=0/micro_step=2960/global_step=1480, RunningAvgSamplesPerSec=24.64859129366692, CurrSamplesPerSec=24.675370173587236, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:06,765] [INFO] [logging.py:96:log_dist] [Rank 0] step=1490, skipped=14, lr=[0.0004930212955572661, 0.0004930212955572661], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:06,768] [INFO] [timer.py:199:stop] epoch=0/micro_step=2980/global_step=1490, RunningAvgSamplesPerSec=24.64857657735941, CurrSamplesPerSec=24.683774566490996, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:13,271] [INFO] [logging.py:96:log_dist] [Rank 0] step=1500, skipped=14, lr=[0.00048765391537133396, 0.00048765391537133396], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:13,274] [INFO] [timer.py:199:stop] epoch=0/micro_step=3000/global_step=1500, RunningAvgSamplesPerSec=24.648619731195733, CurrSamplesPerSec=24.68066082475478, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:19,776] [INFO] [logging.py:96:log_dist] [Rank 0] step=1510, skipped=14, lr=[0.0004822879584189731, 0.0004822879584189731], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:19,779] [INFO] [timer.py:199:stop] epoch=0/micro_step=3020/global_step=1510, RunningAvgSamplesPerSec=24.648684561888707, CurrSamplesPerSec=24.660146126057334, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:26,284] [INFO] [logging.py:96:log_dist] [Rank 0] step=1520, skipped=14, lr=[0.00047692404327767614, 0.00047692404327767614], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:26,287] [INFO] [timer.py:199:stop] epoch=0/micro_step=3040/global_step=1520, RunningAvgSamplesPerSec=24.648656902141372, CurrSamplesPerSec=24.683275225568348, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:30,836] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:56:30,837] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:56:31,464] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1527
[2023-04-14 08:56:31,465] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:56:31,465] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:56:32,776] [INFO] [logging.py:96:log_dist] [Rank 0] step=1530, skipped=15, lr=[0.00047209877646474044, 0.00047209877646474044], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:32,777] [INFO] [timer.py:199:stop] epoch=0/micro_step=3060/global_step=1530, RunningAvgSamplesPerSec=24.649097083434285, CurrSamplesPerSec=24.291174708662044, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:39,271] [INFO] [logging.py:96:log_dist] [Rank 0] step=1540, skipped=15, lr=[0.0004667404440450165, 0.0004667404440450165], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:39,273] [INFO] [timer.py:199:stop] epoch=0/micro_step=3080/global_step=1540, RunningAvgSamplesPerSec=24.64934928954515, CurrSamplesPerSec=24.690013491254966, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:45,778] [INFO] [logging.py:96:log_dist] [Rank 0] step=1550, skipped=15, lr=[0.0004613859457247678, 0.0004613859457247678], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:45,781] [INFO] [timer.py:199:stop] epoch=0/micro_step=3100/global_step=1550, RunningAvgSamplesPerSec=24.64935868242458, CurrSamplesPerSec=24.656893391943587, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:52,275] [INFO] [logging.py:96:log_dist] [Rank 0] step=1560, skipped=15, lr=[0.00045603589876055744, 0.00045603589876055744], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:52,278] [INFO] [timer.py:199:stop] epoch=0/micro_step=3120/global_step=1560, RunningAvgSamplesPerSec=24.649589031391212, CurrSamplesPerSec=24.708558157036574, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:56:58,779] [INFO] [logging.py:96:log_dist] [Rank 0] step=1570, skipped=15, lr=[0.00045069091989580483, 0.00045069091989580483], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:56:58,782] [INFO] [timer.py:199:stop] epoch=0/micro_step=3140/global_step=1570, RunningAvgSamplesPerSec=24.649674433457033, CurrSamplesPerSec=24.648208539641463, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:05,289] [INFO] [logging.py:96:log_dist] [Rank 0] step=1580, skipped=15, lr=[0.0004453516252896876, 0.0004453516252896876], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:05,292] [INFO] [timer.py:199:stop] epoch=0/micro_step=3160/global_step=1580, RunningAvgSamplesPerSec=24.649596136640543, CurrSamplesPerSec=24.707994134191235, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:11,794] [INFO] [logging.py:96:log_dist] [Rank 0] step=1590, skipped=15, lr=[0.0004400186304461136, 0.0004400186304461136], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:11,796] [INFO] [timer.py:199:stop] epoch=0/micro_step=3180/global_step=1590, RunningAvgSamplesPerSec=24.64966963110524, CurrSamplesPerSec=24.63764825609784, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:18,312] [INFO] [logging.py:96:log_dist] [Rank 0] step=1600, skipped=15, lr=[0.00043469255014276487, 0.00043469255014276487], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:18,315] [INFO] [timer.py:199:stop] epoch=0/micro_step=3200/global_step=1600, RunningAvgSamplesPerSec=24.649413198022923, CurrSamplesPerSec=24.683583906663504, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:24,825] [INFO] [logging.py:96:log_dist] [Rank 0] step=1610, skipped=15, lr=[0.00042937399836022916, 0.00042937399836022916], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:24,828] [INFO] [timer.py:199:stop] epoch=0/micro_step=3220/global_step=1610, RunningAvgSamplesPerSec=24.649280501206626, CurrSamplesPerSec=24.70839440581583, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:31,336] [INFO] [logging.py:96:log_dist] [Rank 0] step=1620, skipped=15, lr=[0.0004240635882112196, 0.0004240635882112196], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:31,338] [INFO] [timer.py:199:stop] epoch=0/micro_step=3240/global_step=1620, RunningAvgSamplesPerSec=24.649195816747323, CurrSamplesPerSec=24.65436609359266, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:37,179] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:57:37,180] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:57:37,807] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1629
[2023-04-14 08:57:37,807] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:57:37,807] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:57:37,808] [INFO] [logging.py:96:log_dist] [Rank 0] step=1630, skipped=16, lr=[0.0004192916861562503, 0.0004192916861562503], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:37,809] [INFO] [timer.py:199:stop] epoch=0/micro_step=3260/global_step=1630, RunningAvgSamplesPerSec=24.65004085578897, CurrSamplesPerSec=26.265412355559313, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:44,306] [INFO] [logging.py:96:log_dist] [Rank 0] step=1640, skipped=16, lr=[0.00041399843081817085, 0.00041399843081817085], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:44,308] [INFO] [timer.py:199:stop] epoch=0/micro_step=3280/global_step=1640, RunningAvgSamplesPerSec=24.650210496240074, CurrSamplesPerSec=24.58749537442432, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:50,821] [INFO] [logging.py:96:log_dist] [Rank 0] step=1650, skipped=16, lr=[0.0004087150895803192, 0.0004087150895803192], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:50,823] [INFO] [timer.py:199:stop] epoch=0/micro_step=3300/global_step=1650, RunningAvgSamplesPerSec=24.650014502221268, CurrSamplesPerSec=24.234927126063116, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:57:57,319] [INFO] [logging.py:96:log_dist] [Rank 0] step=1660, skipped=16, lr=[0.000403442271496403, 0.000403442271496403], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:57:57,321] [INFO] [timer.py:199:stop] epoch=0/micro_step=3320/global_step=1660, RunningAvgSamplesPerSec=24.650202007543893, CurrSamplesPerSec=24.67449013095607, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:03,820] [INFO] [logging.py:96:log_dist] [Rank 0] step=1670, skipped=16, lr=[0.0003981805844070407, 0.0003981805844070407], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:03,823] [INFO] [timer.py:199:stop] epoch=0/micro_step=3340/global_step=1670, RunningAvgSamplesPerSec=24.650310564399284, CurrSamplesPerSec=24.711460552682958, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:10,327] [INFO] [logging.py:96:log_dist] [Rank 0] step=1680, skipped=16, lr=[0.00039293063486968953, 0.00039293063486968953], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:10,330] [INFO] [timer.py:199:stop] epoch=0/micro_step=3360/global_step=1680, RunningAvgSamplesPerSec=24.650307754584613, CurrSamplesPerSec=24.6503904972991, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:16,834] [INFO] [logging.py:96:log_dist] [Rank 0] step=1690, skipped=16, lr=[0.00038769302808872475, 0.00038769302808872475], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:16,837] [INFO] [timer.py:199:stop] epoch=0/micro_step=3380/global_step=1690, RunningAvgSamplesPerSec=24.6503094248548, CurrSamplesPerSec=24.448644408830805, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:23,335] [INFO] [logging.py:96:log_dist] [Rank 0] step=1700, skipped=16, lr=[0.0003824683678456711, 0.0003824683678456711], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:23,337] [INFO] [timer.py:199:stop] epoch=0/micro_step=3400/global_step=1700, RunningAvgSamplesPerSec=24.65044266515727, CurrSamplesPerSec=24.670326656682203, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:29,845] [INFO] [logging.py:96:log_dist] [Rank 0] step=1710, skipped=16, lr=[0.00037725725642960046, 0.00037725725642960046], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:29,848] [INFO] [timer.py:199:stop] epoch=0/micro_step=3420/global_step=1710, RunningAvgSamplesPerSec=24.650368815127393, CurrSamplesPerSec=24.65752756244585, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:36,350] [INFO] [logging.py:96:log_dist] [Rank 0] step=1720, skipped=16, lr=[0.00037206029456770184, 0.00037206029456770184], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:36,353] [INFO] [timer.py:199:stop] epoch=0/micro_step=3440/global_step=1720, RunningAvgSamplesPerSec=24.65041427215023, CurrSamplesPerSec=24.672449036614815, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:42,849] [INFO] [logging.py:96:log_dist] [Rank 0] step=1730, skipped=16, lr=[0.0003668780813560295, 0.0003668780813560295], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:42,851] [INFO] [timer.py:199:stop] epoch=0/micro_step=3460/global_step=1730, RunningAvgSamplesPerSec=24.65060310855649, CurrSamplesPerSec=24.691857620019324, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:43,483] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:58:43,483] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:58:44,111] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1731
[2023-04-14 08:58:44,111] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:58:44,111] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:58:49,315] [INFO] [logging.py:96:log_dist] [Rank 0] step=1740, skipped=17, lr=[0.00036222719334538545, 0.00036222719334538545], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:49,318] [INFO] [timer.py:199:stop] epoch=0/micro_step=3480/global_step=1740, RunningAvgSamplesPerSec=24.65148298607978, CurrSamplesPerSec=24.68806973256162, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:58:55,826] [INFO] [logging.py:96:log_dist] [Rank 0] step=1750, skipped=17, lr=[0.00035707464693402006, 0.00035707464693402006], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:58:55,828] [INFO] [timer.py:199:stop] epoch=0/micro_step=3500/global_step=1750, RunningAvgSamplesPerSec=24.651388068201598, CurrSamplesPerSec=24.682331072495046, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:02,339] [INFO] [logging.py:96:log_dist] [Rank 0] step=1760, skipped=17, lr=[0.0003519385766903442, 0.0003519385766903442], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:02,342] [INFO] [timer.py:199:stop] epoch=0/micro_step=3520/global_step=1760, RunningAvgSamplesPerSec=24.65125691802657, CurrSamplesPerSec=24.6710068668982, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:08,858] [INFO] [logging.py:96:log_dist] [Rank 0] step=1770, skipped=17, lr=[0.00034681957469094067, 0.00034681957469094067], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:08,861] [INFO] [timer.py:199:stop] epoch=0/micro_step=3540/global_step=1770, RunningAvgSamplesPerSec=24.650973224742994, CurrSamplesPerSec=24.592721369388123, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:15,358] [INFO] [logging.py:96:log_dist] [Rank 0] step=1780, skipped=17, lr=[0.0003417182310447964, 0.0003417182310447964], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:15,361] [INFO] [timer.py:199:stop] epoch=0/micro_step=3560/global_step=1780, RunningAvgSamplesPerSec=24.651103150063694, CurrSamplesPerSec=24.687334090654925, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:21,865] [INFO] [logging.py:96:log_dist] [Rank 0] step=1790, skipped=17, lr=[0.00033663513382527655, 0.00033663513382527655], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:21,867] [INFO] [timer.py:199:stop] epoch=0/micro_step=3580/global_step=1790, RunningAvgSamplesPerSec=24.65110017200448, CurrSamplesPerSec=24.68523638969272, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:28,385] [INFO] [logging.py:96:log_dist] [Rank 0] step=1800, skipped=17, lr=[0.0003315708690023328, 0.0003315708690023328], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:28,388] [INFO] [timer.py:199:stop] epoch=0/micro_step=3600/global_step=1800, RunningAvgSamplesPerSec=24.65080904902749, CurrSamplesPerSec=24.706365890132677, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:34,884] [INFO] [logging.py:96:log_dist] [Rank 0] step=1810, skipped=17, lr=[0.00032652602037495243, 0.00032652602037495243], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:34,887] [INFO] [timer.py:199:stop] epoch=0/micro_step=3620/global_step=1810, RunningAvgSamplesPerSec=24.650959085181007, CurrSamplesPerSec=24.669084236184656, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:41,393] [INFO] [logging.py:96:log_dist] [Rank 0] step=1820, skipped=17, lr=[0.0003215011695038609, 0.0003215011695038609], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:41,396] [INFO] [timer.py:199:stop] epoch=0/micro_step=3640/global_step=1820, RunningAvgSamplesPerSec=24.650897443982856, CurrSamplesPerSec=24.693529383626416, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:47,905] [INFO] [logging.py:96:log_dist] [Rank 0] step=1830, skipped=17, lr=[0.0003164968956444791, 0.0003164968956444791], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:47,907] [INFO] [timer.py:199:stop] epoch=0/micro_step=3660/global_step=1830, RunningAvgSamplesPerSec=24.65078415933657, CurrSamplesPerSec=24.65520846880885, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 08:59:49,840] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 08:59:49,840] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 08:59:50,468] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1833
[2023-04-14 08:59:50,468] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 08:59:50,468] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 08:59:54,373] [INFO] [logging.py:96:log_dist] [Rank 0] step=1840, skipped=18, lr=[0.00031201111936021464, 0.00031201111936021464], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 08:59:54,376] [INFO] [timer.py:199:stop] epoch=0/micro_step=3680/global_step=1840, RunningAvgSamplesPerSec=24.651566548636268, CurrSamplesPerSec=24.50354524213863, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:00,880] [INFO] [logging.py:96:log_dist] [Rank 0] step=1850, skipped=18, lr=[0.0003070475291225692, 0.0003070475291225692], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:00,883] [INFO] [timer.py:199:stop] epoch=0/micro_step=3700/global_step=1850, RunningAvgSamplesPerSec=24.65153133253062, CurrSamplesPerSec=24.67223133981565, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:07,383] [INFO] [logging.py:96:log_dist] [Rank 0] step=1860, skipped=18, lr=[0.0003021061820852945, 0.0003021061820852945], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:07,386] [INFO] [timer.py:199:stop] epoch=0/micro_step=3720/global_step=1860, RunningAvgSamplesPerSec=24.651602177895626, CurrSamplesPerSec=24.70512893195906, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:13,893] [INFO] [logging.py:96:log_dist] [Rank 0] step=1870, skipped=18, lr=[0.0002971876478776454, 0.0002971876478776454], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:13,895] [INFO] [timer.py:199:stop] epoch=0/micro_step=3740/global_step=1870, RunningAvgSamplesPerSec=24.651519098070832, CurrSamplesPerSec=24.60817807660603, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:20,400] [INFO] [logging.py:96:log_dist] [Rank 0] step=1880, skipped=18, lr=[0.0002922924934990568, 0.0002922924934990568], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:20,403] [INFO] [timer.py:199:stop] epoch=0/micro_step=3760/global_step=1880, RunningAvgSamplesPerSec=24.65151054228362, CurrSamplesPerSec=24.69598292485464, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:26,922] [INFO] [logging.py:96:log_dist] [Rank 0] step=1890, skipped=18, lr=[0.00028742128325378046, 0.00028742128325378046], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:26,923] [INFO] [timer.py:199:stop] epoch=0/micro_step=3780/global_step=1890, RunningAvgSamplesPerSec=24.651231694383267, CurrSamplesPerSec=24.219944348403537, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:33,431] [INFO] [logging.py:96:log_dist] [Rank 0] step=1900, skipped=18, lr=[0.0002825745786858328, 0.0002825745786858328], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:33,434] [INFO] [timer.py:199:stop] epoch=0/micro_step=3800/global_step=1900, RunningAvgSamplesPerSec=24.651150756179355, CurrSamplesPerSec=24.67065315290476, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:39,934] [INFO] [logging.py:96:log_dist] [Rank 0] step=1910, skipped=18, lr=[0.0002777529385142623, 0.0002777529385142623], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:39,936] [INFO] [timer.py:199:stop] epoch=0/micro_step=3820/global_step=1910, RunningAvgSamplesPerSec=24.651243805947406, CurrSamplesPerSec=24.68971373280502, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:46,448] [INFO] [logging.py:96:log_dist] [Rank 0] step=1920, skipped=18, lr=[0.0002729569185687397, 0.0002729569185687397], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:46,450] [INFO] [timer.py:199:stop] epoch=0/micro_step=3840/global_step=1920, RunningAvgSamplesPerSec=24.651087520444722, CurrSamplesPerSec=24.672367399864928, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:52,955] [INFO] [logging.py:96:log_dist] [Rank 0] step=1930, skipped=18, lr=[0.00026818707172548484, 0.00026818707172548484], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:52,957] [INFO] [timer.py:199:stop] epoch=0/micro_step=3860/global_step=1930, RunningAvgSamplesPerSec=24.65108845610022, CurrSamplesPerSec=24.515566972723732, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:00:56,189] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:00:56,189] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:00:56,816] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 1935
[2023-04-14 09:00:56,816] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:00:56,816] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:00:59,413] [INFO] [logging.py:96:log_dist] [Rank 0] step=1940, skipped=19, lr=[0.0002639170420908502, 0.0002639170420908502], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:00:59,416] [INFO] [timer.py:199:stop] epoch=0/micro_step=3880/global_step=1940, RunningAvgSamplesPerSec=24.65202110946983, CurrSamplesPerSec=24.683375092136643, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:05,935] [INFO] [logging.py:96:log_dist] [Rank 0] step=1950, skipped=19, lr=[0.0002591984364587502, 0.0002591984364587502], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:05,938] [INFO] [timer.py:199:stop] epoch=0/micro_step=3900/global_step=1950, RunningAvgSamplesPerSec=24.651725500127796, CurrSamplesPerSec=24.655289992045955, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:12,444] [INFO] [logging.py:96:log_dist] [Rank 0] step=1960, skipped=19, lr=[0.0002545075899810321, 0.0002545075899810321], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:12,445] [INFO] [timer.py:199:stop] epoch=0/micro_step=3920/global_step=1960, RunningAvgSamplesPerSec=24.651706661469188, CurrSamplesPerSec=24.371726140108382, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:18,946] [INFO] [logging.py:96:log_dist] [Rank 0] step=1970, skipped=19, lr=[0.00024984504340971403, 0.00024984504340971403], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:18,948] [INFO] [timer.py:199:stop] epoch=0/micro_step=3940/global_step=1970, RunningAvgSamplesPerSec=24.651765621451663, CurrSamplesPerSec=24.667660594298148, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:25,445] [INFO] [logging.py:96:log_dist] [Rank 0] step=1980, skipped=19, lr=[0.00024521133423445447, 0.00024521133423445447], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:25,448] [INFO] [timer.py:199:stop] epoch=0/micro_step=3960/global_step=1980, RunningAvgSamplesPerSec=24.651894869568636, CurrSamplesPerSec=24.716202295467294, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:31,970] [INFO] [logging.py:96:log_dist] [Rank 0] step=1990, skipped=19, lr=[0.00024060699662059072, 0.00024060699662059072], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:31,972] [INFO] [timer.py:199:stop] epoch=0/micro_step=3980/global_step=1990, RunningAvgSamplesPerSec=24.651557120601453, CurrSamplesPerSec=24.641420685851635, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:38,472] [INFO] [logging.py:96:log_dist] [Rank 0] step=2000, skipped=19, lr=[0.00023603256134756062, 0.00023603256134756062], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:38,475] [INFO] [timer.py:199:stop] epoch=0/micro_step=4000/global_step=2000, RunningAvgSamplesPerSec=24.651645280438412, CurrSamplesPerSec=24.655788201320433, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:44,992] [INFO] [logging.py:96:log_dist] [Rank 0] step=2010, skipped=19, lr=[0.00023148855574771704, 0.00023148855574771704], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:44,994] [INFO] [timer.py:199:stop] epoch=0/micro_step=4020/global_step=2010, RunningAvgSamplesPerSec=24.651394084572186, CurrSamplesPerSec=24.3446986172514, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:51,501] [INFO] [logging.py:96:log_dist] [Rank 0] step=2020, skipped=19, lr=[0.00022697550364553603, 0.00022697550364553603], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:51,504] [INFO] [timer.py:199:stop] epoch=0/micro_step=4040/global_step=2020, RunningAvgSamplesPerSec=24.651346239935066, CurrSamplesPerSec=24.66672670193555, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:01:58,001] [INFO] [logging.py:96:log_dist] [Rank 0] step=2030, skipped=19, lr=[0.00022249392529723305, 0.00022249392529723305], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:01:58,004] [INFO] [timer.py:199:stop] epoch=0/micro_step=4060/global_step=2030, RunningAvgSamplesPerSec=24.65146888633639, CurrSamplesPerSec=24.69421996074469, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:02,538] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:02:02,538] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:02:03,169] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2037
[2023-04-14 09:02:03,169] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:02:03,170] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:02:04,469] [INFO] [logging.py:96:log_dist] [Rank 0] step=2040, skipped=20, lr=[0.00021848784191228038, 0.00021848784191228038], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:04,472] [INFO] [timer.py:199:stop] epoch=0/micro_step=4080/global_step=2040, RunningAvgSamplesPerSec=24.652190483961828, CurrSamplesPerSec=24.670861752238007, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:10,974] [INFO] [logging.py:96:log_dist] [Rank 0] step=2050, skipped=20, lr=[0.00021406748395969306, 0.00021406748395969306], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:10,976] [INFO] [timer.py:199:stop] epoch=0/micro_step=4100/global_step=2050, RunningAvgSamplesPerSec=24.652230856198305, CurrSamplesPerSec=24.681386991648054, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:17,473] [INFO] [logging.py:96:log_dist] [Rank 0] step=2060, skipped=20, lr=[0.00020968008777337815, 0.00020968008777337815], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:17,476] [INFO] [timer.py:199:stop] epoch=0/micro_step=4120/global_step=2060, RunningAvgSamplesPerSec=24.652368603895805, CurrSamplesPerSec=24.675569779618538, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:23,993] [INFO] [logging.py:96:log_dist] [Rank 0] step=2070, skipped=20, lr=[0.0002053261591241717, 0.0002053261591241717], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:23,996] [INFO] [timer.py:199:stop] epoch=0/micro_step=4140/global_step=2070, RunningAvgSamplesPerSec=24.652126543933537, CurrSamplesPerSec=24.66520361174379, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:30,491] [INFO] [logging.py:96:log_dist] [Rank 0] step=2080, skipped=20, lr=[0.00020100619992483438, 0.00020100619992483438], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:30,493] [INFO] [timer.py:199:stop] epoch=0/micro_step=4160/global_step=2080, RunningAvgSamplesPerSec=24.65228067647617, CurrSamplesPerSec=24.679172316200308, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:37,006] [INFO] [logging.py:96:log_dist] [Rank 0] step=2090, skipped=20, lr=[0.0001967207081721924, 0.0001967207081721924], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:37,009] [INFO] [timer.py:199:stop] epoch=0/micro_step=4180/global_step=2090, RunningAvgSamplesPerSec=24.652113682900826, CurrSamplesPerSec=24.67703970529725, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:43,507] [INFO] [logging.py:96:log_dist] [Rank 0] step=2100, skipped=20, lr=[0.0001924701778897287, 0.0001924701778897287], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:43,509] [INFO] [timer.py:199:stop] epoch=0/micro_step=4200/global_step=2100, RunningAvgSamplesPerSec=24.652217967961562, CurrSamplesPerSec=24.669564865768976, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:50,025] [INFO] [logging.py:96:log_dist] [Rank 0] step=2110, skipped=20, lr=[0.00018825509907063325, 0.00018825509907063325], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:50,028] [INFO] [timer.py:199:stop] epoch=0/micro_step=4220/global_step=2110, RunningAvgSamplesPerSec=24.65199904225975, CurrSamplesPerSec=24.6727665180001, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:02:56,539] [INFO] [logging.py:96:log_dist] [Rank 0] step=2120, skipped=20, lr=[0.00018407595762131811, 0.00018407595762131811], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:02:56,541] [INFO] [timer.py:199:stop] epoch=0/micro_step=4240/global_step=2120, RunningAvgSamplesPerSec=24.651863940517302, CurrSamplesPerSec=24.687470317330042, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:03,046] [INFO] [logging.py:96:log_dist] [Rank 0] step=2130, skipped=20, lr=[0.00017993323530540183, 0.00017993323530540183], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:03,048] [INFO] [timer.py:199:stop] epoch=0/micro_step=4260/global_step=2130, RunningAvgSamplesPerSec=24.65186136512357, CurrSamplesPerSec=24.63941219816576, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:08,883] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:03:08,883] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:03:09,510] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2139
[2023-04-14 09:03:09,510] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:03:09,510] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:03:09,511] [INFO] [logging.py:96:log_dist] [Rank 0] step=2140, skipped=21, lr=[0.00017623631837548155, 0.00017623631837548155], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:09,512] [INFO] [timer.py:199:stop] epoch=0/micro_step=4280/global_step=2140, RunningAvgSamplesPerSec=24.652617630884006, CurrSamplesPerSec=26.266974992132692, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:16,011] [INFO] [logging.py:96:log_dist] [Rank 0] step=2150, skipped=21, lr=[0.00017216410459110416, 0.00017216410459110416], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:16,013] [INFO] [timer.py:199:stop] epoch=0/micro_step=4300/global_step=2150, RunningAvgSamplesPerSec=24.652702048411484, CurrSamplesPerSec=24.658361093290356, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:22,521] [INFO] [logging.py:96:log_dist] [Rank 0] step=2160, skipped=21, lr=[0.0001681296831162893, 0.0001681296831162893], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:22,523] [INFO] [timer.py:199:stop] epoch=0/micro_step=4320/global_step=2160, RunningAvgSamplesPerSec=24.65262859689061, CurrSamplesPerSec=24.599229936864205, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:29,023] [INFO] [logging.py:96:log_dist] [Rank 0] step=2170, skipped=21, lr=[0.0001641335190316076, 0.0001641335190316076], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:29,026] [INFO] [timer.py:199:stop] epoch=0/micro_step=4340/global_step=2170, RunningAvgSamplesPerSec=24.652684348000452, CurrSamplesPerSec=24.59507379196782, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:35,529] [INFO] [logging.py:96:log_dist] [Rank 0] step=2180, skipped=21, lr=[0.00016017607300738946, 0.00016017607300738946], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:35,531] [INFO] [timer.py:199:stop] epoch=0/micro_step=4360/global_step=2180, RunningAvgSamplesPerSec=24.652708842622143, CurrSamplesPerSec=24.657989621496576, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:42,040] [INFO] [logging.py:96:log_dist] [Rank 0] step=2190, skipped=21, lr=[0.00015625780125061884, 0.00015625780125061884], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:42,043] [INFO] [timer.py:199:stop] epoch=0/micro_step=4380/global_step=2190, RunningAvgSamplesPerSec=24.652612066836433, CurrSamplesPerSec=24.667705930678565, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:48,539] [INFO] [logging.py:96:log_dist] [Rank 0] step=2200, skipped=21, lr=[0.0001523791554523442, 0.0001523791554523442], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:48,541] [INFO] [timer.py:199:stop] epoch=0/micro_step=4400/global_step=2200, RunningAvgSamplesPerSec=24.652738667583176, CurrSamplesPerSec=24.65162198002417, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:03:55,047] [INFO] [logging.py:96:log_dist] [Rank 0] step=2210, skipped=21, lr=[0.00014854058273560666, 0.00014854058273560666], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:03:55,050] [INFO] [timer.py:199:stop] epoch=0/micro_step=4420/global_step=2210, RunningAvgSamplesPerSec=24.65269697873488, CurrSamplesPerSec=24.68944123143938, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:01,548] [INFO] [logging.py:96:log_dist] [Rank 0] step=2220, skipped=21, lr=[0.00014474252560389827, 0.00014474252560389827], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:01,550] [INFO] [timer.py:199:stop] epoch=0/micro_step=4440/global_step=2220, RunningAvgSamplesPerSec=24.652794964474932, CurrSamplesPerSec=24.66948324810436, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:08,059] [INFO] [logging.py:96:log_dist] [Rank 0] step=2230, skipped=21, lr=[0.0001409854218901493, 0.0001409854218901493], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:08,062] [INFO] [timer.py:199:stop] epoch=0/micro_step=4460/global_step=2230, RunningAvgSamplesPerSec=24.65271325746412, CurrSamplesPerSec=24.681005748701107, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:14,570] [INFO] [logging.py:96:log_dist] [Rank 0] step=2240, skipped=21, lr=[0.00013726970470625706, 0.00013726970470625706], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:14,571] [INFO] [timer.py:199:stop] epoch=0/micro_step=4480/global_step=2240, RunningAvgSamplesPerSec=24.652670274433053, CurrSamplesPerSec=24.591054215929308, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:15,202] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:04:15,202] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:04:15,830] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2241
[2023-04-14 09:04:15,830] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:04:15,830] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:04:21,033] [INFO] [logging.py:96:log_dist] [Rank 0] step=2250, skipped=22, lr=[0.00013396129884694197, 0.00013396129884694197], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:21,036] [INFO] [timer.py:199:stop] epoch=0/micro_step=4500/global_step=2250, RunningAvgSamplesPerSec=24.653358086292577, CurrSamplesPerSec=24.640027375823873, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:27,539] [INFO] [logging.py:96:log_dist] [Rank 0] step=2260, skipped=22, lr=[0.00013032539216605455, 0.00013032539216605455], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:27,541] [INFO] [timer.py:199:stop] epoch=0/micro_step=4520/global_step=2260, RunningAvgSamplesPerSec=24.653367864795957, CurrSamplesPerSec=24.70290090070856, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:34,047] [INFO] [logging.py:96:log_dist] [Rank 0] step=2270, skipped=22, lr=[0.00012673210088333525, 0.00012673210088333525], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:34,050] [INFO] [timer.py:199:stop] epoch=0/micro_step=4540/global_step=2270, RunningAvgSamplesPerSec=24.653315597478198, CurrSamplesPerSec=24.64095019991048, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:40,564] [INFO] [logging.py:96:log_dist] [Rank 0] step=2280, skipped=22, lr=[0.00012318183922669062, 0.00012318183922669062], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:40,567] [INFO] [timer.py:199:stop] epoch=0/micro_step=4560/global_step=2280, RunningAvgSamplesPerSec=24.65314261404672, CurrSamplesPerSec=24.686907256808144, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:47,065] [INFO] [logging.py:96:log_dist] [Rank 0] step=2290, skipped=22, lr=[0.00011967501646365148, 0.00011967501646365148], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:47,067] [INFO] [timer.py:199:stop] epoch=0/micro_step=4580/global_step=2290, RunningAvgSamplesPerSec=24.65322395682049, CurrSamplesPerSec=24.70563825267695, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:04:53,581] [INFO] [logging.py:96:log_dist] [Rank 0] step=2300, skipped=22, lr=[0.00011621203685419484, 0.00011621203685419484], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:04:53,584] [INFO] [timer.py:199:stop] epoch=0/micro_step=4600/global_step=2300, RunningAvgSamplesPerSec=24.653059795481294, CurrSamplesPerSec=24.687515726555844, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:00,100] [INFO] [logging.py:96:log_dist] [Rank 0] step=2310, skipped=22, lr=[0.00011279329960414048, 0.00011279329960414048], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:00,102] [INFO] [timer.py:199:stop] epoch=0/micro_step=4620/global_step=2310, RunningAvgSamplesPerSec=24.652845098392238, CurrSamplesPerSec=24.599338141606026, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:06,602] [INFO] [logging.py:96:log_dist] [Rank 0] step=2320, skipped=22, lr=[0.00010941919881913142, 0.00010941919881913142], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:06,605] [INFO] [timer.py:199:stop] epoch=0/micro_step=4640/global_step=2320, RunningAvgSamplesPerSec=24.65291241722985, CurrSamplesPerSec=24.680651747939017, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:13,113] [INFO] [logging.py:96:log_dist] [Rank 0] step=2330, skipped=22, lr=[0.00010609012345920321, 0.00010609012345920321], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:13,115] [INFO] [timer.py:199:stop] epoch=0/micro_step=4660/global_step=2330, RunningAvgSamplesPerSec=24.65284180240075, CurrSamplesPerSec=24.663499424288098, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:19,614] [INFO] [logging.py:96:log_dist] [Rank 0] step=2340, skipped=22, lr=[0.00010280645729394367, 0.00010280645729394367], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:19,616] [INFO] [timer.py:199:stop] epoch=0/micro_step=4680/global_step=2340, RunningAvgSamplesPerSec=24.652930491649105, CurrSamplesPerSec=24.692820673587104, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:21,549] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:05:21,549] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:05:22,177] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2343
[2023-04-14 09:05:22,177] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:05:22,177] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:05:26,075] [INFO] [logging.py:96:log_dist] [Rank 0] step=2350, skipped=23, lr=[9.989029557461438e-05, 9.989029557461438e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:26,078] [INFO] [timer.py:199:stop] epoch=0/micro_step=4700/global_step=2350, RunningAvgSamplesPerSec=24.65366031633126, CurrSamplesPerSec=24.679971005783766, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:32,581] [INFO] [logging.py:96:log_dist] [Rank 0] step=2360, skipped=23, lr=[9.6693945381235e-05, 9.6693945381235e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:32,584] [INFO] [timer.py:199:stop] epoch=0/micro_step=4720/global_step=2360, RunningAvgSamplesPerSec=24.653652082257377, CurrSamplesPerSec=24.69707354413237, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:39,082] [INFO] [logging.py:96:log_dist] [Rank 0] step=2370, skipped=23, lr=[9.354408755637922e-05, 9.354408755637922e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:39,084] [INFO] [timer.py:199:stop] epoch=0/micro_step=4740/global_step=2370, RunningAvgSamplesPerSec=24.653745093701176, CurrSamplesPerSec=24.683974308517016, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:45,587] [INFO] [logging.py:96:log_dist] [Rank 0] step=2380, skipped=23, lr=[9.044108520977279e-05, 9.044108520977279e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:45,590] [INFO] [timer.py:199:stop] epoch=0/micro_step=4760/global_step=2380, RunningAvgSamplesPerSec=24.653751993644587, CurrSamplesPerSec=24.681323450338983, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:52,084] [INFO] [logging.py:96:log_dist] [Rank 0] step=2390, skipped=23, lr=[8.738529604972917e-05, 8.738529604972917e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:52,087] [INFO] [timer.py:199:stop] epoch=0/micro_step=4780/global_step=2390, RunningAvgSamplesPerSec=24.653886325332905, CurrSamplesPerSec=24.691648665484127, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:05:58,595] [INFO] [logging.py:96:log_dist] [Rank 0] step=2400, skipped=23, lr=[8.437707234191372e-05, 8.437707234191372e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:05:58,598] [INFO] [timer.py:199:stop] epoch=0/micro_step=4800/global_step=2400, RunningAvgSamplesPerSec=24.653798366815597, CurrSamplesPerSec=24.694156353342525, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:05,100] [INFO] [logging.py:96:log_dist] [Rank 0] step=2410, skipped=23, lr=[8.141676086873573e-05, 8.141676086873573e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:05,102] [INFO] [timer.py:199:stop] epoch=0/micro_step=4820/global_step=2410, RunningAvgSamplesPerSec=24.653812512172074, CurrSamplesPerSec=24.661387646386977, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:11,603] [INFO] [logging.py:96:log_dist] [Rank 0] step=2420, skipped=23, lr=[7.850470288937133e-05, 7.850470288937133e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:11,606] [INFO] [timer.py:199:stop] epoch=0/micro_step=4840/global_step=2420, RunningAvgSamplesPerSec=24.653844894466534, CurrSamplesPerSec=24.70171884322885, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:18,110] [INFO] [logging.py:96:log_dist] [Rank 0] step=2430, skipped=23, lr=[7.564123410042428e-05, 7.564123410042428e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:18,112] [INFO] [timer.py:199:stop] epoch=0/micro_step=4860/global_step=2430, RunningAvgSamplesPerSec=24.653831750653616, CurrSamplesPerSec=24.68075159327963, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:24,620] [INFO] [logging.py:96:log_dist] [Rank 0] step=2440, skipped=23, lr=[7.282668459722691e-05, 7.282668459722691e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:24,621] [INFO] [timer.py:199:stop] epoch=0/micro_step=4880/global_step=2440, RunningAvgSamplesPerSec=24.65379604969422, CurrSamplesPerSec=24.560239114132933, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:27,851] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:06:27,851] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:06:28,479] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2445
[2023-04-14 09:06:28,479] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:06:28,479] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:06:31,075] [INFO] [logging.py:96:log_dist] [Rank 0] step=2450, skipped=24, lr=[7.03356843136836e-05, 7.03356843136836e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:31,078] [INFO] [timer.py:199:stop] epoch=0/micro_step=4900/global_step=2450, RunningAvgSamplesPerSec=24.654545509390708, CurrSamplesPerSec=24.67273930498978, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:37,587] [INFO] [logging.py:96:log_dist] [Rank 0] step=2460, skipped=24, lr=[6.761497063866206e-05, 6.761497063866206e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:37,590] [INFO] [timer.py:199:stop] epoch=0/micro_step=4920/global_step=2460, RunningAvgSamplesPerSec=24.654449442667595, CurrSamplesPerSec=24.677856406349058, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:44,090] [INFO] [logging.py:96:log_dist] [Rank 0] step=2470, skipped=24, lr=[6.494410150204766e-05, 6.494410150204766e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:44,093] [INFO] [timer.py:199:stop] epoch=0/micro_step=4940/global_step=2470, RunningAvgSamplesPerSec=24.65448873729532, CurrSamplesPerSec=24.674227037616056, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:50,592] [INFO] [logging.py:96:log_dist] [Rank 0] step=2480, skipped=24, lr=[6.232338479664445e-05, 6.232338479664445e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:50,595] [INFO] [timer.py:199:stop] epoch=0/micro_step=4960/global_step=2480, RunningAvgSamplesPerSec=24.65454027420256, CurrSamplesPerSec=24.690086161066265, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:06:57,094] [INFO] [logging.py:96:log_dist] [Rank 0] step=2490, skipped=24, lr=[5.975312263377852e-05, 5.975312263377852e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:06:57,096] [INFO] [timer.py:199:stop] epoch=0/micro_step=4980/global_step=2490, RunningAvgSamplesPerSec=24.654601918193, CurrSamplesPerSec=24.660680780071218, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:03,600] [INFO] [logging.py:96:log_dist] [Rank 0] step=2500, skipped=24, lr=[5.723361130846999e-05, 5.723361130846999e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:03,602] [INFO] [timer.py:199:stop] epoch=0/micro_step=5000/global_step=2500, RunningAvgSamplesPerSec=24.654586782110634, CurrSamplesPerSec=24.693056905746797, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:10,111] [INFO] [logging.py:96:log_dist] [Rank 0] step=2510, skipped=24, lr=[5.476514126527771e-05, 5.476514126527771e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:10,113] [INFO] [timer.py:199:stop] epoch=0/micro_step=5020/global_step=2510, RunningAvgSamplesPerSec=24.65451574530273, CurrSamplesPerSec=24.66733417727847, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:16,615] [INFO] [logging.py:96:log_dist] [Rank 0] step=2520, skipped=24, lr=[5.234799706481669e-05, 5.234799706481669e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:16,618] [INFO] [timer.py:199:stop] epoch=0/micro_step=5040/global_step=2520, RunningAvgSamplesPerSec=24.65452489922423, CurrSamplesPerSec=24.69795519845369, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:23,123] [INFO] [logging.py:96:log_dist] [Rank 0] step=2530, skipped=24, lr=[4.9982457350954577e-05, 4.9982457350954577e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:23,125] [INFO] [timer.py:199:stop] epoch=0/micro_step=5060/global_step=2530, RunningAvgSamplesPerSec=24.654504371549066, CurrSamplesPerSec=24.680497443092488, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:29,629] [INFO] [logging.py:96:log_dist] [Rank 0] step=2540, skipped=24, lr=[4.766879481869041e-05, 4.766879481869041e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:29,632] [INFO] [timer.py:199:stop] epoch=0/micro_step=5080/global_step=2540, RunningAvgSamplesPerSec=24.654492959753977, CurrSamplesPerSec=24.691476053535528, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:34,164] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:07:34,165] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:07:34,792] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2547
[2023-04-14 09:07:34,793] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:07:34,793] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:07:36,092] [INFO] [logging.py:96:log_dist] [Rank 0] step=2550, skipped=25, lr=[4.563107409367928e-05, 4.563107409367928e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:36,095] [INFO] [timer.py:199:stop] epoch=0/micro_step=5100/global_step=2550, RunningAvgSamplesPerSec=24.655127315827098, CurrSamplesPerSec=24.670961518383503, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:42,600] [INFO] [logging.py:96:log_dist] [Rank 0] step=2560, skipped=25, lr=[4.3416708037924466e-05, 4.3416708037924466e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:42,602] [INFO] [timer.py:199:stop] epoch=0/micro_step=5120/global_step=2560, RunningAvgSamplesPerSec=24.655100593523603, CurrSamplesPerSec=24.686625736178602, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:49,104] [INFO] [logging.py:96:log_dist] [Rank 0] step=2570, skipped=25, lr=[4.1254976051082306e-05, 4.1254976051082306e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:49,107] [INFO] [timer.py:199:stop] epoch=0/micro_step=5140/global_step=2570, RunningAvgSamplesPerSec=24.655120187849786, CurrSamplesPerSec=24.682757748111705, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:07:55,605] [INFO] [logging.py:96:log_dist] [Rank 0] step=2580, skipped=25, lr=[3.914612733357775e-05, 3.914612733357775e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:07:55,608] [INFO] [timer.py:199:stop] epoch=0/micro_step=5160/global_step=2580, RunningAvgSamplesPerSec=24.65519088316272, CurrSamplesPerSec=24.664297099736814, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:02,121] [INFO] [logging.py:96:log_dist] [Rank 0] step=2590, skipped=25, lr=[3.709040498955102e-05, 3.709040498955102e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:02,124] [INFO] [timer.py:199:stop] epoch=0/micro_step=5180/global_step=2590, RunningAvgSamplesPerSec=24.655034111972533, CurrSamplesPerSec=24.692629873988555, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:08,622] [INFO] [logging.py:96:log_dist] [Rank 0] step=2600, skipped=25, lr=[3.508804599883359e-05, 3.508804599883359e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:08,624] [INFO] [timer.py:199:stop] epoch=0/micro_step=5200/global_step=2600, RunningAvgSamplesPerSec=24.655105109889966, CurrSamplesPerSec=24.657509442835938, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:15,130] [INFO] [logging.py:96:log_dist] [Rank 0] step=2610, skipped=25, lr=[3.313928118962906e-05, 3.313928118962906e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:15,133] [INFO] [timer.py:199:stop] epoch=0/micro_step=5220/global_step=2610, RunningAvgSamplesPerSec=24.655074487345562, CurrSamplesPerSec=24.691012738658994, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:21,635] [INFO] [logging.py:96:log_dist] [Rank 0] step=2620, skipped=25, lr=[3.1244335211904175e-05, 3.1244335211904175e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:21,638] [INFO] [timer.py:199:stop] epoch=0/micro_step=5240/global_step=2620, RunningAvgSamplesPerSec=24.655074642743738, CurrSamplesPerSec=24.671850379661024, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:28,146] [INFO] [logging.py:96:log_dist] [Rank 0] step=2630, skipped=25, lr=[2.9403426511490804e-05, 2.9403426511490804e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:28,149] [INFO] [timer.py:199:stop] epoch=0/micro_step=5260/global_step=2630, RunningAvgSamplesPerSec=24.654999434414513, CurrSamplesPerSec=24.704364990255765, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:34,644] [INFO] [logging.py:96:log_dist] [Rank 0] step=2640, skipped=25, lr=[2.7616767304905087e-05, 2.7616767304905087e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:34,647] [INFO] [timer.py:199:stop] epoch=0/micro_step=5280/global_step=2640, RunningAvgSamplesPerSec=24.655090384341396, CurrSamplesPerSec=24.70991373660281, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:40,497] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:08:40,497] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:08:41,125] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2649
[2023-04-14 09:08:41,125] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:08:41,125] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:08:41,126] [INFO] [logging.py:96:log_dist] [Rank 0] step=2650, skipped=26, lr=[2.605532769400476e-05, 2.605532769400476e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:41,127] [INFO] [timer.py:199:stop] epoch=0/micro_step=5300/global_step=2650, RunningAvgSamplesPerSec=24.655470937793527, CurrSamplesPerSec=26.251880712501638, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:47,635] [INFO] [logging.py:96:log_dist] [Rank 0] step=2660, skipped=26, lr=[2.4372304765133523e-05, 2.4372304765133523e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:47,638] [INFO] [timer.py:199:stop] epoch=0/micro_step=5320/global_step=2660, RunningAvgSamplesPerSec=24.655387502342816, CurrSamplesPerSec=24.70101875483557, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:08:54,156] [INFO] [logging.py:96:log_dist] [Rank 0] step=2670, skipped=26, lr=[2.2744111308405534e-05, 2.2744111308405534e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:08:54,157] [INFO] [timer.py:199:stop] epoch=0/micro_step=5340/global_step=2670, RunningAvgSamplesPerSec=24.655202106406644, CurrSamplesPerSec=24.18808128428595, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:00,677] [INFO] [logging.py:96:log_dist] [Rank 0] step=2680, skipped=26, lr=[2.1170935018920758e-05, 2.1170935018920758e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:00,679] [INFO] [timer.py:199:stop] epoch=0/micro_step=5360/global_step=2680, RunningAvgSamplesPerSec=24.65496516733041, CurrSamplesPerSec=24.631987256190218, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:07,179] [INFO] [logging.py:96:log_dist] [Rank 0] step=2690, skipped=26, lr=[1.965295724950289e-05, 1.965295724950289e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:07,182] [INFO] [timer.py:199:stop] epoch=0/micro_step=5380/global_step=2690, RunningAvgSamplesPerSec=24.655006287306826, CurrSamplesPerSec=24.679716871672493, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:13,690] [INFO] [logging.py:96:log_dist] [Rank 0] step=2700, skipped=26, lr=[1.8190352989793323e-05, 1.8190352989793323e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:13,692] [INFO] [timer.py:199:stop] epoch=0/micro_step=5400/global_step=2700, RunningAvgSamplesPerSec=24.65494582677716, CurrSamplesPerSec=24.527134355491214, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:20,192] [INFO] [logging.py:96:log_dist] [Rank 0] step=2710, skipped=26, lr=[1.6783290846078712e-05, 1.6783290846078712e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:20,195] [INFO] [timer.py:199:stop] epoch=0/micro_step=5420/global_step=2710, RunningAvgSamplesPerSec=24.65498861765556, CurrSamplesPerSec=24.612419891837018, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:26,698] [INFO] [logging.py:96:log_dist] [Rank 0] step=2720, skipped=26, lr=[1.5431933021854193e-05, 1.5431933021854193e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:26,700] [INFO] [timer.py:199:stop] epoch=0/micro_step=5440/global_step=2720, RunningAvgSamplesPerSec=24.654990223005562, CurrSamplesPerSec=24.655869728391398, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:33,199] [INFO] [logging.py:96:log_dist] [Rank 0] step=2730, skipped=26, lr=[1.4136435299125338e-05, 1.4136435299125338e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:33,201] [INFO] [timer.py:199:stop] epoch=0/micro_step=5460/global_step=2730, RunningAvgSamplesPerSec=24.655049770325885, CurrSamplesPerSec=24.695746636706943, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:39,713] [INFO] [logging.py:96:log_dist] [Rank 0] step=2740, skipped=26, lr=[1.289694702044919e-05, 1.289694702044919e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:39,714] [INFO] [timer.py:199:stop] epoch=0/micro_step=5480/global_step=2740, RunningAvgSamplesPerSec=24.65495241947876, CurrSamplesPerSec=24.22176263497969, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:46,227] [INFO] [logging.py:96:log_dist] [Rank 0] step=2750, skipped=26, lr=[1.1713611071719033e-05, 1.1713611071719033e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:46,229] [INFO] [timer.py:199:stop] epoch=0/micro_step=5500/global_step=2750, RunningAvgSamplesPerSec=24.654816889876244, CurrSamplesPerSec=24.67043548779618, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:46,866] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:09:46,866] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:09:47,493] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2751
[2023-04-14 09:09:47,493] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:09:47,493] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:09:52,696] [INFO] [logging.py:96:log_dist] [Rank 0] step=2760, skipped=27, lr=[1.0696731839116569e-05, 1.0696731839116569e-05], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:52,699] [INFO] [timer.py:199:stop] epoch=0/micro_step=5520/global_step=2760, RunningAvgSamplesPerSec=24.655320693951086, CurrSamplesPerSec=24.525144454486284, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:09:59,195] [INFO] [logging.py:96:log_dist] [Rank 0] step=2770, skipped=27, lr=[9.620455771699732e-06, 9.620455771699732e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:09:59,198] [INFO] [timer.py:199:stop] epoch=0/micro_step=5540/global_step=2770, RunningAvgSamplesPerSec=24.655405333686847, CurrSamplesPerSec=24.690004407558632, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:05,702] [INFO] [logging.py:96:log_dist] [Rank 0] step=2780, skipped=27, lr=[8.60070974201893e-06, 8.60070974201893e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:05,704] [INFO] [timer.py:199:stop] epoch=0/micro_step=5560/global_step=2780, RunningAvgSamplesPerSec=24.65538313370522, CurrSamplesPerSec=24.681913489366664, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:12,205] [INFO] [logging.py:96:log_dist] [Rank 0] step=2790, skipped=27, lr=[7.637611304491365e-06, 7.637611304491365e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:12,207] [INFO] [timer.py:199:stop] epoch=0/micro_step=5580/global_step=2790, RunningAvgSamplesPerSec=24.655430296969662, CurrSamplesPerSec=24.572037638950068, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:18,713] [INFO] [logging.py:96:log_dist] [Rank 0] step=2800, skipped=27, lr=[6.731271483305656e-06, 6.731271483305656e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:18,716] [INFO] [timer.py:199:stop] epoch=0/micro_step=5600/global_step=2800, RunningAvgSamplesPerSec=24.655375967239188, CurrSamplesPerSec=24.699191438352006, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:25,220] [INFO] [logging.py:96:log_dist] [Rank 0] step=2810, skipped=27, lr=[5.881794759623194e-06, 5.881794759623194e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:25,223] [INFO] [timer.py:199:stop] epoch=0/micro_step=5620/global_step=2810, RunningAvgSamplesPerSec=24.655357989780054, CurrSamplesPerSec=24.681886256175172, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:31,727] [INFO] [logging.py:96:log_dist] [Rank 0] step=2820, skipped=27, lr=[5.089279059533658e-06, 5.089279059533658e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:31,730] [INFO] [timer.py:199:stop] epoch=0/micro_step=5640/global_step=2820, RunningAvgSamplesPerSec=24.65533667801917, CurrSamplesPerSec=24.70323735313454, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:38,235] [INFO] [logging.py:96:log_dist] [Rank 0] step=2830, skipped=27, lr=[4.353815742766387e-06, 4.353815742766387e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:38,237] [INFO] [timer.py:199:stop] epoch=0/micro_step=5660/global_step=2830, RunningAvgSamplesPerSec=24.65530546909418, CurrSamplesPerSec=24.666436575015943, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:44,744] [INFO] [logging.py:96:log_dist] [Rank 0] step=2840, skipped=27, lr=[3.675489592158576e-06, 3.675489592158576e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:44,746] [INFO] [timer.py:199:stop] epoch=0/micro_step=5680/global_step=2840, RunningAvgSamplesPerSec=24.655260573783313, CurrSamplesPerSec=24.694147266597536, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:51,242] [INFO] [logging.py:96:log_dist] [Rank 0] step=2850, skipped=27, lr=[3.0543788038816523e-06, 3.0543788038816523e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:51,245] [INFO] [timer.py:199:stop] epoch=0/micro_step=5700/global_step=2850, RunningAvgSamplesPerSec=24.655353084506157, CurrSamplesPerSec=24.712561639754647, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:10:53,178] [INFO] [fused_optimizer.py:370:_update_scale] No Grad overflow for 100 iterations
[2023-04-14 09:10:53,178] [INFO] [fused_optimizer.py:371:_update_scale] Increasing dynamic loss scale from 65536.0 to 131072.0
[2023-04-14 09:10:53,807] [INFO] [fused_optimizer.py:362:_update_scale] 
Grad overflow on iteration 2853
[2023-04-14 09:10:53,808] [INFO] [fused_optimizer.py:363:_update_scale] Reducing dynamic loss scale from 131072.0 to 65536.0
[2023-04-14 09:10:53,808] [INFO] [logging.py:96:log_dist] [Rank 0] Overflow detected. Skipping step. Attempted loss scale: 131072.0, reducing to 65536.0
[2023-04-14 09:10:57,710] [INFO] [logging.py:96:log_dist] [Rank 0] step=2860, skipped=28, lr=[2.5443575434498935e-06, 2.5443575434498935e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:10:57,713] [INFO] [timer.py:199:stop] epoch=0/micro_step=5720/global_step=2860, RunningAvgSamplesPerSec=24.655854609614632, CurrSamplesPerSec=24.652690572087288, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:11:04,210] [INFO] [logging.py:96:log_dist] [Rank 0] step=2870, skipped=28, lr=[2.0321477449619096e-06, 2.0321477449619096e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:11:04,212] [INFO] [timer.py:199:stop] epoch=0/micro_step=5740/global_step=2870, RunningAvgSamplesPerSec=24.655927639723327, CurrSamplesPerSec=24.68009807480208, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:11:10,714] [INFO] [logging.py:96:log_dist] [Rank 0] step=2880, skipped=28, lr=[1.5773427501828975e-06, 1.5773427501828975e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:11:10,716] [INFO] [timer.py:199:stop] epoch=0/micro_step=5760/global_step=2880, RunningAvgSamplesPerSec=24.65594531027811, CurrSamplesPerSec=24.668422267607106, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:11:17,222] [INFO] [logging.py:96:log_dist] [Rank 0] step=2890, skipped=28, lr=[1.179994988182942e-06, 1.179994988182942e-06], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:11:17,224] [INFO] [timer.py:199:stop] epoch=0/micro_step=5780/global_step=2890, RunningAvgSamplesPerSec=24.655910382346576, CurrSamplesPerSec=24.678473505218935, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:11:23,749] [INFO] [logging.py:96:log_dist] [Rank 0] step=2900, skipped=28, lr=[8.401502644697078e-07, 8.401502644697078e-07], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:11:23,751] [INFO] [timer.py:199:stop] epoch=0/micro_step=5800/global_step=2900, RunningAvgSamplesPerSec=24.655631281512342, CurrSamplesPerSec=24.634871689828095, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:11:30,266] [INFO] [logging.py:96:log_dist] [Rank 0] step=2910, skipped=28, lr=[5.578477557081075e-07, 5.578477557081075e-07], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:11:30,268] [INFO] [timer.py:199:stop] epoch=0/micro_step=5820/global_step=2910, RunningAvgSamplesPerSec=24.655478050131265, CurrSamplesPerSec=24.677883630648456, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
[2023-04-14 09:11:36,766] [INFO] [logging.py:96:log_dist] [Rank 0] step=2920, skipped=28, lr=[3.3312000520391294e-07, 3.3312000520391294e-07], mom=[(0.9, 0.95), (0.9, 0.95)]
[2023-04-14 09:11:36,769] [INFO] [timer.py:199:stop] epoch=0/micro_step=5840/global_step=2920, RunningAvgSamplesPerSec=24.655541052980507, CurrSamplesPerSec=24.690404096520474, MemAllocated=7.79GB, MaxMemAllocated=26.02GB
***** Evaluating perplexity, Epoch 1/1 *****
ppl: 1.9372107982635498
saving the final model ...
[2023-04-14 09:13:04,530] [INFO] [launch.py:460:main] Process 1494 exits successfully.