| slurm submission log: 2024-05-10 08:21:53.371659 | |
| created following sbatch script: | |
| ############################### | |
| #!/bin/bash | |
| #SBATCH --account=nlp | |
| #SBATCH --cpus-per-task=16 | |
| #SBATCH --gres=gpu:2 | |
| #SBATCH --job-name=tthrush-job-2404107 | |
| #SBATCH --mem=400G | |
| #SBATCH --nodelist=sphinx1 | |
| #SBATCH --open-mode=append | |
| #SBATCH --output=/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq/train_job_output.txt | |
| #SBATCH --partition=sphinx | |
| #SBATCH --time=14-0 | |
| # activate your desired anaconda environment | |
| . /nlp/scr/tthrush/miniconda3/etc/profile.d/conda.sh ; conda activate pretraining-coreset-selection | |
| # cd to working directory | |
| cd . | |
| # launch commands | |
| srun --unbuffered run_as_child_processes 'torchrun --master_port 29502 --nproc_per_node=2 train_llm.py --dataset_id /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq --output_dir /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq --output_hub_id pythia-70m_sciq --model_id EleutherAI/pythia-70m --num_train_epochs 1 --learning_rate 1e-3 --warmup_ratio=0.1 --gradient_accumulation_steps 2' | |
| ############################### | |
| submission to slurm complete! | |
| ############################### | |
| slurm submission output | |
| Submitted batch job 7593609 | |
| ############################### | |
| slurm submission log: 2024-05-10 08:23:19.470596 | |
| created following sbatch script: | |
| ############################### | |
| #!/bin/bash | |
| #SBATCH --account=nlp | |
| #SBATCH --cpus-per-task=16 | |
| #SBATCH --gres=gpu:2 | |
| #SBATCH --job-name=tthrush-job-3619453 | |
| #SBATCH --mem=400G | |
| #SBATCH --nodelist=sphinx2 | |
| #SBATCH --open-mode=append | |
| #SBATCH --output=/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq/train_job_output.txt | |
| #SBATCH --partition=sphinx | |
| #SBATCH --time=14-0 | |
| # activate your desired anaconda environment | |
| . /nlp/scr/tthrush/miniconda3/etc/profile.d/conda.sh ; conda activate pretraining-coreset-selection | |
| # cd to working directory | |
| cd . | |
| # launch commands | |
| srun --unbuffered run_as_child_processes 'torchrun --master_port 29502 --nproc_per_node=2 train_llm.py --dataset_id /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq --output_dir /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq --output_hub_id pythia-70m_sciq --model_id EleutherAI/pythia-70m --num_train_epochs 1 --learning_rate 1e-3 --warmup_ratio=0.1 --gradient_accumulation_steps 2' | |
| ############################### | |
| submission to slurm complete! | |
| ############################### | |
| slurm submission output | |
| Submitted batch job 7593622 | |
| ############################### | |
| ############################### | |
| start time: 2024-05-10 15:02:42.471821 | |
| machine: sphinx2 | |
| conda env: pretraining-coreset-selection | |
| ############################### | |
| running following processes | |
| torchrun --master_port 29502 --nproc_per_node=2 train_llm.py --dataset_id /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq --output_dir /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq --output_hub_id pythia-70m_sciq --model_id EleutherAI/pythia-70m --num_train_epochs 1 --learning_rate 1e-3 --warmup_ratio=0.1 --gradient_accumulation_steps 2 | |
| ############################### | |
| command outputs: | |
| [2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] | |
| [2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] ***************************************** | |
| [2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. | |
| [2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] ***************************************** | |
| 05/10/2024 15:02:52 - INFO - __main__ - Script parameters ScriptArguments(dataset_id='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq', output_dir='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq', output_hub_id='pythia-70m_sciq', hf_hub_token=True, model_id='EleutherAI/pythia-70m', per_device_train_batch_size=256, num_train_epochs=1, learning_rate=0.001, gradient_accumulation_steps=2, from_scratch=True, warmup_ratio=0.1, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, weight_decay=0.01, lr_scheduler_type='cosine', local_rank=0, resume_from_checkpoint=False, deepspeed=None, peft=False) | |
| 05/10/2024 15:02:52 - INFO - __main__ - Script parameters ScriptArguments(dataset_id='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq', output_dir='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq', output_hub_id='pythia-70m_sciq', hf_hub_token=True, model_id='EleutherAI/pythia-70m', per_device_train_batch_size=256, num_train_epochs=1, learning_rate=0.001, gradient_accumulation_steps=2, from_scratch=True, warmup_ratio=0.1, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, weight_decay=0.01, lr_scheduler_type='cosine', local_rank=0, resume_from_checkpoint=False, deepspeed=None, peft=False) | |
| 0%| | 0/763 [00:00<?, ?it/s][rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) | |
| [rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration, which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator()) | |
| 0%| | 1/763 [00:45<9:35:14, 45.29s/it] 0%| | 2/763 [00:58<5:37:14, 26.59s/it] 0%| | 3/763 [01:09<4:02:06, 19.11s/it] 1%| | 4/763 [01:15<3:00:21, 14.26s/it] 1%| | 5/763 [01:20<2:18:15, 10.94s/it] 1%| | 6/763 [01:25<1:50:17, 8.74s/it] 1%| | 7/763 [01:28<1:27:09, 6.92s/it] 1%| | 8/763 [01:31<1:10:29, 5.60s/it] 1%| | 9/763 [01:33<56:45, 4.52s/it] 1%|▏ | 10/763 [01:35<45:43, 3.64s/it] 1%|▏ | 11/763 [01:36<37:31, 2.99s/it] 2%|▏ | 12/763 [01:38<31:52, 2.55s/it] 2%|▏ | 13/763 [01:39<27:34, 2.21s/it] 2%|▏ | 14/763 [01:40<23:46, 1.90s/it] 2%|▏ | 15/763 [01:41<20:46, 1.67s/it] 2%|▏ | 16/763 [01:42<17:33, 1.41s/it] 2%|▏ | 17/763 [01:43<14:55, 1.20s/it] 2%|▏ | 18/763 [01:44<13:45, 1.11s/it] 2%|▏ | 19/763 [01:45<12:10, 1.02it/s] 3%|▎ | 20/763 [01:45<11:06, 1.12it/s] 3%|▎ | 21/763 [01:46<10:04, 1.23it/s] 3%|▎ | 22/763 [01:46<09:17, 1.33it/s] 3%|▎ | 23/763 [01:47<08:41, 1.42it/s] 3%|▎ | 24/763 [01:48<08:17, 1.49it/s] 3%|▎ | 25/763 [01:48<08:02, 1.53it/s] {'loss': 9.8343, 'grad_norm': 0.9369164705276489, 'learning_rate': 0.0003246753246753247, 'epoch': 0.03} | |
| 3%|▎ | 25/763 [01:48<08:02, 1.53it/s] 3%|▎ | 26/763 [01:49<08:00, 1.53it/s] 4%|▎ | 27/763 [01:49<07:33, 1.62it/s] 4%|▎ | 28/763 [01:50<07:18, 1.68it/s] 4%|▍ | 29/763 [01:51<07:21, 1.66it/s] 4%|▍ | 30/763 [01:51<07:10, 1.70it/s] 4%|▍ | 31/763 [01:52<06:59, 1.75it/s] 4%|▍ | 32/763 [01:52<06:53, 1.77it/s] 4%|▍ | 33/763 [01:53<06:42, 1.81it/s] 4%|▍ | 34/763 [01:53<06:35, 1.84it/s] 5%|▍ | 35/763 [01:54<06:28, 1.87it/s] 5%|▍ | 36/763 [01:54<06:28, 1.87it/s] 5%|▍ | 37/763 [01:55<06:24, 1.89it/s] 5%|▍ | 38/763 [01:55<06:21, 1.90it/s] 5%|▌ | 39/763 [01:56<06:18, 1.92it/s] 5%|▌ | 40/763 [01:56<06:14, 1.93it/s] 5%|▌ | 41/763 [01:57<06:11, 1.94it/s] 6%|▌ | 42/763 [01:57<06:10, 1.95it/s] 6%|▌ | 43/763 [01:58<06:09, 1.95it/s] 6%|▌ | 44/763 [01:58<06:05, 1.97it/s] 6%|▌ | 45/763 [01:59<06:07, 1.95it/s] 6%|▌ | 46/763 [01:59<06:06, 1.96it/s] 6%|▌ | 47/763 [02:00<06:03, 1.97it/s] 6%|▋ | 48/763 [02:00<06:01, 1.98it/s] 6%|▋ | 49/763 [02:01<06:00, 1.98it/s] 7%|▋ | 50/763 [02:01<05:58, 1.99it/s] {'loss': 7.3748, 'grad_norm': 0.3545825183391571, 'learning_rate': 0.0006493506493506494, 'epoch': 0.07} | |
| 7%|▋ | 50/763 [02:01<05:58, 1.99it/s] 7%|▋ | 51/763 [02:02<05:58, 1.99it/s] 7%|▋ | 52/763 [02:02<05:57, 1.99it/s] 7%|▋ | 53/763 [02:03<05:57, 1.99it/s] 7%|▋ | 54/763 [02:03<05:55, 1.99it/s] 7%|▋ | 55/763 [02:04<05:54, 2.00it/s] 7%|▋ | 56/763 [02:04<05:54, 2.00it/s] 7%|▋ | 57/763 [02:05<05:53, 2.00it/s] 8%|▊ | 58/763 [02:05<05:53, 1.99it/s] 8%|▊ | 59/763 [02:06<05:53, 1.99it/s] 8%|▊ | 60/763 [02:06<05:52, 2.00it/s] 8%|▊ | 61/763 [02:07<05:52, 1.99it/s] 8%|▊ | 62/763 [02:07<05:51, 2.00it/s] 8%|▊ | 63/763 [02:08<05:50, 2.00it/s] 8%|▊ | 64/763 [02:08<05:48, 2.01it/s] 9%|▊ | 65/763 [02:09<05:47, 2.01it/s] 9%|▊ | 66/763 [02:09<05:57, 1.95it/s] 9%|▉ | 67/763 [02:10<05:53, 1.97it/s] 9%|▉ | 68/763 [02:10<05:51, 1.98it/s] 9%|▉ | 69/763 [02:11<05:49, 1.99it/s] 9%|▉ | 70/763 [02:11<05:47, 1.99it/s] 9%|▉ | 71/763 [02:12<05:46, 2.00it/s] 9%|▉ | 72/763 [02:12<05:45, 2.00it/s] 10%|▉ | 73/763 [02:13<05:43, 2.01it/s] 10%|▉ | 74/763 [02:13<05:42, 2.01it/s] 10%|▉ | 75/763 [02:14<05:42, 2.01it/s] {'loss': 5.8974, 'grad_norm': 0.5354925394058228, 'learning_rate': 0.000974025974025974, 'epoch': 0.1} | |
| 10%|▉ | 75/763 [02:14<05:42, 2.01it/s] 10%|▉ | 76/763 [02:14<05:41, 2.01it/s] 10%|█ | 77/763 [02:15<05:41, 2.01it/s] 10%|█ | 78/763 [02:15<05:40, 2.01it/s] 10%|█ | 79/763 [02:16<05:39, 2.01it/s] 10%|█ | 80/763 [02:16<05:39, 2.01it/s] 11%|█ | 81/763 [02:17<05:38, 2.01it/s] 11%|█ | 82/763 [02:17<05:38, 2.01it/s] 11%|█ | 83/763 [02:18<05:37, 2.01it/s] 11%|█ | 84/763 [02:18<05:37, 2.01it/s] 11%|█ | 85/763 [02:19<05:37, 2.01it/s] 11%|█▏ | 86/763 [02:19<05:36, 2.01it/s] 11%|█▏ | 87/763 [02:20<05:34, 2.02it/s] 12%|█▏ | 88/763 [02:20<05:34, 2.02it/s] 12%|█▏ | 89/763 [02:21<05:34, 2.02it/s] 12%|█▏ | 90/763 [02:21<05:33, 2.02it/s] 12%|█▏ | 91/763 [02:22<05:32, 2.02it/s] 12%|█▏ | 92/763 [02:22<05:31, 2.02it/s] 12%|█▏ | 93/763 [02:23<05:32, 2.02it/s] 12%|█▏ | 94/763 [02:23<05:31, 2.02it/s] 12%|█▏ | 95/763 [02:24<05:30, 2.02it/s] 13%|█▎ | 96/763 [02:24<05:30, 2.02it/s] 13%|█▎ | 97/763 [02:25<05:30, 2.01it/s] 13%|█▎ | 98/763 [02:25<05:29, 2.02it/s] 13%|█▎ | 99/763 [02:26<05:29, 2.02it/s] 13%|█▎ | 100/763 [02:26<05:32, 2.00it/s]{'loss': 5.0788, 'grad_norm': 0.4961775541305542, 'learning_rate': 0.0009972289418801728, 'epoch': 0.13} | |
| 13%|█▎ | 100/763 [02:26<05:32, 2.00it/s] 13%|█▎ | 101/763 [02:27<05:31, 2.00it/s] 13%|█▎ | 102/763 [02:27<05:30, 2.00it/s] 13%|█▎ | 103/763 [02:28<05:28, 2.01it/s] 14%|█▎ | 104/763 [02:28<05:27, 2.01it/s] 14%|█▍ | 105/763 [02:29<05:27, 2.01it/s] 14%|█▍ | 106/763 [02:29<05:26, 2.01it/s] 14%|█▍ | 107/763 [02:30<05:25, 2.01it/s] 14%|█▍ | 108/763 [02:30<05:24, 2.02it/s] 14%|█▍ | 109/763 [02:31<05:23, 2.02it/s] 14%|█▍ | 110/763 [02:31<05:23, 2.02it/s] 15%|█▍ | 111/763 [02:32<05:23, 2.02it/s] 15%|█▍ | 112/763 [02:32<05:22, 2.02it/s] 15%|█▍ | 113/763 [02:33<05:21, 2.02it/s] 15%|█▍ | 114/763 [02:33<05:21, 2.02it/s] 15%|█▌ | 115/763 [02:34<05:20, 2.02it/s] 15%|█▌ | 116/763 [02:34<05:19, 2.02it/s] 15%|█▌ | 117/763 [02:35<05:19, 2.02it/s] 15%|█▌ | 118/763 [02:35<05:18, 2.02it/s] 16%|█▌ | 119/763 [02:36<05:18, 2.02it/s] 16%|█▌ | 120/763 [02:36<05:17, 2.02it/s] 16%|█▌ | 121/763 [02:37<05:17, 2.02it/s] 16%|█▌ | 122/763 [02:37<05:17, 2.02it/s] 16%|█▌ | 123/763 [02:38<05:16, 2.02it/s] 16%|█▋ | 124/763 [02:38<05:15, 2.02it/s] 16%|█▋ | 125/763 [02:39<05:16, 2.02it/s] {'loss': 4.6084, 'grad_norm': 0.2889716327190399, 'learning_rate': 0.0009879683689693263, 'epoch': 0.16} | |
| 16%|█▋ | 125/763 [02:39<05:16, 2.02it/s] 17%|█▋ | 126/763 [02:39<05:15, 2.02it/s] 17%|█▋ | 127/763 [02:40<05:15, 2.01it/s] 17%|█▋ | 128/763 [02:40<05:15, 2.02it/s] 17%|█▋ | 129/763 [02:41<05:14, 2.02it/s] 17%|█▋ | 130/763 [02:41<05:13, 2.02it/s] 17%|█▋ | 131/763 [02:42<05:12, 2.02it/s] 17%|█▋ | 132/763 [02:42<05:12, 2.02it/s] 17%|█▋ | 133/763 [02:43<05:12, 2.02it/s] 18%|█▊ | 134/763 [02:43<05:11, 2.02it/s] 18%|█▊ | 135/763 [02:44<05:11, 2.02it/s] 18%|█▊ | 136/763 [02:44<05:10, 2.02it/s] 18%|█▊ | 137/763 [02:45<05:09, 2.02it/s] 18%|█▊ | 138/763 [02:45<05:09, 2.02it/s] 18%|█▊ | 139/763 [02:46<05:08, 2.02it/s] 18%|█▊ | 140/763 [02:46<05:08, 2.02it/s] 18%|█▊ | 141/763 [02:47<05:07, 2.02it/s] 19%|█▊ | 142/763 [02:47<05:07, 2.02it/s] 19%|█▊ | 143/763 [02:48<05:06, 2.02it/s] 19%|█▉ | 144/763 [02:48<05:05, 2.02it/s] 19%|█▉ | 145/763 [02:49<05:05, 2.02it/s] 19%|█▉ | 146/763 [02:49<05:04, 2.02it/s] 19%|█▉ | 147/763 [02:50<05:04, 2.02it/s] 19%|█▉ | 148/763 [02:50<05:04, 2.02it/s] 20%|█▉ | 149/763 [02:51<05:03, 2.02it/s] 20%|█▉ | 150/763 [02:51<05:03, 2.02it/s]{'loss': 4.2413, 'grad_norm': 0.44144755601882935, 'learning_rate': 0.0009723185625357323, 'epoch': 0.2} | |
| 20%|█▉ | 150/763 [02:51<05:03, 2.02it/s] 20%|█▉ | 151/763 [02:52<05:03, 2.02it/s] 20%|█▉ | 152/763 [02:52<05:02, 2.02it/s] 20%|██ | 153/763 [02:53<05:01, 2.02it/s] 20%|██ | 154/763 [02:53<05:01, 2.02it/s] 20%|██ | 155/763 [02:54<05:00, 2.02it/s] 20%|██ | 156/763 [02:54<05:00, 2.02it/s] 21%|██ | 157/763 [02:55<04:59, 2.02it/s] 21%|██ | 158/763 [02:55<04:59, 2.02it/s] 21%|██ | 159/763 [02:56<04:58, 2.02it/s] 21%|██ | 160/763 [02:56<04:58, 2.02it/s] 21%|██ | 161/763 [02:57<04:57, 2.02it/s] 21%|██ | 162/763 [02:57<04:57, 2.02it/s] 21%|██▏ | 163/763 [02:58<04:57, 2.02it/s] 21%|██▏ | 164/763 [02:58<04:56, 2.02it/s] 22%|██▏ | 165/763 [02:59<04:55, 2.02it/s] 22%|██▏ | 166/763 [02:59<04:55, 2.02it/s] 22%|██▏ | 167/763 [03:00<04:54, 2.02it/s] 22%|██▏ | 168/763 [03:00<04:54, 2.02it/s] 22%|██▏ | 169/763 [03:01<04:53, 2.02it/s] 22%|██▏ | 170/763 [03:01<04:52, 2.02it/s] 22%|██▏ | 171/763 [03:02<04:52, 2.02it/s] 23%|██▎ | 172/763 [03:02<04:51, 2.02it/s] 23%|██▎ | 173/763 [03:03<04:51, 2.02it/s] 23%|██▎ | 174/763 [03:03<04:51, 2.02it/s] 23%|██▎ | 175/763 [03:04<04:50, 2.02it/s]{'loss': 3.9435, 'grad_norm': 0.39299631118774414, 'learning_rate': 0.0009504844339512095, 'epoch': 0.23} | |
| 23%|██▎ | 175/763 [03:04<04:50, 2.02it/s] 23%|██▎ | 176/763 [03:04<04:50, 2.02it/s] 23%|██▎ | 177/763 [03:04<04:50, 2.02it/s] 23%|██▎ | 178/763 [03:05<04:49, 2.02it/s] 23%|██▎ | 179/763 [03:05<04:48, 2.02it/s] 24%|██▎ | 180/763 [03:06<04:48, 2.02it/s] 24%|██▎ | 181/763 [03:06<04:47, 2.02it/s] 24%|██▍ | 182/763 [03:07<04:47, 2.02it/s] 24%|██▍ | 183/763 [03:07<04:46, 2.02it/s] 24%|██▍ | 184/763 [03:08<04:46, 2.02it/s] 24%|██▍ | 185/763 [03:08<04:46, 2.02it/s] 24%|██▍ | 186/763 [03:09<04:45, 2.02it/s] 25%|██▍ | 187/763 [03:09<04:45, 2.02it/s] 25%|██▍ | 188/763 [03:10<04:44, 2.02it/s] 25%|██▍ | 189/763 [03:10<04:43, 2.02it/s] 25%|██▍ | 190/763 [03:11<04:43, 2.02it/s] 25%|██▌ | 191/763 [03:11<04:42, 2.02it/s] 25%|██▌ | 192/763 [03:12<04:42, 2.02it/s] 25%|██▌ | 193/763 [03:12<04:41, 2.02it/s] 25%|██▌ | 194/763 [03:13<04:41, 2.02it/s] 26%|██▌ | 195/763 [03:13<04:41, 2.02it/s] 26%|██▌ | 196/763 [03:14<04:40, 2.02it/s] 26%|██▌ | 197/763 [03:14<04:39, 2.02it/s] 26%|██▌ | 198/763 [03:15<04:39, 2.02it/s] 26%|██▌ | 199/763 [03:15<04:38, 2.02it/s] 26%|██▌ | 200/763 [03:16<04:38, 2.02it/s]{'loss': 3.6901, 'grad_norm': 0.34031203389167786, 'learning_rate': 0.0009227518692591244, 'epoch': 0.26} | |
| 26%|██▌ | 200/763 [03:16<04:38, 2.02it/s] 26%|██▋ | 201/763 [03:16<04:38, 2.02it/s] 26%|██▋ | 202/763 [03:17<04:37, 2.02it/s] 27%|██▋ | 203/763 [03:17<04:36, 2.02it/s] 27%|██▋ | 204/763 [03:18<04:36, 2.02it/s] 27%|██▋ | 205/763 [03:18<04:35, 2.03it/s] 27%|██▋ | 206/763 [03:19<04:35, 2.03it/s] 27%|██▋ | 207/763 [03:19<04:34, 2.02it/s] 27%|██▋ | 208/763 [03:20<04:34, 2.02it/s] 27%|██▋ | 209/763 [03:20<04:33, 2.02it/s] 28%|██▊ | 210/763 [03:21<04:33, 2.02it/s] 28%|██▊ | 211/763 [03:21<04:32, 2.02it/s] 28%|██▊ | 212/763 [03:22<04:32, 2.02it/s] 28%|██▊ | 213/763 [03:22<04:31, 2.02it/s] 28%|██▊ | 214/763 [03:23<04:31, 2.02it/s] 28%|██▊ | 215/763 [03:23<04:30, 2.03it/s] 28%|██▊ | 216/763 [03:24<04:30, 2.02it/s] 28%|██▊ | 217/763 [03:24<04:29, 2.03it/s] 29%|██▊ | 218/763 [03:25<04:29, 2.03it/s] 29%|██▊ | 219/763 [03:25<04:28, 2.03it/s] 29%|██▉ | 220/763 [03:26<04:28, 2.03it/s] 29%|██▉ | 221/763 [03:26<04:27, 2.02it/s] 29%|██▉ | 222/763 [03:27<04:26, 2.03it/s] 29%|██▉ | 223/763 [03:27<04:26, 2.03it/s] 29%|██▉ | 224/763 [03:28<04:26, 2.02it/s] 29%|██▉ | 225/763 [03:28<04:25, 2.02it/s]{'loss': 3.4966, 'grad_norm': 0.37306490540504456, 'learning_rate': 0.0008894839859139472, 'epoch': 0.29} | |
| 29%|██▉ | 225/763 [03:28<04:25, 2.02it/s] 30%|██▉ | 226/763 [03:29<04:25, 2.02it/s] 30%|██▉ | 227/763 [03:29<04:24, 2.02it/s] 30%|██▉ | 228/763 [03:30<04:24, 2.02it/s] 30%|███ | 229/763 [03:30<04:23, 2.02it/s] 30%|███ | 230/763 [03:31<04:23, 2.02it/s] 30%|███ | 231/763 [03:31<04:22, 2.03it/s] 30%|███ | 232/763 [03:32<04:22, 2.02it/s] 31%|███ | 233/763 [03:32<04:22, 2.02it/s] 31%|███ | 234/763 [03:33<04:21, 2.02it/s] 31%|███ | 235/763 [03:33<04:20, 2.02it/s] 31%|███ | 236/763 [03:34<04:20, 2.02it/s] 31%|███ | 237/763 [03:34<04:20, 2.02it/s] 31%|███ | 238/763 [03:35<04:19, 2.02it/s] 31%|███▏ | 239/763 [03:35<04:18, 2.02it/s] 31%|███▏ | 240/763 [03:36<04:18, 2.02it/s] 32%|███▏ | 241/763 [03:36<04:17, 2.02it/s] 32%|███▏ | 242/763 [03:37<04:17, 2.02it/s] 32%|███▏ | 243/763 [03:37<04:17, 2.02it/s] 32%|███▏ | 244/763 [03:38<04:16, 2.02it/s] 32%|███▏ | 245/763 [03:38<04:16, 2.02it/s] 32%|███▏ | 246/763 [03:39<04:15, 2.02it/s] 32%|███▏ | 247/763 [03:39<04:15, 2.02it/s] 33%|███▎ | 248/763 [03:40<04:14, 2.02it/s] 33%|███▎ | 249/763 [03:40<04:14, 2.02it/s] 33%|███▎ | 250/763 [03:41<04:13, 2.03it/s]{'loss': 3.3449, 'grad_norm': 0.4239332973957062, 'learning_rate': 0.0008511163782882168, 'epoch': 0.33} | |
| 33%|███▎ | 250/763 [03:41<04:13, 2.03it/s] 33%|███▎ | 251/763 [03:41<04:13, 2.02it/s] 33%|███▎ | 252/763 [03:42<04:12, 2.02it/s] 33%|███▎ | 253/763 [03:42<04:12, 2.02it/s] 33%|███▎ | 254/763 [03:43<04:11, 2.02it/s] 33%|███▎ | 255/763 [03:43<04:11, 2.02it/s] 34%|███▎ | 256/763 [03:44<04:10, 2.02it/s] 34%|███▎ | 257/763 [03:44<04:09, 2.02it/s] 34%|███▍ | 258/763 [03:45<04:09, 2.03it/s] 34%|███▍ | 259/763 [03:45<04:08, 2.03it/s] 34%|███▍ | 260/763 [03:46<04:08, 2.02it/s] 34%|███▍ | 261/763 [03:46<04:08, 2.02it/s] 34%|███▍ | 262/763 [03:47<04:07, 2.02it/s] 34%|███▍ | 263/763 [03:47<04:07, 2.02it/s] 35%|███▍ | 264/763 [03:47<04:06, 2.02it/s] 35%|███▍ | 265/763 [03:48<04:06, 2.02it/s] 35%|███▍ | 266/763 [03:48<04:05, 2.02it/s] 35%|███▍ | 267/763 [03:49<04:05, 2.02it/s] 35%|███▌ | 268/763 [03:49<04:04, 2.02it/s] 35%|███▌ | 269/763 [03:50<04:04, 2.02it/s] 35%|███▌ | 270/763 [03:50<04:03, 2.02it/s] 36%|███▌ | 271/763 [03:51<04:03, 2.02it/s] 36%|███▌ | 272/763 [03:51<04:02, 2.02it/s] 36%|███▌ | 273/763 [03:52<04:02, 2.02it/s] 36%|███▌ | 274/763 [03:52<04:01, 2.02it/s] 36%|███▌ | 275/763 [03:53<04:01, 2.02it/s] {'loss': 3.1922, 'grad_norm': 0.4128451645374298, 'learning_rate': 0.0008081514142000517, 'epoch': 0.36} | |
| 36%|███▌ | 275/763 [03:53<04:01, 2.02it/s] 36%|███▌ | 276/763 [03:53<04:00, 2.02it/s] 36%|███▋ | 277/763 [03:54<04:00, 2.02it/s] 36%|███▋ | 278/763 [03:54<03:59, 2.02it/s] 37%|███▋ | 279/763 [03:55<03:59, 2.02it/s] 37%|███▋ | 280/763 [03:55<03:58, 2.02it/s] 37%|███▋ | 281/763 [03:56<03:58, 2.02it/s] 37%|███▋ | 282/763 [03:56<03:57, 2.02it/s] 37%|███▋ | 283/763 [03:57<03:57, 2.02it/s] 37%|███▋ | 284/763 [03:57<03:56, 2.02it/s] 37%|███▋ | 285/763 [03:58<03:56, 2.02it/s] 37%|███▋ | 286/763 [03:58<03:55, 2.02it/s] 38%|███▊ | 287/763 [03:59<03:55, 2.02it/s] 38%|███▊ | 288/763 [03:59<03:55, 2.02it/s] 38%|███▊ | 289/763 [04:00<03:54, 2.02it/s] 38%|███▊ | 290/763 [04:00<03:54, 2.02it/s] 38%|███▊ | 291/763 [04:01<03:53, 2.02it/s] 38%|███▊ | 292/763 [04:01<03:53, 2.02it/s] 38%|███▊ | 293/763 [04:02<03:52, 2.02it/s] 39%|███▊ | 294/763 [04:02<03:51, 2.02it/s] 39%|███▊ | 295/763 [04:03<03:51, 2.02it/s] 39%|███▉ | 296/763 [04:03<03:50, 2.02it/s] 39%|███▉ | 297/763 [04:04<03:50, 2.02it/s] 39%|███▉ | 298/763 [04:04<03:49, 2.02it/s] 39%|███▉ | 299/763 [04:05<03:49, 2.02it/s] 39%|███▉ | 300/763 [04:05<03:48, 2.02it/s] {'loss': 3.0883, 'grad_norm': 0.3450576961040497, 'learning_rate': 0.0007611516571398591, 'epoch': 0.39} | |
| 39%|███▉ | 300/763 [04:05<03:48, 2.02it/s] 39%|███▉ | 301/763 [04:06<03:48, 2.02it/s] 40%|███▉ | 302/763 [04:06<03:48, 2.02it/s] 40%|███▉ | 303/763 [04:07<03:47, 2.02it/s] 40%|███▉ | 304/763 [04:07<03:47, 2.02it/s] 40%|███▉ | 305/763 [04:08<03:46, 2.02it/s] 40%|████ | 306/763 [04:08<03:45, 2.02it/s] 40%|████ | 307/763 [04:09<03:45, 2.02it/s] 40%|████ | 308/763 [04:09<03:44, 2.02it/s] 40%|████ | 309/763 [04:10<03:44, 2.02it/s] 41%|████ | 310/763 [04:10<03:43, 2.02it/s] 41%|████ | 311/763 [04:11<03:43, 2.02it/s] 41%|████ | 312/763 [04:11<03:42, 2.02it/s] 41%|████ | 313/763 [04:12<03:42, 2.02it/s] 41%|████ | 314/763 [04:12<03:41, 2.02it/s] 41%|████▏ | 315/763 [04:13<03:41, 2.02it/s] 41%|████▏ | 316/763 [04:13<03:40, 2.02it/s] 42%|████▏ | 317/763 [04:14<03:40, 2.02it/s] 42%|████▏ | 318/763 [04:14<03:39, 2.02it/s] 42%|████▏ | 319/763 [04:15<03:39, 2.02it/s] 42%|████▏ | 320/763 [04:15<03:38, 2.02it/s] 42%|████▏ | 321/763 [04:16<03:38, 2.02it/s] 42%|████▏ | 322/763 [04:16<03:38, 2.02it/s] 42%|████▏ | 323/763 [04:17<03:37, 2.02it/s] 42%|████▏ | 324/763 [04:17<03:36, 2.02it/s] 43%|████▎ | 325/763 [04:18<03:36, 2.03it/s] {'loss': 2.9938, 'grad_norm': 0.4092760384082794, 'learning_rate': 0.000710732500322581, 'epoch': 0.43} | |
| 43%|████▎ | 325/763 [04:18<03:36, 2.03it/s] 43%|████▎ | 326/763 [04:18<03:36, 2.02it/s] 43%|████▎ | 327/763 [04:19<03:35, 2.02it/s] 43%|████▎ | 328/763 [04:19<03:35, 2.02it/s] 43%|████▎ | 329/763 [04:20<03:34, 2.02it/s] 43%|████▎ | 330/763 [04:20<03:33, 2.02it/s] 43%|████▎ | 331/763 [04:21<03:33, 2.02it/s] 44%|████▎ | 332/763 [04:21<03:33, 2.02it/s] 44%|████▎ | 333/763 [04:22<03:32, 2.02it/s] 44%|████▍ | 334/763 [04:22<03:31, 2.02it/s] 44%|████▍ | 335/763 [04:23<03:31, 2.02it/s] 44%|████▍ | 336/763 [04:23<03:31, 2.02it/s] 44%|████▍ | 337/763 [04:24<03:30, 2.02it/s] 44%|████▍ | 338/763 [04:24<03:30, 2.02it/s] 44%|████▍ | 339/763 [04:25<03:29, 2.02it/s] 45%|████▍ | 340/763 [04:25<03:29, 2.02it/s] 45%|████▍ | 341/763 [04:26<03:28, 2.02it/s] 45%|████▍ | 342/763 [04:26<03:28, 2.02it/s] 45%|████▍ | 343/763 [04:27<03:27, 2.02it/s] 45%|████▌ | 344/763 [04:27<03:26, 2.02it/s] 45%|████▌ | 345/763 [04:28<03:26, 2.03it/s] 45%|████▌ | 346/763 [04:28<03:26, 2.02it/s] 45%|████▌ | 347/763 [04:29<03:25, 2.02it/s] 46%|████▌ | 348/763 [04:29<03:25, 2.02it/s] 46%|████▌ | 349/763 [04:30<03:24, 2.02it/s] 46%|████▌ | 350/763 [04:30<03:24, 2.02it/s]{'loss': 2.9083, 'grad_norm': 0.3510383367538452, 'learning_rate': 0.0006575541090118104, 'epoch': 0.46} | |
| 46%|████▌ | 350/763 [04:30<03:24, 2.02it/s] 46%|████▌ | 351/763 [04:31<03:24, 2.02it/s] 46%|████▌ | 352/763 [04:31<03:23, 2.02it/s] 46%|████▋ | 353/763 [04:32<03:22, 2.02it/s] 46%|████▋ | 354/763 [04:32<03:22, 2.02it/s] 47%|████▋ | 355/763 [04:32<03:21, 2.02it/s] 47%|████▋ | 356/763 [04:33<03:21, 2.02it/s] 47%|████▋ | 357/763 [04:33<03:20, 2.02it/s] 47%|████▋ | 358/763 [04:34<03:20, 2.02it/s] 47%|████▋ | 359/763 [04:34<03:19, 2.02it/s] 47%|████▋ | 360/763 [04:35<03:19, 2.02it/s] 47%|████▋ | 361/763 [04:35<03:18, 2.02it/s] 47%|████▋ | 362/763 [04:36<03:18, 2.02it/s] 48%|████▊ | 363/763 [04:36<03:17, 2.02it/s] 48%|████▊ | 364/763 [04:37<03:17, 2.02it/s] 48%|████▊ | 365/763 [04:37<03:16, 2.02it/s] 48%|████▊ | 366/763 [04:38<03:16, 2.02it/s] 48%|████▊ | 367/763 [04:38<03:15, 2.02it/s] 48%|████▊ | 368/763 [04:39<03:15, 2.02it/s] 48%|████▊ | 369/763 [04:39<03:14, 2.02it/s] 48%|████▊ | 370/763 [04:40<03:14, 2.02it/s] 49%|████▊ | 371/763 [04:40<03:13, 2.02it/s] 49%|████▉ | 372/763 [04:41<03:13, 2.02it/s] 49%|████▉ | 373/763 [04:41<03:12, 2.03it/s] 49%|████▉ | 374/763 [04:42<03:12, 2.03it/s] 49%|████▉ | 375/763 [04:42<03:11, 2.02it/s]{'loss': 2.8176, 'grad_norm': 0.3813249170780182, 'learning_rate': 0.0006023127766192824, 'epoch': 0.49} | |
| 49%|████▉ | 375/763 [04:42<03:11, 2.02it/s] 49%|████▉ | 376/763 [04:43<03:11, 2.02it/s] 49%|████▉ | 377/763 [04:43<03:10, 2.02it/s] 50%|████▉ | 378/763 [04:44<03:10, 2.02it/s] 50%|████▉ | 379/763 [04:44<03:09, 2.02it/s] 50%|████▉ | 380/763 [04:45<03:09, 2.03it/s] 50%|████▉ | 381/763 [04:45<03:08, 2.02it/s] 50%|█████ | 382/763 [04:46<03:08, 2.02it/s] 50%|█████ | 383/763 [04:46<03:07, 2.02it/s] 50%|█████ | 384/763 [04:47<03:07, 2.02it/s] 50%|█████ | 385/763 [04:47<03:07, 2.02it/s] 51%|█████ | 386/763 [04:48<03:06, 2.02it/s] 51%|█████ | 387/763 [04:48<03:05, 2.02it/s] 51%|█████ | 388/763 [04:49<03:05, 2.02it/s] 51%|█████ | 389/763 [04:49<03:04, 2.03it/s] 51%|█████ | 390/763 [04:50<03:04, 2.02it/s] 51%|█████ | 391/763 [04:50<03:03, 2.03it/s] 51%|█████▏ | 392/763 [04:51<03:03, 2.02it/s] 52%|█████▏ | 393/763 [04:51<03:02, 2.02it/s] 52%|█████▏ | 394/763 [04:52<03:02, 2.02it/s] 52%|█████▏ | 395/763 [04:52<03:02, 2.02it/s] 52%|█████▏ | 396/763 [04:53<03:01, 2.02it/s] 52%|█████▏ | 397/763 [04:53<03:00, 2.02it/s] 52%|█████▏ | 398/763 [04:54<03:00, 2.02it/s] 52%|█████▏ | 399/763 [04:54<02:59, 2.02it/s] 52%|█████▏ | 400/763 [04:55<02:59, 2.02it/s]{'loss': 2.7491, 'grad_norm': 0.34457528591156006, 'learning_rate': 0.0005457318077590012, 'epoch': 0.52} | |
| 52%|█████▏ | 400/763 [04:55<02:59, 2.02it/s] 53%|█████▎ | 401/763 [04:55<02:59, 2.02it/s] 53%|█████▎ | 402/763 [04:56<02:58, 2.02it/s] 53%|█████▎ | 403/763 [04:56<02:58, 2.02it/s] 53%|█████▎ | 404/763 [04:57<02:57, 2.02it/s] 53%|█████▎ | 405/763 [04:57<02:57, 2.02it/s] 53%|█████▎ | 406/763 [04:58<02:56, 2.02it/s] 53%|█████▎ | 407/763 [04:58<02:56, 2.02it/s] 53%|█████▎ | 408/763 [04:59<02:55, 2.02it/s] 54%|█████▎ | 409/763 [04:59<02:55, 2.02it/s] 54%|█████▎ | 410/763 [05:00<02:54, 2.02it/s] 54%|█████▍ | 411/763 [05:00<02:54, 2.02it/s] 54%|█████▍ | 412/763 [05:01<02:53, 2.02it/s] 54%|█████▍ | 413/763 [05:01<02:53, 2.02it/s] 54%|█████▍ | 414/763 [05:02<02:52, 2.02it/s] 54%|█████▍ | 415/763 [05:02<02:51, 2.02it/s] 55%|█████▍ | 416/763 [05:03<02:51, 2.02it/s] 55%|█████▍ | 417/763 [05:03<02:50, 2.03it/s] 55%|█████▍ | 418/763 [05:04<02:50, 2.03it/s] 55%|█████▍ | 419/763 [05:04<02:49, 2.03it/s] 55%|█████▌ | 420/763 [05:05<02:49, 2.03it/s] 55%|█████▌ | 421/763 [05:05<02:48, 2.03it/s] 55%|█████▌ | 422/763 [05:06<02:48, 2.03it/s] 55%|█████▌ | 423/763 [05:06<02:47, 2.02it/s] 56%|█████▌ | 424/763 [05:07<02:47, 2.03it/s] 56%|█████▌ | 425/763 [05:07<02:47, 2.02it/s]{'loss': 2.6858, 'grad_norm': 0.365237295627594, 'learning_rate': 0.0004885520476290998, 'epoch': 0.56} | |
| 56%|█████▌ | 425/763 [05:07<02:47, 2.02it/s] 56%|█████▌ | 426/763 [05:08<02:46, 2.02it/s] 56%|█████▌ | 427/763 [05:08<02:46, 2.02it/s] 56%|█████▌ | 428/763 [05:09<02:45, 2.02it/s] 56%|█████▌ | 429/763 [05:09<02:45, 2.02it/s] 56%|█████▋ | 430/763 [05:10<02:44, 2.02it/s] 56%|█████▋ | 431/763 [05:10<02:43, 2.03it/s] 57%|█████▋ | 432/763 [05:11<02:43, 2.03it/s] 57%|█████▋ | 433/763 [05:11<02:42, 2.02it/s] 57%|█████▋ | 434/763 [05:12<02:42, 2.02it/s] 57%|█████▋ | 435/763 [05:12<02:42, 2.02it/s] 57%|█████▋ | 436/763 [05:13<02:41, 2.02it/s] 57%|█████▋ | 437/763 [05:13<02:40, 2.03it/s] 57%|█████▋ | 438/763 [05:14<02:40, 2.02it/s] 58%|█████▊ | 439/763 [05:14<02:40, 2.02it/s] 58%|█████▊ | 440/763 [05:15<02:39, 2.02it/s] 58%|█████▊ | 441/763 [05:15<02:38, 2.03it/s] 58%|█████▊ | 442/763 [05:15<02:38, 2.02it/s] 58%|█████▊ | 443/763 [05:16<02:37, 2.03it/s] 58%|█████▊ | 444/763 [05:16<02:37, 2.02it/s] 58%|█████▊ | 445/763 [05:17<02:36, 2.03it/s] 58%|█████▊ | 446/763 [05:17<02:36, 2.03it/s] 59%|█████▊ | 447/763 [05:18<02:36, 2.03it/s] 59%|█████▊ | 448/763 [05:18<02:35, 2.03it/s] 59%|█████▉ | 449/763 [05:19<02:34, 2.03it/s] 59%|█████▉ | 450/763 [05:19<02:34, 2.03it/s]{'loss': 2.6339, 'grad_norm': 0.3599332869052887, 'learning_rate': 0.00043152218172535383, 'epoch': 0.59} | |
| 59%|█████▉ | 450/763 [05:19<02:34, 2.03it/s] 59%|█████▉ | 451/763 [05:20<02:34, 2.02it/s] 59%|█████▉ | 452/763 [05:20<02:33, 2.02it/s] 59%|█████▉ | 453/763 [05:21<02:33, 2.02it/s] 60%|█████▉ | 454/763 [05:21<02:32, 2.02it/s] 60%|█████▉ | 455/763 [05:22<02:32, 2.02it/s] 60%|█████▉ | 456/763 [05:22<02:31, 2.02it/s] 60%|█████▉ | 457/763 [05:23<02:31, 2.02it/s] 60%|██████ | 458/763 [05:23<02:30, 2.03it/s] 60%|██████ | 459/763 [05:24<02:30, 2.02it/s] 60%|██████ | 460/763 [05:24<02:29, 2.02it/s] 60%|██████ | 461/763 [05:25<02:29, 2.02it/s] 61%|██████ | 462/763 [05:25<02:28, 2.02it/s] 61%|██████ | 463/763 [05:26<02:28, 2.02it/s] 61%|██████ | 464/763 [05:26<02:27, 2.02it/s] 61%|██████ | 465/763 [05:27<02:27, 2.02it/s] 61%|██████ | 466/763 [05:27<02:26, 2.02it/s] 61%|██████ | 467/763 [05:28<02:26, 2.02it/s] 61%|██████▏ | 468/763 [05:28<02:25, 2.02it/s] 61%|██████▏ | 469/763 [05:29<02:25, 2.03it/s] 62%|██████▏ | 470/763 [05:29<02:24, 2.02it/s] 62%|██████▏ | 471/763 [05:30<02:24, 2.02it/s] 62%|██████▏ | 472/763 [05:30<02:23, 2.02it/s] 62%|██████▏ | 473/763 [05:31<02:23, 2.02it/s] 62%|██████▏ | 474/763 [05:31<02:22, 2.02it/s] 62%|██████▏ | 475/763 [05:32<02:22, 2.02it/s]{'loss': 2.5753, 'grad_norm': 0.36764204502105713, 'learning_rate': 0.0003753889328974423, 'epoch': 0.62} | |
| 62%|██████▏ | 475/763 [05:32<02:22, 2.02it/s] 62%|██████▏ | 476/763 [05:32<02:22, 2.01it/s] 63%|██████▎ | 477/763 [05:33<02:21, 2.02it/s] 63%|██████▎ | 478/763 [05:33<02:21, 2.02it/s] 63%|██████▎ | 479/763 [05:34<02:20, 2.02it/s] 63%|██████▎ | 480/763 [05:34<02:20, 2.02it/s] 63%|██████▎ | 481/763 [05:35<02:19, 2.02it/s] 63%|██████▎ | 482/763 [05:35<02:19, 2.02it/s] 63%|██████▎ | 483/763 [05:36<02:18, 2.02it/s] 63%|██████▎ | 484/763 [05:36<02:17, 2.02it/s] 64%|██████▎ | 485/763 [05:37<02:17, 2.02it/s] 64%|██████▎ | 486/763 [05:37<02:16, 2.02it/s] 64%|██████▍ | 487/763 [05:38<02:16, 2.02it/s] 64%|██████▍ | 488/763 [05:38<02:15, 2.02it/s] 64%|██████▍ | 489/763 [05:39<02:15, 2.02it/s] 64%|██████▍ | 490/763 [05:39<02:14, 2.03it/s] 64%|██████▍ | 491/763 [05:40<02:14, 2.02it/s] 64%|██████▍ | 492/763 [05:40<02:13, 2.03it/s] 65%|██████▍ | 493/763 [05:41<02:13, 2.02it/s] 65%|██████▍ | 494/763 [05:41<02:12, 2.03it/s] 65%|██████▍ | 495/763 [05:42<02:12, 2.03it/s] 65%|██████▌ | 496/763 [05:42<02:11, 2.02it/s] 65%|██████▌ | 497/763 [05:43<02:11, 2.03it/s] 65%|██████▌ | 498/763 [05:43<02:10, 2.03it/s] 65%|██████▌ | 499/763 [05:44<02:10, 2.02it/s] 66%|██████▌ | 500/763 [05:44<02:09, 2.02it/s]{'loss': 2.5207, 'grad_norm': 0.35472121834754944, 'learning_rate': 0.00032088728410319416, 'epoch': 0.66} | |
| 66%|██████▌ | 500/763 [05:44<02:09, 2.02it/s] 66%|██████▌ | 501/763 [05:45<02:09, 2.02it/s] 66%|██████▌ | 502/763 [05:45<02:09, 2.02it/s] 66%|██████▌ | 503/763 [05:46<02:08, 2.02it/s] 66%|██████▌ | 504/763 [05:46<02:08, 2.02it/s] 66%|██████▌ | 505/763 [05:47<02:07, 2.02it/s] 66%|██████▋ | 506/763 [05:47<02:07, 2.02it/s] 66%|██████▋ | 507/763 [05:48<02:06, 2.02it/s] 67%|██████▋ | 508/763 [05:48<02:05, 2.03it/s] 67%|██████▋ | 509/763 [05:49<02:05, 2.03it/s] 67%|██████▋ | 510/763 [05:49<02:04, 2.03it/s] 67%|██████▋ | 511/763 [05:50<02:04, 2.02it/s] 67%|██████▋ | 512/763 [05:50<02:03, 2.02it/s] 67%|██████▋ | 513/763 [05:51<02:03, 2.02it/s] 67%|██████▋ | 514/763 [05:51<02:03, 2.02it/s] 67%|██████▋ | 515/763 [05:52<02:02, 2.02it/s] 68%|██████▊ | 516/763 [05:52<02:01, 2.02it/s] 68%|██████▊ | 517/763 [05:53<02:01, 2.02it/s] 68%|██████▊ | 518/763 [05:53<02:00, 2.03it/s] 68%|██████▊ | 519/763 [05:54<02:00, 2.03it/s] 68%|██████▊ | 520/763 [05:54<01:59, 2.03it/s] 68%|██████▊ | 521/763 [05:55<01:59, 2.03it/s] 68%|██████▊ | 522/763 [05:55<01:59, 2.02it/s] 69%|██████▊ | 523/763 [05:56<01:58, 2.03it/s] 69%|██████▊ | 524/763 [05:56<01:58, 2.02it/s] 69%|██████▉ | 525/763 [05:57<01:57, 2.03it/s]{'loss': 2.4755, 'grad_norm': 0.342868447303772, 'learning_rate': 0.0002687308548795825, 'epoch': 0.69} | |
| 69%|██████▉ | 525/763 [05:57<01:57, 2.03it/s] 69%|██████▉ | 526/763 [05:57<01:57, 2.02it/s] 69%|██████▉ | 527/763 [05:57<01:56, 2.02it/s] 69%|██████▉ | 528/763 [05:58<01:56, 2.02it/s] 69%|██████▉ | 529/763 [05:58<01:55, 2.03it/s] 69%|██████▉ | 530/763 [05:59<01:55, 2.02it/s] 70%|██████▉ | 531/763 [05:59<01:54, 2.02it/s] 70%|██████▉ | 532/763 [06:00<01:54, 2.02it/s] 70%|██████▉ | 533/763 [06:00<01:53, 2.02it/s] 70%|██████▉ | 534/763 [06:01<01:53, 2.02it/s] 70%|███████ | 535/763 [06:01<01:52, 2.02it/s] 70%|███████ | 536/763 [06:02<01:52, 2.02it/s] 70%|███████ | 537/763 [06:02<01:51, 2.02it/s] 71%|███████ | 538/763 [06:03<01:51, 2.02it/s] 71%|███████ | 539/763 [06:03<01:50, 2.02it/s] 71%|███████ | 540/763 [06:04<01:50, 2.02it/s] 71%|███████ | 541/763 [06:04<01:49, 2.02it/s] 71%|███████ | 542/763 [06:05<01:49, 2.02it/s] 71%|███████ | 543/763 [06:05<01:48, 2.02it/s] 71%|███████▏ | 544/763 [06:06<01:48, 2.02it/s] 71%|███████▏ | 545/763 [06:06<01:47, 2.02it/s] 72%|███████▏ | 546/763 [06:07<01:47, 2.02it/s] 72%|███████▏ | 547/763 [06:07<01:46, 2.02it/s] 72%|███████▏ | 548/763 [06:08<01:46, 2.02it/s] 72%|███████▏ | 549/763 [06:08<01:45, 2.02it/s] 72%|███████▏ | 550/763 [06:09<01:45, 2.02it/s]{'loss': 2.436, 'grad_norm': 0.36740800738334656, 'learning_rate': 0.00021960255753653008, 'epoch': 0.72} | |
| 72%|███████▏ | 550/763 [06:09<01:45, 2.02it/s] 72%|███████▏ | 551/763 [06:09<01:44, 2.02it/s] 72%|███████▏ | 552/763 [06:10<01:44, 2.02it/s] 72%|███████▏ | 553/763 [06:10<01:43, 2.02it/s] 73%|███████▎ | 554/763 [06:11<01:43, 2.02it/s] 73%|███████▎ | 555/763 [06:11<01:42, 2.02it/s] 73%|███████▎ | 556/763 [06:12<01:42, 2.02it/s] 73%|███████▎ | 557/763 [06:12<01:41, 2.02it/s] 73%|███████▎ | 558/763 [06:13<01:41, 2.02it/s] 73%|███████▎ | 559/763 [06:13<01:40, 2.02it/s] 73%|███████▎ | 560/763 [06:14<01:40, 2.02it/s] 74%|███████▎ | 561/763 [06:14<01:39, 2.02it/s] 74%|███████▎ | 562/763 [06:15<01:39, 2.02it/s] 74%|███████▍ | 563/763 [06:15<01:38, 2.02it/s] 74%|███████▍ | 564/763 [06:16<01:38, 2.02it/s] 74%|███████▍ | 565/763 [06:16<01:37, 2.02it/s] 74%|███████▍ | 566/763 [06:17<01:37, 2.02it/s] 74%|███████▍ | 567/763 [06:17<01:36, 2.02it/s] 74%|███████▍ | 568/763 [06:18<01:36, 2.02it/s] 75%|███████▍ | 569/763 [06:18<01:36, 2.02it/s] 75%|███████▍ | 570/763 [06:19<01:35, 2.02it/s] 75%|███████▍ | 571/763 [06:19<01:34, 2.02it/s] 75%|███████▍ | 572/763 [06:20<01:34, 2.02it/s] 75%|███████▌ | 573/763 [06:20<01:33, 2.02it/s] 75%|███████▌ | 574/763 [06:21<01:33, 2.02it/s] 75%|███████▌ | 575/763 [06:21<01:32, 2.02it/s]{'loss': 2.4095, 'grad_norm': 0.3562281131744385, 'learning_rate': 0.00017414565541703342, 'epoch': 0.75} | |
| 75%|███████▌ | 575/763 [06:21<01:32, 2.02it/s] 75%|███████▌ | 576/763 [06:22<01:32, 2.02it/s] 76%|███████▌ | 577/763 [06:22<01:32, 2.02it/s] 76%|███████▌ | 578/763 [06:23<01:31, 2.02it/s] 76%|███████▌ | 579/763 [06:23<01:31, 2.02it/s] 76%|███████▌ | 580/763 [06:24<01:30, 2.02it/s] 76%|███████▌ | 581/763 [06:24<01:29, 2.02it/s] 76%|███████▋ | 582/763 [06:25<01:29, 2.02it/s] 76%|███████▋ | 583/763 [06:25<01:28, 2.02it/s] 77%|███████▋ | 584/763 [06:26<01:28, 2.02it/s] 77%|███████▋ | 585/763 [06:26<01:28, 2.02it/s] 77%|███████▋ | 586/763 [06:27<01:27, 2.02it/s] 77%|███████▋ | 587/763 [06:27<01:27, 2.02it/s] 77%|███████▋ | 588/763 [06:28<01:26, 2.02it/s] 77%|███████▋ | 589/763 [06:28<01:25, 2.02it/s] 77%|███████▋ | 590/763 [06:29<01:25, 2.02it/s] 77%|███████▋ | 591/763 [06:29<01:24, 2.02it/s] 78%|███████▊ | 592/763 [06:30<01:24, 2.02it/s] 78%|███████▊ | 593/763 [06:30<01:24, 2.02it/s] 78%|███████▊ | 594/763 [06:31<01:23, 2.02it/s] 78%|███████▊ | 595/763 [06:31<01:23, 2.02it/s] 78%|███████▊ | 596/763 [06:32<01:22, 2.02it/s] 78%|███████▊ | 597/763 [06:32<01:22, 2.02it/s] 78%|███████▊ | 598/763 [06:33<01:21, 2.02it/s] 79%|███████▊ | 599/763 [06:33<01:21, 2.02it/s] 79%|███████▊ | 600/763 [06:34<01:20, 2.02it/s] {'loss': 2.3795, 'grad_norm': 0.35245397686958313, 'learning_rate': 0.0001329553403026331, 'epoch': 0.79} | |
| 79%|███████▊ | 600/763 [06:34<01:20, 2.02it/s] 79%|███████▉ | 601/763 [06:34<01:20, 2.02it/s] 79%|███████▉ | 602/763 [06:35<01:19, 2.02it/s] 79%|███████▉ | 603/763 [06:35<01:18, 2.03it/s] 79%|███████▉ | 604/763 [06:36<01:18, 2.02it/s] 79%|███████▉ | 605/763 [06:36<01:17, 2.03it/s] 79%|███████▉ | 606/763 [06:37<01:17, 2.03it/s] 80%|███████▉ | 607/763 [06:37<01:17, 2.03it/s] 80%|███████▉ | 608/763 [06:38<01:16, 2.02it/s] 80%|███████▉ | 609/763 [06:38<01:22, 1.86it/s] 80%|███████▉ | 610/763 [06:39<01:20, 1.91it/s] 80%|████████ | 611/763 [06:39<01:18, 1.94it/s] 80%|████████ | 612/763 [06:40<01:16, 1.97it/s] 80%|████████ | 613/763 [06:40<01:15, 1.98it/s] 80%|████████ | 614/763 [06:41<01:14, 2.00it/s] 81%|████████ | 615/763 [06:41<01:13, 2.00it/s] 81%|████████ | 616/763 [06:42<01:19, 1.85it/s] 81%|████████ | 617/763 [06:42<01:16, 1.90it/s] 81%|████████ | 618/763 [06:43<01:14, 1.94it/s] 81%|████████ | 619/763 [06:43<01:13, 1.96it/s] 81%|████████▏ | 620/763 [06:44<01:12, 1.98it/s] 81%|████████▏ | 621/763 [06:44<01:11, 1.99it/s] 82%|████████▏ | 622/763 [06:45<01:10, 2.00it/s] 82%|████████▏ | 623/763 [06:45<01:09, 2.01it/s] 82%|████████▏ | 624/763 [06:46<01:09, 2.01it/s] 82%|████████▏ | 625/763 [06:46<01:08, 2.01it/s]{'loss': 2.3607, 'grad_norm': 0.3452637791633606, 'learning_rate': 9.657093924581261e-05, 'epoch': 0.82} | |
| 82%|████████▏ | 625/763 [06:46<01:08, 2.01it/s] 82%|████████▏ | 626/763 [06:47<01:07, 2.02it/s] 82%|████████▏ | 627/763 [06:47<01:07, 2.02it/s] 82%|████████▏ | 628/763 [06:48<01:06, 2.02it/s] 82%|████████▏ | 629/763 [06:48<01:06, 2.02it/s] 83%|████████▎ | 630/763 [06:49<01:05, 2.02it/s] 83%|████████▎ | 631/763 [06:49<01:05, 2.02it/s] 83%|████████▎ | 632/763 [06:50<01:04, 2.02it/s] 83%|████████▎ | 633/763 [06:50<01:04, 2.03it/s] 83%|████████▎ | 634/763 [06:51<01:03, 2.02it/s] 83%|████████▎ | 635/763 [06:51<01:03, 2.03it/s] 83%|████████▎ | 636/763 [06:52<01:02, 2.03it/s] 83%|████████▎ | 637/763 [06:52<01:02, 2.02it/s] 84%|████████▎ | 638/763 [06:53<01:01, 2.02it/s] 84%|████████▎ | 639/763 [06:53<01:01, 2.02it/s] 84%|████████▍ | 640/763 [06:54<01:00, 2.02it/s] 84%|████████▍ | 641/763 [06:54<01:00, 2.02it/s] 84%|████████▍ | 642/763 [06:55<00:59, 2.02it/s] 84%|████████▍ | 643/763 [06:55<00:59, 2.02it/s] 84%|████████▍ | 644/763 [06:56<00:58, 2.02it/s] 85%|████████▍ | 645/763 [06:56<00:58, 2.02it/s] 85%|████████▍ | 646/763 [06:57<00:57, 2.02it/s] 85%|████████▍ | 647/763 [06:57<00:57, 2.02it/s] 85%|████████▍ | 648/763 [06:58<00:56, 2.02it/s] 85%|████████▌ | 649/763 [06:58<00:56, 2.02it/s] 85%|████████▌ | 650/763 [06:59<00:55, 2.02it/s]{'loss': 2.3353, 'grad_norm': 0.34281203150749207, 'learning_rate': 6.546885286948184e-05, 'epoch': 0.85} | |
| 85%|████████▌ | 650/763 [06:59<00:55, 2.02it/s] 85%|████████▌ | 651/763 [06:59<00:55, 2.01it/s] 85%|████████▌ | 652/763 [07:00<00:54, 2.02it/s] 86%|████████▌ | 653/763 [07:00<00:54, 2.02it/s] 86%|████████▌ | 654/763 [07:01<00:53, 2.02it/s] 86%|████████▌ | 655/763 [07:01<00:53, 2.02it/s] 86%|████████▌ | 656/763 [07:02<00:52, 2.03it/s] 86%|████████▌ | 657/763 [07:02<00:52, 2.03it/s] 86%|████████▌ | 658/763 [07:03<00:51, 2.03it/s] 86%|████████▋ | 659/763 [07:03<00:51, 2.03it/s] 87%|████████▋ | 660/763 [07:04<00:50, 2.03it/s] 87%|████████▋ | 661/763 [07:04<00:50, 2.03it/s] 87%|████████▋ | 662/763 [07:05<00:49, 2.03it/s] 87%|████████▋ | 663/763 [07:05<00:49, 2.03it/s] 87%|████████▋ | 664/763 [07:05<00:48, 2.03it/s] 87%|████████▋ | 665/763 [07:06<00:48, 2.03it/s] 87%|████████▋ | 666/763 [07:06<00:47, 2.03it/s] 87%|████████▋ | 667/763 [07:07<00:47, 2.03it/s] 88%|████████▊ | 668/763 [07:07<00:46, 2.03it/s] 88%|████████▊ | 669/763 [07:08<00:46, 2.03it/s] 88%|████████▊ | 670/763 [07:08<00:45, 2.03it/s] 88%|████████▊ | 671/763 [07:09<00:45, 2.03it/s] 88%|████████▊ | 672/763 [07:09<00:44, 2.03it/s] 88%|████████▊ | 673/763 [07:10<00:44, 2.03it/s] 88%|████████▊ | 674/763 [07:10<00:43, 2.03it/s] 88%|████████▊ | 675/763 [07:11<00:43, 2.03it/s]{'loss': 2.3277, 'grad_norm': 0.33725234866142273, 'learning_rate': 4.0056317596204094e-05, 'epoch': 0.88} | |
| 88%|████████▊ | 675/763 [07:11<00:43, 2.03it/s] 89%|████████▊ | 676/763 [07:11<00:42, 2.02it/s] 89%|████████▊ | 677/763 [07:12<00:42, 2.02it/s] 89%|████████▉ | 678/763 [07:12<00:41, 2.03it/s] 89%|████████▉ | 679/763 [07:13<00:41, 2.03it/s] 89%|████████▉ | 680/763 [07:13<00:40, 2.03it/s] 89%|████████▉ | 681/763 [07:14<00:40, 2.03it/s] 89%|████████▉ | 682/763 [07:14<00:39, 2.03it/s] 90%|████████▉ | 683/763 [07:15<00:39, 2.03it/s] 90%|████████▉ | 684/763 [07:15<00:38, 2.03it/s] 90%|████████▉ | 685/763 [07:16<00:38, 2.03it/s] 90%|████████▉ | 686/763 [07:16<00:37, 2.03it/s] 90%|█████████ | 687/763 [07:17<00:37, 2.03it/s] 90%|█████████ | 688/763 [07:17<00:36, 2.03it/s] 90%|█████████ | 689/763 [07:18<00:36, 2.03it/s] 90%|█████████ | 690/763 [07:18<00:36, 2.03it/s] 91%|█████████ | 691/763 [07:19<00:35, 2.03it/s] 91%|█████████ | 692/763 [07:19<00:35, 2.03it/s] 91%|█████████ | 693/763 [07:20<00:34, 2.03it/s] 91%|█████████ | 694/763 [07:20<00:34, 2.03it/s] 91%|█████████ | 695/763 [07:21<00:33, 2.03it/s] 91%|█████████ | 696/763 [07:21<00:33, 2.03it/s] 91%|█████████▏| 697/763 [07:22<00:32, 2.03it/s] 91%|█████████▏| 698/763 [07:22<00:32, 2.03it/s] 92%|█████████▏| 699/763 [07:23<00:31, 2.03it/s] 92%|█████████▏| 700/763 [07:23<00:31, 2.03it/s]{'loss': 2.3087, 'grad_norm': 0.33684036135673523, 'learning_rate': 2.0666073481669712e-05, 'epoch': 0.92} | |
| 92%|█████████▏| 700/763 [07:23<00:31, 2.03it/s] 92%|█████████▏| 701/763 [07:24<00:30, 2.02it/s] 92%|█████████▏| 702/763 [07:24<00:30, 2.02it/s] 92%|█████████▏| 703/763 [07:25<00:29, 2.03it/s] 92%|█████████▏| 704/763 [07:25<00:29, 2.03it/s] 92%|█████████▏| 705/763 [07:26<00:28, 2.03it/s] 93%|█████████▎| 706/763 [07:26<00:28, 2.03it/s] 93%|█████████▎| 707/763 [07:27<00:27, 2.03it/s] 93%|█████████▎| 708/763 [07:27<00:27, 2.03it/s] 93%|█████████▎| 709/763 [07:28<00:26, 2.03it/s] 93%|█████████▎| 710/763 [07:28<00:26, 2.03it/s] 93%|█████████▎| 711/763 [07:29<00:25, 2.03it/s] 93%|█████████▎| 712/763 [07:29<00:25, 2.03it/s] 93%|█████████▎| 713/763 [07:30<00:24, 2.03it/s] 94%|█████████▎| 714/763 [07:30<00:24, 2.02it/s] 94%|█████████▎| 715/763 [07:31<00:23, 2.00it/s] 94%|█████████▍| 716/763 [07:31<00:23, 1.99it/s] 94%|█████████▍| 717/763 [07:32<00:23, 1.98it/s] 94%|█████████▍| 718/763 [07:32<00:22, 1.98it/s] 94%|█████████▍| 719/763 [07:33<00:22, 1.97it/s] 94%|█████████▍| 720/763 [07:33<00:21, 1.98it/s] 94%|█████████▍| 721/763 [07:34<00:21, 1.99it/s] 95%|█████████▍| 722/763 [07:34<00:20, 2.00it/s] 95%|█████████▍| 723/763 [07:35<00:19, 2.01it/s] 95%|█████████▍| 724/763 [07:35<00:19, 2.01it/s] 95%|█████████▌| 725/763 [07:36<00:18, 2.02it/s] {'loss': 2.3126, 'grad_norm': 0.33316919207572937, 'learning_rate': 7.552007469355249e-06, 'epoch': 0.95} | |
| 95%|█████████▌| 725/763 [07:36<00:18, 2.02it/s] 95%|█████████▌| 726/763 [07:36<00:18, 2.02it/s] 95%|█████████▌| 727/763 [07:37<00:17, 2.02it/s] 95%|█████████▌| 728/763 [07:37<00:17, 2.02it/s] 96%|█████████▌| 729/763 [07:38<00:16, 2.02it/s] 96%|█████████▌| 730/763 [07:38<00:16, 2.03it/s] 96%|█████████▌| 731/763 [07:39<00:15, 2.03it/s] 96%|█████████▌| 732/763 [07:39<00:15, 2.03it/s] 96%|█████████▌| 733/763 [07:40<00:14, 2.03it/s] 96%|█████████▌| 734/763 [07:40<00:14, 2.03it/s] 96%|█████████▋| 735/763 [07:41<00:13, 2.03it/s] 96%|█████████▋| 736/763 [07:41<00:13, 2.03it/s] 97%|█████████▋| 737/763 [07:42<00:12, 2.03it/s] 97%|█████████▋| 738/763 [07:42<00:12, 2.03it/s] 97%|█████████▋| 739/763 [07:43<00:11, 2.03it/s] 97%|█████████▋| 740/763 [07:43<00:11, 2.03it/s] 97%|█████████▋| 741/763 [07:44<00:10, 2.03it/s] 97%|█████████▋| 742/763 [07:44<00:10, 2.03it/s] 97%|█████████▋| 743/763 [07:45<00:09, 2.02it/s] 98%|█████████▊| 744/763 [07:45<00:09, 2.00it/s] 98%|█████████▊| 745/763 [07:46<00:09, 1.98it/s] 98%|█████████▊| 746/763 [07:46<00:08, 1.98it/s] 98%|█████████▊| 747/763 [07:47<00:08, 1.98it/s] 98%|█████████▊| 748/763 [07:47<00:07, 1.98it/s] 98%|█████████▊| 749/763 [07:48<00:07, 1.99it/s] 98%|█████████▊| 750/763 [07:48<00:06, 2.00it/s]{'loss': 2.2993, 'grad_norm': 0.3305748701095581, 'learning_rate': 8.858291115876327e-07, 'epoch': 0.98} | |
| 98%|█████████▊| 750/763 [07:48<00:06, 2.00it/s] 98%|█████████▊| 751/763 [07:49<00:05, 2.00it/s] 99%|█████████▊| 752/763 [07:49<00:05, 2.01it/s] 99%|█████████▊| 753/763 [07:50<00:04, 2.01it/s] 99%|█████████▉| 754/763 [07:50<00:04, 2.02it/s] 99%|█████████▉| 755/763 [07:51<00:03, 2.02it/s] 99%|█████████▉| 756/763 [07:51<00:03, 2.02it/s] 99%|█████████▉| 757/763 [07:52<00:02, 2.02it/s] 99%|█████████▉| 758/763 [07:52<00:02, 2.02it/s] 99%|█████████▉| 759/763 [07:53<00:01, 2.02it/s] 100%|█████████▉| 760/763 [07:53<00:01, 2.02it/s] 100%|█████████▉| 761/763 [07:54<00:00, 2.02it/s] 100%|█████████▉| 762/763 [07:54<00:00, 2.02it/s] 100%|██████████| 763/763 [07:55<00:00, 2.04it/s] {'train_runtime': 493.6697, 'train_samples_per_second': 1582.627, 'train_steps_per_second': 1.546, 'train_loss': 3.4241124799335925, 'epoch': 1.0} | |
| 100%|██████████| 763/763 [08:13<00:00, 2.04it/s] 100%|██████████| 763/763 [08:13<00:00, 1.55it/s] | |
| Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. | |