slurm submission log: 2024-05-10 08:21:53.371659
created following sbatch script: 

###############################

#!/bin/bash

#SBATCH --account=nlp
#SBATCH --cpus-per-task=16
#SBATCH --gres=gpu:2
#SBATCH --job-name=tthrush-job-2404107
#SBATCH --mem=400G
#SBATCH --nodelist=sphinx1
#SBATCH --open-mode=append
#SBATCH --output=/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq/train_job_output.txt
#SBATCH --partition=sphinx
#SBATCH --time=14-0

# activate your desired anaconda environment
. /nlp/scr/tthrush/miniconda3/etc/profile.d/conda.sh ; conda activate pretraining-coreset-selection

# cd to working directory
cd .

# launch commands
srun --unbuffered run_as_child_processes 'torchrun --master_port 29502 --nproc_per_node=2 train_llm.py --dataset_id /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq --output_dir /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq --output_hub_id pythia-70m_sciq --model_id EleutherAI/pythia-70m --num_train_epochs 1 --learning_rate 1e-3 --warmup_ratio=0.1 --gradient_accumulation_steps 2'

###############################

submission to slurm complete!


###############################
slurm submission output

Submitted batch job 7593609



###############################

slurm submission log: 2024-05-10 08:23:19.470596
created following sbatch script: 

###############################

#!/bin/bash

#SBATCH --account=nlp
#SBATCH --cpus-per-task=16
#SBATCH --gres=gpu:2
#SBATCH --job-name=tthrush-job-3619453
#SBATCH --mem=400G
#SBATCH --nodelist=sphinx2
#SBATCH --open-mode=append
#SBATCH --output=/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq/train_job_output.txt
#SBATCH --partition=sphinx
#SBATCH --time=14-0

# activate your desired anaconda environment
. /nlp/scr/tthrush/miniconda3/etc/profile.d/conda.sh ; conda activate pretraining-coreset-selection

# cd to working directory
cd .

# launch commands
srun --unbuffered run_as_child_processes 'torchrun --master_port 29502 --nproc_per_node=2 train_llm.py --dataset_id /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq --output_dir /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq --output_hub_id pythia-70m_sciq --model_id EleutherAI/pythia-70m --num_train_epochs 1 --learning_rate 1e-3 --warmup_ratio=0.1 --gradient_accumulation_steps 2'

###############################

submission to slurm complete!


###############################
slurm submission output

Submitted batch job 7593622



###############################

###############################
start time: 2024-05-10 15:02:42.471821
machine: sphinx2
conda env: pretraining-coreset-selection
###############################
running following processes

	torchrun --master_port 29502 --nproc_per_node=2 train_llm.py --dataset_id /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq --output_dir /juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq --output_hub_id pythia-70m_sciq --model_id EleutherAI/pythia-70m --num_train_epochs 1 --learning_rate 1e-3 --warmup_ratio=0.1 --gradient_accumulation_steps 2


###############################
command outputs: 


[2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] 
[2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] *****************************************
[2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
[2024-05-10 15:02:45,277] torch.distributed.run: [WARNING] *****************************************
05/10/2024 15:02:52 - INFO - __main__ - Script parameters ScriptArguments(dataset_id='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq', output_dir='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq', output_hub_id='pythia-70m_sciq', hf_hub_token=True, model_id='EleutherAI/pythia-70m', per_device_train_batch_size=256, num_train_epochs=1, learning_rate=0.001, gradient_accumulation_steps=2, from_scratch=True, warmup_ratio=0.1, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, weight_decay=0.01, lr_scheduler_type='cosine', local_rank=0, resume_from_checkpoint=False, deepspeed=None, peft=False)
05/10/2024 15:02:52 - INFO - __main__ - Script parameters ScriptArguments(dataset_id='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/train_data_3/sciq', output_dir='/juice5/scr5/tthrush/pretraining-coreset-selection/llm_pretraining/llms_3/pythia-70m_sciq', output_hub_id='pythia-70m_sciq', hf_hub_token=True, model_id='EleutherAI/pythia-70m', per_device_train_batch_size=256, num_train_epochs=1, learning_rate=0.001, gradient_accumulation_steps=2, from_scratch=True, warmup_ratio=0.1, adam_beta1=0.9, adam_beta2=0.95, adam_epsilon=1e-08, weight_decay=0.01, lr_scheduler_type='cosine', local_rank=0, resume_from_checkpoint=False, deepspeed=None, peft=False)

  0%|          | 0/763 [00:00<?, ?it/s][rank0]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
[rank1]:[W reducer.cpp:1360] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())

  0%|          | 1/763 [00:45<9:35:14, 45.29s/it]
  0%|          | 2/763 [00:58<5:37:14, 26.59s/it]
  0%|          | 3/763 [01:09<4:02:06, 19.11s/it]
  1%|          | 4/763 [01:15<3:00:21, 14.26s/it]
  1%|          | 5/763 [01:20<2:18:15, 10.94s/it]
  1%|          | 6/763 [01:25<1:50:17,  8.74s/it]
  1%|          | 7/763 [01:28<1:27:09,  6.92s/it]
  1%|          | 8/763 [01:31<1:10:29,  5.60s/it]
  1%|          | 9/763 [01:33<56:45,  4.52s/it]  
  1%|▏         | 10/763 [01:35<45:43,  3.64s/it]
  1%|▏         | 11/763 [01:36<37:31,  2.99s/it]
  2%|▏         | 12/763 [01:38<31:52,  2.55s/it]
  2%|▏         | 13/763 [01:39<27:34,  2.21s/it]
  2%|▏         | 14/763 [01:40<23:46,  1.90s/it]
  2%|▏         | 15/763 [01:41<20:46,  1.67s/it]
  2%|▏         | 16/763 [01:42<17:33,  1.41s/it]
  2%|▏         | 17/763 [01:43<14:55,  1.20s/it]
  2%|▏         | 18/763 [01:44<13:45,  1.11s/it]
  2%|▏         | 19/763 [01:45<12:10,  1.02it/s]
  3%|▎         | 20/763 [01:45<11:06,  1.12it/s]
  3%|▎         | 21/763 [01:46<10:04,  1.23it/s]
  3%|▎         | 22/763 [01:46<09:17,  1.33it/s]
  3%|▎         | 23/763 [01:47<08:41,  1.42it/s]
  3%|▎         | 24/763 [01:48<08:17,  1.49it/s]
  3%|▎         | 25/763 [01:48<08:02,  1.53it/s]
                                                {'loss': 9.8343, 'grad_norm': 0.9369164705276489, 'learning_rate': 0.0003246753246753247, 'epoch': 0.03}

  3%|▎         | 25/763 [01:48<08:02,  1.53it/s]
  3%|▎         | 26/763 [01:49<08:00,  1.53it/s]
  4%|▎         | 27/763 [01:49<07:33,  1.62it/s]
  4%|▎         | 28/763 [01:50<07:18,  1.68it/s]
  4%|▍         | 29/763 [01:51<07:21,  1.66it/s]
  4%|▍         | 30/763 [01:51<07:10,  1.70it/s]
  4%|▍         | 31/763 [01:52<06:59,  1.75it/s]
  4%|▍         | 32/763 [01:52<06:53,  1.77it/s]
  4%|▍         | 33/763 [01:53<06:42,  1.81it/s]
  4%|▍         | 34/763 [01:53<06:35,  1.84it/s]
  5%|▍         | 35/763 [01:54<06:28,  1.87it/s]
  5%|▍         | 36/763 [01:54<06:28,  1.87it/s]
  5%|▍         | 37/763 [01:55<06:24,  1.89it/s]
  5%|▍         | 38/763 [01:55<06:21,  1.90it/s]
  5%|▌         | 39/763 [01:56<06:18,  1.92it/s]
  5%|▌         | 40/763 [01:56<06:14,  1.93it/s]
  5%|▌         | 41/763 [01:57<06:11,  1.94it/s]
  6%|▌         | 42/763 [01:57<06:10,  1.95it/s]
  6%|▌         | 43/763 [01:58<06:09,  1.95it/s]
  6%|▌         | 44/763 [01:58<06:05,  1.97it/s]
  6%|▌         | 45/763 [01:59<06:07,  1.95it/s]
  6%|▌         | 46/763 [01:59<06:06,  1.96it/s]
  6%|▌         | 47/763 [02:00<06:03,  1.97it/s]
  6%|▋         | 48/763 [02:00<06:01,  1.98it/s]
  6%|▋         | 49/763 [02:01<06:00,  1.98it/s]
  7%|▋         | 50/763 [02:01<05:58,  1.99it/s]
                                                {'loss': 7.3748, 'grad_norm': 0.3545825183391571, 'learning_rate': 0.0006493506493506494, 'epoch': 0.07}

  7%|▋         | 50/763 [02:01<05:58,  1.99it/s]
  7%|▋         | 51/763 [02:02<05:58,  1.99it/s]
  7%|▋         | 52/763 [02:02<05:57,  1.99it/s]
  7%|▋         | 53/763 [02:03<05:57,  1.99it/s]
  7%|▋         | 54/763 [02:03<05:55,  1.99it/s]
  7%|▋         | 55/763 [02:04<05:54,  2.00it/s]
  7%|▋         | 56/763 [02:04<05:54,  2.00it/s]
  7%|▋         | 57/763 [02:05<05:53,  2.00it/s]
  8%|▊         | 58/763 [02:05<05:53,  1.99it/s]
  8%|▊         | 59/763 [02:06<05:53,  1.99it/s]
  8%|▊         | 60/763 [02:06<05:52,  2.00it/s]
  8%|▊         | 61/763 [02:07<05:52,  1.99it/s]
  8%|▊         | 62/763 [02:07<05:51,  2.00it/s]
  8%|▊         | 63/763 [02:08<05:50,  2.00it/s]
  8%|▊         | 64/763 [02:08<05:48,  2.01it/s]
  9%|▊         | 65/763 [02:09<05:47,  2.01it/s]
  9%|▊         | 66/763 [02:09<05:57,  1.95it/s]
  9%|▉         | 67/763 [02:10<05:53,  1.97it/s]
  9%|▉         | 68/763 [02:10<05:51,  1.98it/s]
  9%|▉         | 69/763 [02:11<05:49,  1.99it/s]
  9%|▉         | 70/763 [02:11<05:47,  1.99it/s]
  9%|▉         | 71/763 [02:12<05:46,  2.00it/s]
  9%|▉         | 72/763 [02:12<05:45,  2.00it/s]
 10%|▉         | 73/763 [02:13<05:43,  2.01it/s]
 10%|▉         | 74/763 [02:13<05:42,  2.01it/s]
 10%|▉         | 75/763 [02:14<05:42,  2.01it/s]
                                                
{'loss': 5.8974, 'grad_norm': 0.5354925394058228, 'learning_rate': 0.000974025974025974, 'epoch': 0.1}

 10%|▉         | 75/763 [02:14<05:42,  2.01it/s]
 10%|▉         | 76/763 [02:14<05:41,  2.01it/s]
 10%|█         | 77/763 [02:15<05:41,  2.01it/s]
 10%|█         | 78/763 [02:15<05:40,  2.01it/s]
 10%|█         | 79/763 [02:16<05:39,  2.01it/s]
 10%|█         | 80/763 [02:16<05:39,  2.01it/s]
 11%|█         | 81/763 [02:17<05:38,  2.01it/s]
 11%|█         | 82/763 [02:17<05:38,  2.01it/s]
 11%|█         | 83/763 [02:18<05:37,  2.01it/s]
 11%|█         | 84/763 [02:18<05:37,  2.01it/s]
 11%|█         | 85/763 [02:19<05:37,  2.01it/s]
 11%|█▏        | 86/763 [02:19<05:36,  2.01it/s]
 11%|█▏        | 87/763 [02:20<05:34,  2.02it/s]
 12%|█▏        | 88/763 [02:20<05:34,  2.02it/s]
 12%|█▏        | 89/763 [02:21<05:34,  2.02it/s]
 12%|█▏        | 90/763 [02:21<05:33,  2.02it/s]
 12%|█▏        | 91/763 [02:22<05:32,  2.02it/s]
 12%|█▏        | 92/763 [02:22<05:31,  2.02it/s]
 12%|█▏        | 93/763 [02:23<05:32,  2.02it/s]
 12%|█▏        | 94/763 [02:23<05:31,  2.02it/s]
 12%|█▏        | 95/763 [02:24<05:30,  2.02it/s]
 13%|█▎        | 96/763 [02:24<05:30,  2.02it/s]
 13%|█▎        | 97/763 [02:25<05:30,  2.01it/s]
 13%|█▎        | 98/763 [02:25<05:29,  2.02it/s]
 13%|█▎        | 99/763 [02:26<05:29,  2.02it/s]
 13%|█▎        | 100/763 [02:26<05:32,  2.00it/s]{'loss': 5.0788, 'grad_norm': 0.4961775541305542, 'learning_rate': 0.0009972289418801728, 'epoch': 0.13}

                                                 

 13%|█▎        | 100/763 [02:26<05:32,  2.00it/s]
 13%|█▎        | 101/763 [02:27<05:31,  2.00it/s]
 13%|█▎        | 102/763 [02:27<05:30,  2.00it/s]
 13%|█▎        | 103/763 [02:28<05:28,  2.01it/s]
 14%|█▎        | 104/763 [02:28<05:27,  2.01it/s]
 14%|█▍        | 105/763 [02:29<05:27,  2.01it/s]
 14%|█▍        | 106/763 [02:29<05:26,  2.01it/s]
 14%|█▍        | 107/763 [02:30<05:25,  2.01it/s]
 14%|█▍        | 108/763 [02:30<05:24,  2.02it/s]
 14%|█▍        | 109/763 [02:31<05:23,  2.02it/s]
 14%|█▍        | 110/763 [02:31<05:23,  2.02it/s]
 15%|█▍        | 111/763 [02:32<05:23,  2.02it/s]
 15%|█▍        | 112/763 [02:32<05:22,  2.02it/s]
 15%|█▍        | 113/763 [02:33<05:21,  2.02it/s]
 15%|█▍        | 114/763 [02:33<05:21,  2.02it/s]
 15%|█▌        | 115/763 [02:34<05:20,  2.02it/s]
 15%|█▌        | 116/763 [02:34<05:19,  2.02it/s]
 15%|█▌        | 117/763 [02:35<05:19,  2.02it/s]
 15%|█▌        | 118/763 [02:35<05:18,  2.02it/s]
 16%|█▌        | 119/763 [02:36<05:18,  2.02it/s]
 16%|█▌        | 120/763 [02:36<05:17,  2.02it/s]
 16%|█▌        | 121/763 [02:37<05:17,  2.02it/s]
 16%|█▌        | 122/763 [02:37<05:17,  2.02it/s]
 16%|█▌        | 123/763 [02:38<05:16,  2.02it/s]
 16%|█▋        | 124/763 [02:38<05:15,  2.02it/s]
 16%|█▋        | 125/763 [02:39<05:16,  2.02it/s]
                                                 {'loss': 4.6084, 'grad_norm': 0.2889716327190399, 'learning_rate': 0.0009879683689693263, 'epoch': 0.16}

 16%|█▋        | 125/763 [02:39<05:16,  2.02it/s]
 17%|█▋        | 126/763 [02:39<05:15,  2.02it/s]
 17%|█▋        | 127/763 [02:40<05:15,  2.01it/s]
 17%|█▋        | 128/763 [02:40<05:15,  2.02it/s]
 17%|█▋        | 129/763 [02:41<05:14,  2.02it/s]
 17%|█▋        | 130/763 [02:41<05:13,  2.02it/s]
 17%|█▋        | 131/763 [02:42<05:12,  2.02it/s]
 17%|█▋        | 132/763 [02:42<05:12,  2.02it/s]
 17%|█▋        | 133/763 [02:43<05:12,  2.02it/s]
 18%|█▊        | 134/763 [02:43<05:11,  2.02it/s]
 18%|█▊        | 135/763 [02:44<05:11,  2.02it/s]
 18%|█▊        | 136/763 [02:44<05:10,  2.02it/s]
 18%|█▊        | 137/763 [02:45<05:09,  2.02it/s]
 18%|█▊        | 138/763 [02:45<05:09,  2.02it/s]
 18%|█▊        | 139/763 [02:46<05:08,  2.02it/s]
 18%|█▊        | 140/763 [02:46<05:08,  2.02it/s]
 18%|█▊        | 141/763 [02:47<05:07,  2.02it/s]
 19%|█▊        | 142/763 [02:47<05:07,  2.02it/s]
 19%|█▊        | 143/763 [02:48<05:06,  2.02it/s]
 19%|█▉        | 144/763 [02:48<05:05,  2.02it/s]
 19%|█▉        | 145/763 [02:49<05:05,  2.02it/s]
 19%|█▉        | 146/763 [02:49<05:04,  2.02it/s]
 19%|█▉        | 147/763 [02:50<05:04,  2.02it/s]
 19%|█▉        | 148/763 [02:50<05:04,  2.02it/s]
 20%|█▉        | 149/763 [02:51<05:03,  2.02it/s]
 20%|█▉        | 150/763 [02:51<05:03,  2.02it/s]{'loss': 4.2413, 'grad_norm': 0.44144755601882935, 'learning_rate': 0.0009723185625357323, 'epoch': 0.2}

                                                 

 20%|█▉        | 150/763 [02:51<05:03,  2.02it/s]
 20%|█▉        | 151/763 [02:52<05:03,  2.02it/s]
 20%|█▉        | 152/763 [02:52<05:02,  2.02it/s]
 20%|██        | 153/763 [02:53<05:01,  2.02it/s]
 20%|██        | 154/763 [02:53<05:01,  2.02it/s]
 20%|██        | 155/763 [02:54<05:00,  2.02it/s]
 20%|██        | 156/763 [02:54<05:00,  2.02it/s]
 21%|██        | 157/763 [02:55<04:59,  2.02it/s]
 21%|██        | 158/763 [02:55<04:59,  2.02it/s]
 21%|██        | 159/763 [02:56<04:58,  2.02it/s]
 21%|██        | 160/763 [02:56<04:58,  2.02it/s]
 21%|██        | 161/763 [02:57<04:57,  2.02it/s]
 21%|██        | 162/763 [02:57<04:57,  2.02it/s]
 21%|██▏       | 163/763 [02:58<04:57,  2.02it/s]
 21%|██▏       | 164/763 [02:58<04:56,  2.02it/s]
 22%|██▏       | 165/763 [02:59<04:55,  2.02it/s]
 22%|██▏       | 166/763 [02:59<04:55,  2.02it/s]
 22%|██▏       | 167/763 [03:00<04:54,  2.02it/s]
 22%|██▏       | 168/763 [03:00<04:54,  2.02it/s]
 22%|██▏       | 169/763 [03:01<04:53,  2.02it/s]
 22%|██▏       | 170/763 [03:01<04:52,  2.02it/s]
 22%|██▏       | 171/763 [03:02<04:52,  2.02it/s]
 23%|██▎       | 172/763 [03:02<04:51,  2.02it/s]
 23%|██▎       | 173/763 [03:03<04:51,  2.02it/s]
 23%|██▎       | 174/763 [03:03<04:51,  2.02it/s]
 23%|██▎       | 175/763 [03:04<04:50,  2.02it/s]{'loss': 3.9435, 'grad_norm': 0.39299631118774414, 'learning_rate': 0.0009504844339512095, 'epoch': 0.23}

                                                 

 23%|██▎       | 175/763 [03:04<04:50,  2.02it/s]
 23%|██▎       | 176/763 [03:04<04:50,  2.02it/s]
 23%|██▎       | 177/763 [03:04<04:50,  2.02it/s]
 23%|██▎       | 178/763 [03:05<04:49,  2.02it/s]
 23%|██▎       | 179/763 [03:05<04:48,  2.02it/s]
 24%|██▎       | 180/763 [03:06<04:48,  2.02it/s]
 24%|██▎       | 181/763 [03:06<04:47,  2.02it/s]
 24%|██▍       | 182/763 [03:07<04:47,  2.02it/s]
 24%|██▍       | 183/763 [03:07<04:46,  2.02it/s]
 24%|██▍       | 184/763 [03:08<04:46,  2.02it/s]
 24%|██▍       | 185/763 [03:08<04:46,  2.02it/s]
 24%|██▍       | 186/763 [03:09<04:45,  2.02it/s]
 25%|██▍       | 187/763 [03:09<04:45,  2.02it/s]
 25%|██▍       | 188/763 [03:10<04:44,  2.02it/s]
 25%|██▍       | 189/763 [03:10<04:43,  2.02it/s]
 25%|██▍       | 190/763 [03:11<04:43,  2.02it/s]
 25%|██▌       | 191/763 [03:11<04:42,  2.02it/s]
 25%|██▌       | 192/763 [03:12<04:42,  2.02it/s]
 25%|██▌       | 193/763 [03:12<04:41,  2.02it/s]
 25%|██▌       | 194/763 [03:13<04:41,  2.02it/s]
 26%|██▌       | 195/763 [03:13<04:41,  2.02it/s]
 26%|██▌       | 196/763 [03:14<04:40,  2.02it/s]
 26%|██▌       | 197/763 [03:14<04:39,  2.02it/s]
 26%|██▌       | 198/763 [03:15<04:39,  2.02it/s]
 26%|██▌       | 199/763 [03:15<04:38,  2.02it/s]
 26%|██▌       | 200/763 [03:16<04:38,  2.02it/s]{'loss': 3.6901, 'grad_norm': 0.34031203389167786, 'learning_rate': 0.0009227518692591244, 'epoch': 0.26}

                                                 

 26%|██▌       | 200/763 [03:16<04:38,  2.02it/s]
 26%|██▋       | 201/763 [03:16<04:38,  2.02it/s]
 26%|██▋       | 202/763 [03:17<04:37,  2.02it/s]
 27%|██▋       | 203/763 [03:17<04:36,  2.02it/s]
 27%|██▋       | 204/763 [03:18<04:36,  2.02it/s]
 27%|██▋       | 205/763 [03:18<04:35,  2.03it/s]
 27%|██▋       | 206/763 [03:19<04:35,  2.03it/s]
 27%|██▋       | 207/763 [03:19<04:34,  2.02it/s]
 27%|██▋       | 208/763 [03:20<04:34,  2.02it/s]
 27%|██▋       | 209/763 [03:20<04:33,  2.02it/s]
 28%|██▊       | 210/763 [03:21<04:33,  2.02it/s]
 28%|██▊       | 211/763 [03:21<04:32,  2.02it/s]
 28%|██▊       | 212/763 [03:22<04:32,  2.02it/s]
 28%|██▊       | 213/763 [03:22<04:31,  2.02it/s]
 28%|██▊       | 214/763 [03:23<04:31,  2.02it/s]
 28%|██▊       | 215/763 [03:23<04:30,  2.03it/s]
 28%|██▊       | 216/763 [03:24<04:30,  2.02it/s]
 28%|██▊       | 217/763 [03:24<04:29,  2.03it/s]
 29%|██▊       | 218/763 [03:25<04:29,  2.03it/s]
 29%|██▊       | 219/763 [03:25<04:28,  2.03it/s]
 29%|██▉       | 220/763 [03:26<04:28,  2.03it/s]
 29%|██▉       | 221/763 [03:26<04:27,  2.02it/s]
 29%|██▉       | 222/763 [03:27<04:26,  2.03it/s]
 29%|██▉       | 223/763 [03:27<04:26,  2.03it/s]
 29%|██▉       | 224/763 [03:28<04:26,  2.02it/s]
 29%|██▉       | 225/763 [03:28<04:25,  2.02it/s]{'loss': 3.4966, 'grad_norm': 0.37306490540504456, 'learning_rate': 0.0008894839859139472, 'epoch': 0.29}

                                                 

 29%|██▉       | 225/763 [03:28<04:25,  2.02it/s]
 30%|██▉       | 226/763 [03:29<04:25,  2.02it/s]
 30%|██▉       | 227/763 [03:29<04:24,  2.02it/s]
 30%|██▉       | 228/763 [03:30<04:24,  2.02it/s]
 30%|███       | 229/763 [03:30<04:23,  2.02it/s]
 30%|███       | 230/763 [03:31<04:23,  2.02it/s]
 30%|███       | 231/763 [03:31<04:22,  2.03it/s]
 30%|███       | 232/763 [03:32<04:22,  2.02it/s]
 31%|███       | 233/763 [03:32<04:22,  2.02it/s]
 31%|███       | 234/763 [03:33<04:21,  2.02it/s]
 31%|███       | 235/763 [03:33<04:20,  2.02it/s]
 31%|███       | 236/763 [03:34<04:20,  2.02it/s]
 31%|███       | 237/763 [03:34<04:20,  2.02it/s]
 31%|███       | 238/763 [03:35<04:19,  2.02it/s]
 31%|███▏      | 239/763 [03:35<04:18,  2.02it/s]
 31%|███▏      | 240/763 [03:36<04:18,  2.02it/s]
 32%|███▏      | 241/763 [03:36<04:17,  2.02it/s]
 32%|███▏      | 242/763 [03:37<04:17,  2.02it/s]
 32%|███▏      | 243/763 [03:37<04:17,  2.02it/s]
 32%|███▏      | 244/763 [03:38<04:16,  2.02it/s]
 32%|███▏      | 245/763 [03:38<04:16,  2.02it/s]
 32%|███▏      | 246/763 [03:39<04:15,  2.02it/s]
 32%|███▏      | 247/763 [03:39<04:15,  2.02it/s]
 33%|███▎      | 248/763 [03:40<04:14,  2.02it/s]
 33%|███▎      | 249/763 [03:40<04:14,  2.02it/s]
 33%|███▎      | 250/763 [03:41<04:13,  2.03it/s]{'loss': 3.3449, 'grad_norm': 0.4239332973957062, 'learning_rate': 0.0008511163782882168, 'epoch': 0.33}

                                                 

 33%|███▎      | 250/763 [03:41<04:13,  2.03it/s]
 33%|███▎      | 251/763 [03:41<04:13,  2.02it/s]
 33%|███▎      | 252/763 [03:42<04:12,  2.02it/s]
 33%|███▎      | 253/763 [03:42<04:12,  2.02it/s]
 33%|███▎      | 254/763 [03:43<04:11,  2.02it/s]
 33%|███▎      | 255/763 [03:43<04:11,  2.02it/s]
 34%|███▎      | 256/763 [03:44<04:10,  2.02it/s]
 34%|███▎      | 257/763 [03:44<04:09,  2.02it/s]
 34%|███▍      | 258/763 [03:45<04:09,  2.03it/s]
 34%|███▍      | 259/763 [03:45<04:08,  2.03it/s]
 34%|███▍      | 260/763 [03:46<04:08,  2.02it/s]
 34%|███▍      | 261/763 [03:46<04:08,  2.02it/s]
 34%|███▍      | 262/763 [03:47<04:07,  2.02it/s]
 34%|███▍      | 263/763 [03:47<04:07,  2.02it/s]
 35%|███▍      | 264/763 [03:47<04:06,  2.02it/s]
 35%|███▍      | 265/763 [03:48<04:06,  2.02it/s]
 35%|███▍      | 266/763 [03:48<04:05,  2.02it/s]
 35%|███▍      | 267/763 [03:49<04:05,  2.02it/s]
 35%|███▌      | 268/763 [03:49<04:04,  2.02it/s]
 35%|███▌      | 269/763 [03:50<04:04,  2.02it/s]
 35%|███▌      | 270/763 [03:50<04:03,  2.02it/s]
 36%|███▌      | 271/763 [03:51<04:03,  2.02it/s]
 36%|███▌      | 272/763 [03:51<04:02,  2.02it/s]
 36%|███▌      | 273/763 [03:52<04:02,  2.02it/s]
 36%|███▌      | 274/763 [03:52<04:01,  2.02it/s]
 36%|███▌      | 275/763 [03:53<04:01,  2.02it/s]
                                                 
{'loss': 3.1922, 'grad_norm': 0.4128451645374298, 'learning_rate': 0.0008081514142000517, 'epoch': 0.36}

 36%|███▌      | 275/763 [03:53<04:01,  2.02it/s]
 36%|███▌      | 276/763 [03:53<04:00,  2.02it/s]
 36%|███▋      | 277/763 [03:54<04:00,  2.02it/s]
 36%|███▋      | 278/763 [03:54<03:59,  2.02it/s]
 37%|███▋      | 279/763 [03:55<03:59,  2.02it/s]
 37%|███▋      | 280/763 [03:55<03:58,  2.02it/s]
 37%|███▋      | 281/763 [03:56<03:58,  2.02it/s]
 37%|███▋      | 282/763 [03:56<03:57,  2.02it/s]
 37%|███▋      | 283/763 [03:57<03:57,  2.02it/s]
 37%|███▋      | 284/763 [03:57<03:56,  2.02it/s]
 37%|███▋      | 285/763 [03:58<03:56,  2.02it/s]
 37%|███▋      | 286/763 [03:58<03:55,  2.02it/s]
 38%|███▊      | 287/763 [03:59<03:55,  2.02it/s]
 38%|███▊      | 288/763 [03:59<03:55,  2.02it/s]
 38%|███▊      | 289/763 [04:00<03:54,  2.02it/s]
 38%|███▊      | 290/763 [04:00<03:54,  2.02it/s]
 38%|███▊      | 291/763 [04:01<03:53,  2.02it/s]
 38%|███▊      | 292/763 [04:01<03:53,  2.02it/s]
 38%|███▊      | 293/763 [04:02<03:52,  2.02it/s]
 39%|███▊      | 294/763 [04:02<03:51,  2.02it/s]
 39%|███▊      | 295/763 [04:03<03:51,  2.02it/s]
 39%|███▉      | 296/763 [04:03<03:50,  2.02it/s]
 39%|███▉      | 297/763 [04:04<03:50,  2.02it/s]
 39%|███▉      | 298/763 [04:04<03:49,  2.02it/s]
 39%|███▉      | 299/763 [04:05<03:49,  2.02it/s]
 39%|███▉      | 300/763 [04:05<03:48,  2.02it/s]
                                                 {'loss': 3.0883, 'grad_norm': 0.3450576961040497, 'learning_rate': 0.0007611516571398591, 'epoch': 0.39}


 39%|███▉      | 300/763 [04:05<03:48,  2.02it/s]
 39%|███▉      | 301/763 [04:06<03:48,  2.02it/s]
 40%|███▉      | 302/763 [04:06<03:48,  2.02it/s]
 40%|███▉      | 303/763 [04:07<03:47,  2.02it/s]
 40%|███▉      | 304/763 [04:07<03:47,  2.02it/s]
 40%|███▉      | 305/763 [04:08<03:46,  2.02it/s]
 40%|████      | 306/763 [04:08<03:45,  2.02it/s]
 40%|████      | 307/763 [04:09<03:45,  2.02it/s]
 40%|████      | 308/763 [04:09<03:44,  2.02it/s]
 40%|████      | 309/763 [04:10<03:44,  2.02it/s]
 41%|████      | 310/763 [04:10<03:43,  2.02it/s]
 41%|████      | 311/763 [04:11<03:43,  2.02it/s]
 41%|████      | 312/763 [04:11<03:42,  2.02it/s]
 41%|████      | 313/763 [04:12<03:42,  2.02it/s]
 41%|████      | 314/763 [04:12<03:41,  2.02it/s]
 41%|████▏     | 315/763 [04:13<03:41,  2.02it/s]
 41%|████▏     | 316/763 [04:13<03:40,  2.02it/s]
 42%|████▏     | 317/763 [04:14<03:40,  2.02it/s]
 42%|████▏     | 318/763 [04:14<03:39,  2.02it/s]
 42%|████▏     | 319/763 [04:15<03:39,  2.02it/s]
 42%|████▏     | 320/763 [04:15<03:38,  2.02it/s]
 42%|████▏     | 321/763 [04:16<03:38,  2.02it/s]
 42%|████▏     | 322/763 [04:16<03:38,  2.02it/s]
 42%|████▏     | 323/763 [04:17<03:37,  2.02it/s]
 42%|████▏     | 324/763 [04:17<03:36,  2.02it/s]
 43%|████▎     | 325/763 [04:18<03:36,  2.03it/s]
                                                 
{'loss': 2.9938, 'grad_norm': 0.4092760384082794, 'learning_rate': 0.000710732500322581, 'epoch': 0.43}

 43%|████▎     | 325/763 [04:18<03:36,  2.03it/s]
 43%|████▎     | 326/763 [04:18<03:36,  2.02it/s]
 43%|████▎     | 327/763 [04:19<03:35,  2.02it/s]
 43%|████▎     | 328/763 [04:19<03:35,  2.02it/s]
 43%|████▎     | 329/763 [04:20<03:34,  2.02it/s]
 43%|████▎     | 330/763 [04:20<03:33,  2.02it/s]
 43%|████▎     | 331/763 [04:21<03:33,  2.02it/s]
 44%|████▎     | 332/763 [04:21<03:33,  2.02it/s]
 44%|████▎     | 333/763 [04:22<03:32,  2.02it/s]
 44%|████▍     | 334/763 [04:22<03:31,  2.02it/s]
 44%|████▍     | 335/763 [04:23<03:31,  2.02it/s]
 44%|████▍     | 336/763 [04:23<03:31,  2.02it/s]
 44%|████▍     | 337/763 [04:24<03:30,  2.02it/s]
 44%|████▍     | 338/763 [04:24<03:30,  2.02it/s]
 44%|████▍     | 339/763 [04:25<03:29,  2.02it/s]
 45%|████▍     | 340/763 [04:25<03:29,  2.02it/s]
 45%|████▍     | 341/763 [04:26<03:28,  2.02it/s]
 45%|████▍     | 342/763 [04:26<03:28,  2.02it/s]
 45%|████▍     | 343/763 [04:27<03:27,  2.02it/s]
 45%|████▌     | 344/763 [04:27<03:26,  2.02it/s]
 45%|████▌     | 345/763 [04:28<03:26,  2.03it/s]
 45%|████▌     | 346/763 [04:28<03:26,  2.02it/s]
 45%|████▌     | 347/763 [04:29<03:25,  2.02it/s]
 46%|████▌     | 348/763 [04:29<03:25,  2.02it/s]
 46%|████▌     | 349/763 [04:30<03:24,  2.02it/s]
 46%|████▌     | 350/763 [04:30<03:24,  2.02it/s]{'loss': 2.9083, 'grad_norm': 0.3510383367538452, 'learning_rate': 0.0006575541090118104, 'epoch': 0.46}

                                                 

 46%|████▌     | 350/763 [04:30<03:24,  2.02it/s]
 46%|████▌     | 351/763 [04:31<03:24,  2.02it/s]
 46%|████▌     | 352/763 [04:31<03:23,  2.02it/s]
 46%|████▋     | 353/763 [04:32<03:22,  2.02it/s]
 46%|████▋     | 354/763 [04:32<03:22,  2.02it/s]
 47%|████▋     | 355/763 [04:32<03:21,  2.02it/s]
 47%|████▋     | 356/763 [04:33<03:21,  2.02it/s]
 47%|████▋     | 357/763 [04:33<03:20,  2.02it/s]
 47%|████▋     | 358/763 [04:34<03:20,  2.02it/s]
 47%|████▋     | 359/763 [04:34<03:19,  2.02it/s]
 47%|████▋     | 360/763 [04:35<03:19,  2.02it/s]
 47%|████▋     | 361/763 [04:35<03:18,  2.02it/s]
 47%|████▋     | 362/763 [04:36<03:18,  2.02it/s]
 48%|████▊     | 363/763 [04:36<03:17,  2.02it/s]
 48%|████▊     | 364/763 [04:37<03:17,  2.02it/s]
 48%|████▊     | 365/763 [04:37<03:16,  2.02it/s]
 48%|████▊     | 366/763 [04:38<03:16,  2.02it/s]
 48%|████▊     | 367/763 [04:38<03:15,  2.02it/s]
 48%|████▊     | 368/763 [04:39<03:15,  2.02it/s]
 48%|████▊     | 369/763 [04:39<03:14,  2.02it/s]
 48%|████▊     | 370/763 [04:40<03:14,  2.02it/s]
 49%|████▊     | 371/763 [04:40<03:13,  2.02it/s]
 49%|████▉     | 372/763 [04:41<03:13,  2.02it/s]
 49%|████▉     | 373/763 [04:41<03:12,  2.03it/s]
 49%|████▉     | 374/763 [04:42<03:12,  2.03it/s]
 49%|████▉     | 375/763 [04:42<03:11,  2.02it/s]{'loss': 2.8176, 'grad_norm': 0.3813249170780182, 'learning_rate': 0.0006023127766192824, 'epoch': 0.49}

                                                 

 49%|████▉     | 375/763 [04:42<03:11,  2.02it/s]
 49%|████▉     | 376/763 [04:43<03:11,  2.02it/s]
 49%|████▉     | 377/763 [04:43<03:10,  2.02it/s]
 50%|████▉     | 378/763 [04:44<03:10,  2.02it/s]
 50%|████▉     | 379/763 [04:44<03:09,  2.02it/s]
 50%|████▉     | 380/763 [04:45<03:09,  2.03it/s]
 50%|████▉     | 381/763 [04:45<03:08,  2.02it/s]
 50%|█████     | 382/763 [04:46<03:08,  2.02it/s]
 50%|█████     | 383/763 [04:46<03:07,  2.02it/s]
 50%|█████     | 384/763 [04:47<03:07,  2.02it/s]
 50%|█████     | 385/763 [04:47<03:07,  2.02it/s]
 51%|█████     | 386/763 [04:48<03:06,  2.02it/s]
 51%|█████     | 387/763 [04:48<03:05,  2.02it/s]
 51%|█████     | 388/763 [04:49<03:05,  2.02it/s]
 51%|█████     | 389/763 [04:49<03:04,  2.03it/s]
 51%|█████     | 390/763 [04:50<03:04,  2.02it/s]
 51%|█████     | 391/763 [04:50<03:03,  2.03it/s]
 51%|█████▏    | 392/763 [04:51<03:03,  2.02it/s]
 52%|█████▏    | 393/763 [04:51<03:02,  2.02it/s]
 52%|█████▏    | 394/763 [04:52<03:02,  2.02it/s]
 52%|█████▏    | 395/763 [04:52<03:02,  2.02it/s]
 52%|█████▏    | 396/763 [04:53<03:01,  2.02it/s]
 52%|█████▏    | 397/763 [04:53<03:00,  2.02it/s]
 52%|█████▏    | 398/763 [04:54<03:00,  2.02it/s]
 52%|█████▏    | 399/763 [04:54<02:59,  2.02it/s]
 52%|█████▏    | 400/763 [04:55<02:59,  2.02it/s]{'loss': 2.7491, 'grad_norm': 0.34457528591156006, 'learning_rate': 0.0005457318077590012, 'epoch': 0.52}

                                                 

 52%|█████▏    | 400/763 [04:55<02:59,  2.02it/s]
 53%|█████▎    | 401/763 [04:55<02:59,  2.02it/s]
 53%|█████▎    | 402/763 [04:56<02:58,  2.02it/s]
 53%|█████▎    | 403/763 [04:56<02:58,  2.02it/s]
 53%|█████▎    | 404/763 [04:57<02:57,  2.02it/s]
 53%|█████▎    | 405/763 [04:57<02:57,  2.02it/s]
 53%|█████▎    | 406/763 [04:58<02:56,  2.02it/s]
 53%|█████▎    | 407/763 [04:58<02:56,  2.02it/s]
 53%|█████▎    | 408/763 [04:59<02:55,  2.02it/s]
 54%|█████▎    | 409/763 [04:59<02:55,  2.02it/s]
 54%|█████▎    | 410/763 [05:00<02:54,  2.02it/s]
 54%|█████▍    | 411/763 [05:00<02:54,  2.02it/s]
 54%|█████▍    | 412/763 [05:01<02:53,  2.02it/s]
 54%|█████▍    | 413/763 [05:01<02:53,  2.02it/s]
 54%|█████▍    | 414/763 [05:02<02:52,  2.02it/s]
 54%|█████▍    | 415/763 [05:02<02:51,  2.02it/s]
 55%|█████▍    | 416/763 [05:03<02:51,  2.02it/s]
 55%|█████▍    | 417/763 [05:03<02:50,  2.03it/s]
 55%|█████▍    | 418/763 [05:04<02:50,  2.03it/s]
 55%|█████▍    | 419/763 [05:04<02:49,  2.03it/s]
 55%|█████▌    | 420/763 [05:05<02:49,  2.03it/s]
 55%|█████▌    | 421/763 [05:05<02:48,  2.03it/s]
 55%|█████▌    | 422/763 [05:06<02:48,  2.03it/s]
 55%|█████▌    | 423/763 [05:06<02:47,  2.02it/s]
 56%|█████▌    | 424/763 [05:07<02:47,  2.03it/s]
 56%|█████▌    | 425/763 [05:07<02:47,  2.02it/s]{'loss': 2.6858, 'grad_norm': 0.365237295627594, 'learning_rate': 0.0004885520476290998, 'epoch': 0.56}

                                                 

 56%|█████▌    | 425/763 [05:07<02:47,  2.02it/s]
 56%|█████▌    | 426/763 [05:08<02:46,  2.02it/s]
 56%|█████▌    | 427/763 [05:08<02:46,  2.02it/s]
 56%|█████▌    | 428/763 [05:09<02:45,  2.02it/s]
 56%|█████▌    | 429/763 [05:09<02:45,  2.02it/s]
 56%|█████▋    | 430/763 [05:10<02:44,  2.02it/s]
 56%|█████▋    | 431/763 [05:10<02:43,  2.03it/s]
 57%|█████▋    | 432/763 [05:11<02:43,  2.03it/s]
 57%|█████▋    | 433/763 [05:11<02:42,  2.02it/s]
 57%|█████▋    | 434/763 [05:12<02:42,  2.02it/s]
 57%|█████▋    | 435/763 [05:12<02:42,  2.02it/s]
 57%|█████▋    | 436/763 [05:13<02:41,  2.02it/s]
 57%|█████▋    | 437/763 [05:13<02:40,  2.03it/s]
 57%|█████▋    | 438/763 [05:14<02:40,  2.02it/s]
 58%|█████▊    | 439/763 [05:14<02:40,  2.02it/s]
 58%|█████▊    | 440/763 [05:15<02:39,  2.02it/s]
 58%|█████▊    | 441/763 [05:15<02:38,  2.03it/s]
 58%|█████▊    | 442/763 [05:15<02:38,  2.02it/s]
 58%|█████▊    | 443/763 [05:16<02:37,  2.03it/s]
 58%|█████▊    | 444/763 [05:16<02:37,  2.02it/s]
 58%|█████▊    | 445/763 [05:17<02:36,  2.03it/s]
 58%|█████▊    | 446/763 [05:17<02:36,  2.03it/s]
 59%|█████▊    | 447/763 [05:18<02:36,  2.03it/s]
 59%|█████▊    | 448/763 [05:18<02:35,  2.03it/s]
 59%|█████▉    | 449/763 [05:19<02:34,  2.03it/s]
 59%|█████▉    | 450/763 [05:19<02:34,  2.03it/s]{'loss': 2.6339, 'grad_norm': 0.3599332869052887, 'learning_rate': 0.00043152218172535383, 'epoch': 0.59}

                                                 

 59%|█████▉    | 450/763 [05:19<02:34,  2.03it/s]
 59%|█████▉    | 451/763 [05:20<02:34,  2.02it/s]
 59%|█████▉    | 452/763 [05:20<02:33,  2.02it/s]
 59%|█████▉    | 453/763 [05:21<02:33,  2.02it/s]
 60%|█████▉    | 454/763 [05:21<02:32,  2.02it/s]
 60%|█████▉    | 455/763 [05:22<02:32,  2.02it/s]
 60%|█████▉    | 456/763 [05:22<02:31,  2.02it/s]
 60%|█████▉    | 457/763 [05:23<02:31,  2.02it/s]
 60%|██████    | 458/763 [05:23<02:30,  2.03it/s]
 60%|██████    | 459/763 [05:24<02:30,  2.02it/s]
 60%|██████    | 460/763 [05:24<02:29,  2.02it/s]
 60%|██████    | 461/763 [05:25<02:29,  2.02it/s]
 61%|██████    | 462/763 [05:25<02:28,  2.02it/s]
 61%|██████    | 463/763 [05:26<02:28,  2.02it/s]
 61%|██████    | 464/763 [05:26<02:27,  2.02it/s]
 61%|██████    | 465/763 [05:27<02:27,  2.02it/s]
 61%|██████    | 466/763 [05:27<02:26,  2.02it/s]
 61%|██████    | 467/763 [05:28<02:26,  2.02it/s]
 61%|██████▏   | 468/763 [05:28<02:25,  2.02it/s]
 61%|██████▏   | 469/763 [05:29<02:25,  2.03it/s]
 62%|██████▏   | 470/763 [05:29<02:24,  2.02it/s]
 62%|██████▏   | 471/763 [05:30<02:24,  2.02it/s]
 62%|██████▏   | 472/763 [05:30<02:23,  2.02it/s]
 62%|██████▏   | 473/763 [05:31<02:23,  2.02it/s]
 62%|██████▏   | 474/763 [05:31<02:22,  2.02it/s]
 62%|██████▏   | 475/763 [05:32<02:22,  2.02it/s]{'loss': 2.5753, 'grad_norm': 0.36764204502105713, 'learning_rate': 0.0003753889328974423, 'epoch': 0.62}

                                                 

 62%|██████▏   | 475/763 [05:32<02:22,  2.02it/s]
 62%|██████▏   | 476/763 [05:32<02:22,  2.01it/s]
 63%|██████▎   | 477/763 [05:33<02:21,  2.02it/s]
 63%|██████▎   | 478/763 [05:33<02:21,  2.02it/s]
 63%|██████▎   | 479/763 [05:34<02:20,  2.02it/s]
 63%|██████▎   | 480/763 [05:34<02:20,  2.02it/s]
 63%|██████▎   | 481/763 [05:35<02:19,  2.02it/s]
 63%|██████▎   | 482/763 [05:35<02:19,  2.02it/s]
 63%|██████▎   | 483/763 [05:36<02:18,  2.02it/s]
 63%|██████▎   | 484/763 [05:36<02:17,  2.02it/s]
 64%|██████▎   | 485/763 [05:37<02:17,  2.02it/s]
 64%|██████▎   | 486/763 [05:37<02:16,  2.02it/s]
 64%|██████▍   | 487/763 [05:38<02:16,  2.02it/s]
 64%|██████▍   | 488/763 [05:38<02:15,  2.02it/s]
 64%|██████▍   | 489/763 [05:39<02:15,  2.02it/s]
 64%|██████▍   | 490/763 [05:39<02:14,  2.03it/s]
 64%|██████▍   | 491/763 [05:40<02:14,  2.02it/s]
 64%|██████▍   | 492/763 [05:40<02:13,  2.03it/s]
 65%|██████▍   | 493/763 [05:41<02:13,  2.02it/s]
 65%|██████▍   | 494/763 [05:41<02:12,  2.03it/s]
 65%|██████▍   | 495/763 [05:42<02:12,  2.03it/s]
 65%|██████▌   | 496/763 [05:42<02:11,  2.02it/s]
 65%|██████▌   | 497/763 [05:43<02:11,  2.03it/s]
 65%|██████▌   | 498/763 [05:43<02:10,  2.03it/s]
 65%|██████▌   | 499/763 [05:44<02:10,  2.02it/s]
 66%|██████▌   | 500/763 [05:44<02:09,  2.02it/s]{'loss': 2.5207, 'grad_norm': 0.35472121834754944, 'learning_rate': 0.00032088728410319416, 'epoch': 0.66}

                                                 

 66%|██████▌   | 500/763 [05:44<02:09,  2.02it/s]
 66%|██████▌   | 501/763 [05:45<02:09,  2.02it/s]
 66%|██████▌   | 502/763 [05:45<02:09,  2.02it/s]
 66%|██████▌   | 503/763 [05:46<02:08,  2.02it/s]
 66%|██████▌   | 504/763 [05:46<02:08,  2.02it/s]
 66%|██████▌   | 505/763 [05:47<02:07,  2.02it/s]
 66%|██████▋   | 506/763 [05:47<02:07,  2.02it/s]
 66%|██████▋   | 507/763 [05:48<02:06,  2.02it/s]
 67%|██████▋   | 508/763 [05:48<02:05,  2.03it/s]
 67%|██████▋   | 509/763 [05:49<02:05,  2.03it/s]
 67%|██████▋   | 510/763 [05:49<02:04,  2.03it/s]
 67%|██████▋   | 511/763 [05:50<02:04,  2.02it/s]
 67%|██████▋   | 512/763 [05:50<02:03,  2.02it/s]
 67%|██████▋   | 513/763 [05:51<02:03,  2.02it/s]
 67%|██████▋   | 514/763 [05:51<02:03,  2.02it/s]
 67%|██████▋   | 515/763 [05:52<02:02,  2.02it/s]
 68%|██████▊   | 516/763 [05:52<02:01,  2.02it/s]
 68%|██████▊   | 517/763 [05:53<02:01,  2.02it/s]
 68%|██████▊   | 518/763 [05:53<02:00,  2.03it/s]
 68%|██████▊   | 519/763 [05:54<02:00,  2.03it/s]
 68%|██████▊   | 520/763 [05:54<01:59,  2.03it/s]
 68%|██████▊   | 521/763 [05:55<01:59,  2.03it/s]
 68%|██████▊   | 522/763 [05:55<01:59,  2.02it/s]
 69%|██████▊   | 523/763 [05:56<01:58,  2.03it/s]
 69%|██████▊   | 524/763 [05:56<01:58,  2.02it/s]
 69%|██████▉   | 525/763 [05:57<01:57,  2.03it/s]{'loss': 2.4755, 'grad_norm': 0.342868447303772, 'learning_rate': 0.0002687308548795825, 'epoch': 0.69}

                                                 

 69%|██████▉   | 525/763 [05:57<01:57,  2.03it/s]
 69%|██████▉   | 526/763 [05:57<01:57,  2.02it/s]
 69%|██████▉   | 527/763 [05:57<01:56,  2.02it/s]
 69%|██████▉   | 528/763 [05:58<01:56,  2.02it/s]
 69%|██████▉   | 529/763 [05:58<01:55,  2.03it/s]
 69%|██████▉   | 530/763 [05:59<01:55,  2.02it/s]
 70%|██████▉   | 531/763 [05:59<01:54,  2.02it/s]
 70%|██████▉   | 532/763 [06:00<01:54,  2.02it/s]
 70%|██████▉   | 533/763 [06:00<01:53,  2.02it/s]
 70%|██████▉   | 534/763 [06:01<01:53,  2.02it/s]
 70%|███████   | 535/763 [06:01<01:52,  2.02it/s]
 70%|███████   | 536/763 [06:02<01:52,  2.02it/s]
 70%|███████   | 537/763 [06:02<01:51,  2.02it/s]
 71%|███████   | 538/763 [06:03<01:51,  2.02it/s]
 71%|███████   | 539/763 [06:03<01:50,  2.02it/s]
 71%|███████   | 540/763 [06:04<01:50,  2.02it/s]
 71%|███████   | 541/763 [06:04<01:49,  2.02it/s]
 71%|███████   | 542/763 [06:05<01:49,  2.02it/s]
 71%|███████   | 543/763 [06:05<01:48,  2.02it/s]
 71%|███████▏  | 544/763 [06:06<01:48,  2.02it/s]
 71%|███████▏  | 545/763 [06:06<01:47,  2.02it/s]
 72%|███████▏  | 546/763 [06:07<01:47,  2.02it/s]
 72%|███████▏  | 547/763 [06:07<01:46,  2.02it/s]
 72%|███████▏  | 548/763 [06:08<01:46,  2.02it/s]
 72%|███████▏  | 549/763 [06:08<01:45,  2.02it/s]
 72%|███████▏  | 550/763 [06:09<01:45,  2.02it/s]{'loss': 2.436, 'grad_norm': 0.36740800738334656, 'learning_rate': 0.00021960255753653008, 'epoch': 0.72}

                                                 

 72%|███████▏  | 550/763 [06:09<01:45,  2.02it/s]
 72%|███████▏  | 551/763 [06:09<01:44,  2.02it/s]
 72%|███████▏  | 552/763 [06:10<01:44,  2.02it/s]
 72%|███████▏  | 553/763 [06:10<01:43,  2.02it/s]
 73%|███████▎  | 554/763 [06:11<01:43,  2.02it/s]
 73%|███████▎  | 555/763 [06:11<01:42,  2.02it/s]
 73%|███████▎  | 556/763 [06:12<01:42,  2.02it/s]
 73%|███████▎  | 557/763 [06:12<01:41,  2.02it/s]
 73%|███████▎  | 558/763 [06:13<01:41,  2.02it/s]
 73%|███████▎  | 559/763 [06:13<01:40,  2.02it/s]
 73%|███████▎  | 560/763 [06:14<01:40,  2.02it/s]
 74%|███████▎  | 561/763 [06:14<01:39,  2.02it/s]
 74%|███████▎  | 562/763 [06:15<01:39,  2.02it/s]
 74%|███████▍  | 563/763 [06:15<01:38,  2.02it/s]
 74%|███████▍  | 564/763 [06:16<01:38,  2.02it/s]
 74%|███████▍  | 565/763 [06:16<01:37,  2.02it/s]
 74%|███████▍  | 566/763 [06:17<01:37,  2.02it/s]
 74%|███████▍  | 567/763 [06:17<01:36,  2.02it/s]
 74%|███████▍  | 568/763 [06:18<01:36,  2.02it/s]
 75%|███████▍  | 569/763 [06:18<01:36,  2.02it/s]
 75%|███████▍  | 570/763 [06:19<01:35,  2.02it/s]
 75%|███████▍  | 571/763 [06:19<01:34,  2.02it/s]
 75%|███████▍  | 572/763 [06:20<01:34,  2.02it/s]
 75%|███████▌  | 573/763 [06:20<01:33,  2.02it/s]
 75%|███████▌  | 574/763 [06:21<01:33,  2.02it/s]
 75%|███████▌  | 575/763 [06:21<01:32,  2.02it/s]{'loss': 2.4095, 'grad_norm': 0.3562281131744385, 'learning_rate': 0.00017414565541703342, 'epoch': 0.75}

                                                 

 75%|███████▌  | 575/763 [06:21<01:32,  2.02it/s]
 75%|███████▌  | 576/763 [06:22<01:32,  2.02it/s]
 76%|███████▌  | 577/763 [06:22<01:32,  2.02it/s]
 76%|███████▌  | 578/763 [06:23<01:31,  2.02it/s]
 76%|███████▌  | 579/763 [06:23<01:31,  2.02it/s]
 76%|███████▌  | 580/763 [06:24<01:30,  2.02it/s]
 76%|███████▌  | 581/763 [06:24<01:29,  2.02it/s]
 76%|███████▋  | 582/763 [06:25<01:29,  2.02it/s]
 76%|███████▋  | 583/763 [06:25<01:28,  2.02it/s]
 77%|███████▋  | 584/763 [06:26<01:28,  2.02it/s]
 77%|███████▋  | 585/763 [06:26<01:28,  2.02it/s]
 77%|███████▋  | 586/763 [06:27<01:27,  2.02it/s]
 77%|███████▋  | 587/763 [06:27<01:27,  2.02it/s]
 77%|███████▋  | 588/763 [06:28<01:26,  2.02it/s]
 77%|███████▋  | 589/763 [06:28<01:25,  2.02it/s]
 77%|███████▋  | 590/763 [06:29<01:25,  2.02it/s]
 77%|███████▋  | 591/763 [06:29<01:24,  2.02it/s]
 78%|███████▊  | 592/763 [06:30<01:24,  2.02it/s]
 78%|███████▊  | 593/763 [06:30<01:24,  2.02it/s]
 78%|███████▊  | 594/763 [06:31<01:23,  2.02it/s]
 78%|███████▊  | 595/763 [06:31<01:23,  2.02it/s]
 78%|███████▊  | 596/763 [06:32<01:22,  2.02it/s]
 78%|███████▊  | 597/763 [06:32<01:22,  2.02it/s]
 78%|███████▊  | 598/763 [06:33<01:21,  2.02it/s]
 79%|███████▊  | 599/763 [06:33<01:21,  2.02it/s]
 79%|███████▊  | 600/763 [06:34<01:20,  2.02it/s]
                                                 {'loss': 2.3795, 'grad_norm': 0.35245397686958313, 'learning_rate': 0.0001329553403026331, 'epoch': 0.79}

 79%|███████▊  | 600/763 [06:34<01:20,  2.02it/s]
 79%|███████▉  | 601/763 [06:34<01:20,  2.02it/s]
 79%|███████▉  | 602/763 [06:35<01:19,  2.02it/s]
 79%|███████▉  | 603/763 [06:35<01:18,  2.03it/s]
 79%|███████▉  | 604/763 [06:36<01:18,  2.02it/s]
 79%|███████▉  | 605/763 [06:36<01:17,  2.03it/s]
 79%|███████▉  | 606/763 [06:37<01:17,  2.03it/s]
 80%|███████▉  | 607/763 [06:37<01:17,  2.03it/s]
 80%|███████▉  | 608/763 [06:38<01:16,  2.02it/s]
 80%|███████▉  | 609/763 [06:38<01:22,  1.86it/s]
 80%|███████▉  | 610/763 [06:39<01:20,  1.91it/s]
 80%|████████  | 611/763 [06:39<01:18,  1.94it/s]
 80%|████████  | 612/763 [06:40<01:16,  1.97it/s]
 80%|████████  | 613/763 [06:40<01:15,  1.98it/s]
 80%|████████  | 614/763 [06:41<01:14,  2.00it/s]
 81%|████████  | 615/763 [06:41<01:13,  2.00it/s]
 81%|████████  | 616/763 [06:42<01:19,  1.85it/s]
 81%|████████  | 617/763 [06:42<01:16,  1.90it/s]
 81%|████████  | 618/763 [06:43<01:14,  1.94it/s]
 81%|████████  | 619/763 [06:43<01:13,  1.96it/s]
 81%|████████▏ | 620/763 [06:44<01:12,  1.98it/s]
 81%|████████▏ | 621/763 [06:44<01:11,  1.99it/s]
 82%|████████▏ | 622/763 [06:45<01:10,  2.00it/s]
 82%|████████▏ | 623/763 [06:45<01:09,  2.01it/s]
 82%|████████▏ | 624/763 [06:46<01:09,  2.01it/s]
 82%|████████▏ | 625/763 [06:46<01:08,  2.01it/s]{'loss': 2.3607, 'grad_norm': 0.3452637791633606, 'learning_rate': 9.657093924581261e-05, 'epoch': 0.82}
                                                 

 82%|████████▏ | 625/763 [06:46<01:08,  2.01it/s]
 82%|████████▏ | 626/763 [06:47<01:07,  2.02it/s]
 82%|████████▏ | 627/763 [06:47<01:07,  2.02it/s]
 82%|████████▏ | 628/763 [06:48<01:06,  2.02it/s]
 82%|████████▏ | 629/763 [06:48<01:06,  2.02it/s]
 83%|████████▎ | 630/763 [06:49<01:05,  2.02it/s]
 83%|████████▎ | 631/763 [06:49<01:05,  2.02it/s]
 83%|████████▎ | 632/763 [06:50<01:04,  2.02it/s]
 83%|████████▎ | 633/763 [06:50<01:04,  2.03it/s]
 83%|████████▎ | 634/763 [06:51<01:03,  2.02it/s]
 83%|████████▎ | 635/763 [06:51<01:03,  2.03it/s]
 83%|████████▎ | 636/763 [06:52<01:02,  2.03it/s]
 83%|████████▎ | 637/763 [06:52<01:02,  2.02it/s]
 84%|████████▎ | 638/763 [06:53<01:01,  2.02it/s]
 84%|████████▎ | 639/763 [06:53<01:01,  2.02it/s]
 84%|████████▍ | 640/763 [06:54<01:00,  2.02it/s]
 84%|████████▍ | 641/763 [06:54<01:00,  2.02it/s]
 84%|████████▍ | 642/763 [06:55<00:59,  2.02it/s]
 84%|████████▍ | 643/763 [06:55<00:59,  2.02it/s]
 84%|████████▍ | 644/763 [06:56<00:58,  2.02it/s]
 85%|████████▍ | 645/763 [06:56<00:58,  2.02it/s]
 85%|████████▍ | 646/763 [06:57<00:57,  2.02it/s]
 85%|████████▍ | 647/763 [06:57<00:57,  2.02it/s]
 85%|████████▍ | 648/763 [06:58<00:56,  2.02it/s]
 85%|████████▌ | 649/763 [06:58<00:56,  2.02it/s]
 85%|████████▌ | 650/763 [06:59<00:55,  2.02it/s]{'loss': 2.3353, 'grad_norm': 0.34281203150749207, 'learning_rate': 6.546885286948184e-05, 'epoch': 0.85}

                                                 

 85%|████████▌ | 650/763 [06:59<00:55,  2.02it/s]
 85%|████████▌ | 651/763 [06:59<00:55,  2.01it/s]
 85%|████████▌ | 652/763 [07:00<00:54,  2.02it/s]
 86%|████████▌ | 653/763 [07:00<00:54,  2.02it/s]
 86%|████████▌ | 654/763 [07:01<00:53,  2.02it/s]
 86%|████████▌ | 655/763 [07:01<00:53,  2.02it/s]
 86%|████████▌ | 656/763 [07:02<00:52,  2.03it/s]
 86%|████████▌ | 657/763 [07:02<00:52,  2.03it/s]
 86%|████████▌ | 658/763 [07:03<00:51,  2.03it/s]
 86%|████████▋ | 659/763 [07:03<00:51,  2.03it/s]
 87%|████████▋ | 660/763 [07:04<00:50,  2.03it/s]
 87%|████████▋ | 661/763 [07:04<00:50,  2.03it/s]
 87%|████████▋ | 662/763 [07:05<00:49,  2.03it/s]
 87%|████████▋ | 663/763 [07:05<00:49,  2.03it/s]
 87%|████████▋ | 664/763 [07:05<00:48,  2.03it/s]
 87%|████████▋ | 665/763 [07:06<00:48,  2.03it/s]
 87%|████████▋ | 666/763 [07:06<00:47,  2.03it/s]
 87%|████████▋ | 667/763 [07:07<00:47,  2.03it/s]
 88%|████████▊ | 668/763 [07:07<00:46,  2.03it/s]
 88%|████████▊ | 669/763 [07:08<00:46,  2.03it/s]
 88%|████████▊ | 670/763 [07:08<00:45,  2.03it/s]
 88%|████████▊ | 671/763 [07:09<00:45,  2.03it/s]
 88%|████████▊ | 672/763 [07:09<00:44,  2.03it/s]
 88%|████████▊ | 673/763 [07:10<00:44,  2.03it/s]
 88%|████████▊ | 674/763 [07:10<00:43,  2.03it/s]
 88%|████████▊ | 675/763 [07:11<00:43,  2.03it/s]{'loss': 2.3277, 'grad_norm': 0.33725234866142273, 'learning_rate': 4.0056317596204094e-05, 'epoch': 0.88}

                                                 

 88%|████████▊ | 675/763 [07:11<00:43,  2.03it/s]
 89%|████████▊ | 676/763 [07:11<00:42,  2.02it/s]
 89%|████████▊ | 677/763 [07:12<00:42,  2.02it/s]
 89%|████████▉ | 678/763 [07:12<00:41,  2.03it/s]
 89%|████████▉ | 679/763 [07:13<00:41,  2.03it/s]
 89%|████████▉ | 680/763 [07:13<00:40,  2.03it/s]
 89%|████████▉ | 681/763 [07:14<00:40,  2.03it/s]
 89%|████████▉ | 682/763 [07:14<00:39,  2.03it/s]
 90%|████████▉ | 683/763 [07:15<00:39,  2.03it/s]
 90%|████████▉ | 684/763 [07:15<00:38,  2.03it/s]
 90%|████████▉ | 685/763 [07:16<00:38,  2.03it/s]
 90%|████████▉ | 686/763 [07:16<00:37,  2.03it/s]
 90%|█████████ | 687/763 [07:17<00:37,  2.03it/s]
 90%|█████████ | 688/763 [07:17<00:36,  2.03it/s]
 90%|█████████ | 689/763 [07:18<00:36,  2.03it/s]
 90%|█████████ | 690/763 [07:18<00:36,  2.03it/s]
 91%|█████████ | 691/763 [07:19<00:35,  2.03it/s]
 91%|█████████ | 692/763 [07:19<00:35,  2.03it/s]
 91%|█████████ | 693/763 [07:20<00:34,  2.03it/s]
 91%|█████████ | 694/763 [07:20<00:34,  2.03it/s]
 91%|█████████ | 695/763 [07:21<00:33,  2.03it/s]
 91%|█████████ | 696/763 [07:21<00:33,  2.03it/s]
 91%|█████████▏| 697/763 [07:22<00:32,  2.03it/s]
 91%|█████████▏| 698/763 [07:22<00:32,  2.03it/s]
 92%|█████████▏| 699/763 [07:23<00:31,  2.03it/s]
 92%|█████████▏| 700/763 [07:23<00:31,  2.03it/s]{'loss': 2.3087, 'grad_norm': 0.33684036135673523, 'learning_rate': 2.0666073481669712e-05, 'epoch': 0.92}

                                                 

 92%|█████████▏| 700/763 [07:23<00:31,  2.03it/s]
 92%|█████████▏| 701/763 [07:24<00:30,  2.02it/s]
 92%|█████████▏| 702/763 [07:24<00:30,  2.02it/s]
 92%|█████████▏| 703/763 [07:25<00:29,  2.03it/s]
 92%|█████████▏| 704/763 [07:25<00:29,  2.03it/s]
 92%|█████████▏| 705/763 [07:26<00:28,  2.03it/s]
 93%|█████████▎| 706/763 [07:26<00:28,  2.03it/s]
 93%|█████████▎| 707/763 [07:27<00:27,  2.03it/s]
 93%|█████████▎| 708/763 [07:27<00:27,  2.03it/s]
 93%|█████████▎| 709/763 [07:28<00:26,  2.03it/s]
 93%|█████████▎| 710/763 [07:28<00:26,  2.03it/s]
 93%|█████████▎| 711/763 [07:29<00:25,  2.03it/s]
 93%|█████████▎| 712/763 [07:29<00:25,  2.03it/s]
 93%|█████████▎| 713/763 [07:30<00:24,  2.03it/s]
 94%|█████████▎| 714/763 [07:30<00:24,  2.02it/s]
 94%|█████████▎| 715/763 [07:31<00:23,  2.00it/s]
 94%|█████████▍| 716/763 [07:31<00:23,  1.99it/s]
 94%|█████████▍| 717/763 [07:32<00:23,  1.98it/s]
 94%|█████████▍| 718/763 [07:32<00:22,  1.98it/s]
 94%|█████████▍| 719/763 [07:33<00:22,  1.97it/s]
 94%|█████████▍| 720/763 [07:33<00:21,  1.98it/s]
 94%|█████████▍| 721/763 [07:34<00:21,  1.99it/s]
 95%|█████████▍| 722/763 [07:34<00:20,  2.00it/s]
 95%|█████████▍| 723/763 [07:35<00:19,  2.01it/s]
 95%|█████████▍| 724/763 [07:35<00:19,  2.01it/s]
 95%|█████████▌| 725/763 [07:36<00:18,  2.02it/s]
                                                 
{'loss': 2.3126, 'grad_norm': 0.33316919207572937, 'learning_rate': 7.552007469355249e-06, 'epoch': 0.95}

 95%|█████████▌| 725/763 [07:36<00:18,  2.02it/s]
 95%|█████████▌| 726/763 [07:36<00:18,  2.02it/s]
 95%|█████████▌| 727/763 [07:37<00:17,  2.02it/s]
 95%|█████████▌| 728/763 [07:37<00:17,  2.02it/s]
 96%|█████████▌| 729/763 [07:38<00:16,  2.02it/s]
 96%|█████████▌| 730/763 [07:38<00:16,  2.03it/s]
 96%|█████████▌| 731/763 [07:39<00:15,  2.03it/s]
 96%|█████████▌| 732/763 [07:39<00:15,  2.03it/s]
 96%|█████████▌| 733/763 [07:40<00:14,  2.03it/s]
 96%|█████████▌| 734/763 [07:40<00:14,  2.03it/s]
 96%|█████████▋| 735/763 [07:41<00:13,  2.03it/s]
 96%|█████████▋| 736/763 [07:41<00:13,  2.03it/s]
 97%|█████████▋| 737/763 [07:42<00:12,  2.03it/s]
 97%|█████████▋| 738/763 [07:42<00:12,  2.03it/s]
 97%|█████████▋| 739/763 [07:43<00:11,  2.03it/s]
 97%|█████████▋| 740/763 [07:43<00:11,  2.03it/s]
 97%|█████████▋| 741/763 [07:44<00:10,  2.03it/s]
 97%|█████████▋| 742/763 [07:44<00:10,  2.03it/s]
 97%|█████████▋| 743/763 [07:45<00:09,  2.02it/s]
 98%|█████████▊| 744/763 [07:45<00:09,  2.00it/s]
 98%|█████████▊| 745/763 [07:46<00:09,  1.98it/s]
 98%|█████████▊| 746/763 [07:46<00:08,  1.98it/s]
 98%|█████████▊| 747/763 [07:47<00:08,  1.98it/s]
 98%|█████████▊| 748/763 [07:47<00:07,  1.98it/s]
 98%|█████████▊| 749/763 [07:48<00:07,  1.99it/s]
 98%|█████████▊| 750/763 [07:48<00:06,  2.00it/s]{'loss': 2.2993, 'grad_norm': 0.3305748701095581, 'learning_rate': 8.858291115876327e-07, 'epoch': 0.98}

                                                 

 98%|█████████▊| 750/763 [07:48<00:06,  2.00it/s]
 98%|█████████▊| 751/763 [07:49<00:05,  2.00it/s]
 99%|█████████▊| 752/763 [07:49<00:05,  2.01it/s]
 99%|█████████▊| 753/763 [07:50<00:04,  2.01it/s]
 99%|█████████▉| 754/763 [07:50<00:04,  2.02it/s]
 99%|█████████▉| 755/763 [07:51<00:03,  2.02it/s]
 99%|█████████▉| 756/763 [07:51<00:03,  2.02it/s]
 99%|█████████▉| 757/763 [07:52<00:02,  2.02it/s]
 99%|█████████▉| 758/763 [07:52<00:02,  2.02it/s]
 99%|█████████▉| 759/763 [07:53<00:01,  2.02it/s]
100%|█████████▉| 760/763 [07:53<00:01,  2.02it/s]
100%|█████████▉| 761/763 [07:54<00:00,  2.02it/s]
100%|█████████▉| 762/763 [07:54<00:00,  2.02it/s]
100%|██████████| 763/763 [07:55<00:00,  2.04it/s]
                                                 
{'train_runtime': 493.6697, 'train_samples_per_second': 1582.627, 'train_steps_per_second': 1.546, 'train_loss': 3.4241124799335925, 'epoch': 1.0}

100%|██████████| 763/763 [08:13<00:00,  2.04it/s]
100%|██████████| 763/763 [08:13<00:00,  1.55it/s]
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.