Training in progress, step 18500

.gitattributes

@@ -26,3 +26,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zstandard filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 wandb/run-20220203_135844-2tzexn1o/run-2tzexn1o.wandb filter=lfs diff=lfs merge=lfs -text
+wandb/run-20220206_014057-iss04l8x/run-iss04l8x.wandb filter=lfs diff=lfs merge=lfs -text

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:f44b2861a901303bf11ed8159197e9830963e83950734a2f6068e6fc84f162c1
+oid sha256:90f77037d236d8da993fb6bb4cd3c46c0b5c75dcf755f9214213b68b0dba67b6
 size 1262112241

wandb/run-20220206_014057-iss04l8x/files/output.log

@@ -1729,3 +1729,870 @@ Saving model checkpoint to ./checkpoint-18000
 Configuration saved in ./checkpoint-18000/config.json███████████████████████████████████████████▍                                | 18000/24416 [5:58:08<17:33:20,  9.85s/it]
 Model weights saved in ./checkpoint-18000/pytorch_model.bin
 Configuration saved in ./checkpoint-18000/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-16000] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 74%|███████████████████████████████████████████████████████████████████████████████████████████▉                                | 18099/24416 [6:12:22<17:35:29, 10.03s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 75%|████████████████████████████████████████████████████████████████████████████████████████████▍                               | 18199/24416 [6:25:07<17:24:59, 10.09s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 75%|████████████████████████████████████████████████████████████████████████████████████████████▉                               | 18300/24416 [6:37:50<16:50:54,  9.92s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 75%|█████████████████████████████████████████████████████████████████████████████████████████████▍                              | 18400/24416 [6:50:29<16:33:45,  9.91s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ 76%|█████████████████████████████████████████████████████████████████████████████████████████████▉                              | 18499/24416 [7:02:52<16:23:07,  9.97s/it]
+***** Running Evaluation *****████████████████████████████████████████████████████████████████████▉                              | 18500/24416 [7:03:02<16:12:06,  9.86s/it]
+  Num examples = 16021
+  Batch size = 16
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1002/1002 [12:00<00:00,  2.31it/s]
+Saving model checkpoint to ./checkpoint-18500
+Configuration saved in ./checkpoint-18500/config.json████████████████████████████████████████████���▉                              | 18500/24416 [7:15:16<16:12:06,  9.86s/it]
+Model weights saved in ./checkpoint-18500/pytorch_model.bin
+Configuration saved in ./checkpoint-18500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+02/06/2022 08:57:43 - WARNING - huggingface_hub.repository - Adding files tracked by Git LFS: ['wandb/run-20220206_014057-iss04l8x/run-iss04l8x.wandb']. This may take a bit of time if the files are large.