Training in progress, step 1000

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:72685a6be9d4cf07d2dca978c04109b7eaf6f1812b0c6e98ad4e1009fcb3ebb0
+oid sha256:1eb88e4c0f29d23d10bfda68d68f4540b36c13d540a2158dba247f5eceb9d45c
 size 1262817457

wandb/run-20220129_215451-1vipdbow/files/output.log

@@ -592,3 +592,598 @@ Configuration saved in ./checkpoint-500/config.json
 {'eval_loss': 3.878422975540161, 'eval_wer': 1.0, 'eval_runtime': 285.8223, 'eval_samples_per_second': 20.264, 'eval_steps_per_second': 0.318, 'epoch': 0.07}
 Model weights saved in ./checkpoint-500/pytorch_model.bin
 Configuration saved in ./checkpoint-500/preprocessor_config.json
+Configuration saved in ./preprocessor_config.json
+Deleting older checkpoint [checkpoint-7500] due to args.save_total_limit
+Deleting older checkpoint [checkpoint-8000] due to args.save_total_limit
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  4%|██████▉                                                                                                                                                        | 599/13822 [37:50<6:34:02,  1.79s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  5%|████████                                                                                                                                                       | 700/13822 [43:10<6:08:25,  1.68s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  6%|█████████▏                                                                                                                                                     | 799/13822 [48:26<6:30:17,  1.80s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  7%|██████████▎                                                                                                                                                    | 900/13822 [53:43<6:11:25,  1.72s/it]
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+  7%|███████████▍                                                                                                                                                  | 1000/13822 [58:59<6:02:01,  1.69s/it]The following columns in the evaluation set  don't have a corresponding argument in `Wav2Vec2ForCTC.forward` and have been ignored: input_length.
+***** Running Evaluation *****
+  Num examples = 5792
+  Batch size = 64
+{'loss': 2.8068, 'learning_rate': 6.646666666666667e-05, 'epoch': 0.14}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 91/91 [04:08<00:00,  2.37s/it]
+
+Configuration saved in ./checkpoint-1000/config.json
+Model weights saved in ./checkpoint-1000/pytorch_model.bin
+Configuration saved in ./checkpoint-1000/preprocessor_config.json