| 0%| | 0/5000 [00:00<?, ?it/s] | |
| Reading metadata...: 23332it [00:04, 5566.60it/s] | |
| [INFO|trainer_utils.py:693] 2023-05-06 11:35:39,880 >> The following columns in the training set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| /home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector. | |
| warnings.warn('Was asked to gather along dimension 0, but all ' | |
| 0%| | 25/5000 [14:20<40:49:08, 29.54s/it] | |
| 1%| | 49/5000 [25:57<39:52:38, 29.00s/it] | |
| 1%|▏ | 74/5000 [38:10<39:26:07, 28.82s/it] | |
| 2%|▏ | 99/5000 [50:13<39:03:58, 28.70s/it] | |
| 2%|▏ | 124/5000 [1:02:25<38:47:32, 28.64s/it] | |
| 3%|▎ | 150/5000 [1:15:09<40:31:35, 30.08s/it] | |
| 3%|▎ | 163/5000 [1:20:20<21:29:00, 15.99s/it] | |
| Reading metadata...: 22927it [00:03, 10857.89it/s] | |
| 3%|▎ | 174/5000 [1:26:45<37:53:20, 28.26s/it] | |
| 4%|▍ | 200/5000 [1:38:49<36:55:58, 27.70s/it] | |
| 4%|▍ | 224/5000 [1:49:53<36:01:48, 27.16s/it] | |
| 5%|▍ | 249/5000 [2:01:25<35:37:24, 26.99s/it] | |
| 5%|▌ | 274/5000 [2:12:59<36:06:06, 27.50s/it] | |
| 6%|▌ | 299/5000 [2:24:40<37:31:10, 28.73s/it] | |
| 6%|▋ | 324/5000 [2:35:58<29:09:59, 22.45s/it] | |
| 7%|▋ | 327/5000 [2:36:27<18:05:01, 13.93s/it] | |
| Reading metadata...: 24506it [00:05, 5432.25it/s] | |
| 7%|▋ | 349/5000 [2:48:13<35:54:20, 27.79s/it] | |
| 7%|▋ | 374/5000 [3:00:12<40:33:48, 31.57s/it] | |
| 8%|▊ | 400/5000 [3:12:45<36:54:59, 28.89s/it] | |
| 8%|▊ | 425/5000 [3:24:31<35:49:54, 28.20s/it] | |
| 9%|▉ | 449/5000 [3:35:37<34:58:52, 27.67s/it] | |
| 10%|▉ | 475/5000 [3:47:38<34:53:17, 27.76s/it] | |
| 10%|▉ | 490/5000 [3:53:33<19:04:32, 15.23s/it] | |
| Reading metadata...: 10438it [00:00, 25934.78it/s] | |
| 10%|▉ | 499/5000 [3:58:50<37:18:31, 29.84s/it] | |
| 10%|█ | 525/5000 [4:10:52<34:41:42, 27.91s/it] | |
| 11%|█ | 550/5000 [4:22:29<34:29:35, 27.90s/it] | |
| 12%|█▏ | 575/5000 [4:34:05<34:32:46, 28.11s/it] | |
| 12%|█▏ | 600/5000 [4:45:35<34:09:51, 27.95s/it] | |
| 12%|█▎ | 625/5000 [4:57:10<33:31:38, 27.59s/it] | |
| 13%|█▎ | 650/5000 [5:08:39<31:35:25, 26.14s/it] | |
| 13%|█▎ | 654/5000 [5:09:19<16:25:57, 13.61s/it] | |
| Reading metadata...: 10438it [00:00, 25770.40it/s] | |
| 13%|█▎ | 674/5000 [5:19:54<34:45:47, 28.93s/it] | |
| 14%|█▍ | 700/5000 [5:31:55<32:45:38, 27.43s/it] | |
| 14%|█▍ | 724/5000 [5:43:03<33:14:52, 27.99s/it] | |
| 15%|█▌ | 750/5000 [5:55:12<34:08:13, 28.92s/it] | |
| 16%|█▌ | 775/5000 [6:06:46<32:23:45, 27.60s/it] | |
| 16%|█▌ | 800/5000 [6:18:21<32:13:15, 27.62s/it] | |
| Reading metadata...: 28043it [00:01, 19987.99it/s]s/it] | |
| Reading metadata...: 10438it [00:00, 25735.72it/s] | |
| 16%|█▋ | 825/5000 [6:29:57<33:54:47, 29.24s/it] | |
| 17%|█▋ | 850/5000 [6:41:29<31:44:09, 27.53s/it] | |
| 18%|█▊ | 875/5000 [6:53:08<31:50:31, 27.79s/it] | |
| 18%|█▊ | 900/5000 [7:04:43<31:41:55, 27.83s/it] | |
| 18%|█▊ | 925/5000 [7:16:16<31:19:18, 27.67s/it] | |
| 19%|█▉ | 949/5000 [7:27:24<31:16:42, 27.80s/it] | |
| 20%|█▉ | 975/5000 [7:39:31<30:54:27, 27.64s/it] | |
| Reading metadata...: 28043it [00:00, 28643.05it/s]s/it] | |
| Reading metadata...: 10438it [00:00, 24507.54it/s] | |
| 20%|██ | 1000/5000 [7:51:12<31:18:37, 28.18s/it][INFO|trainer.py:3138] 2023-05-06 19:25:00,699 >> ***** Running Evaluation ***** | |
| [INFO|trainer.py:3142] 2023-05-06 19:25:00,699 >> Num examples: Unknown | |
| [INFO|trainer.py:3143] 2023-05-06 19:25:00,699 >> Batch size = 64 | |
| {'loss': 0.0517, 'learning_rate': 8.893333333333333e-06, 'epoch': 6.0} | |
| [INFO|trainer_utils.py:693] 2023-05-06 19:25:16,854 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`, you can safely ignore this message. | |
| {'eval_loss': 0.43406692147254944, 'eval_wer': 54.476, 'eval_runtime': 2259.524, 'eval_samples_per_second': 4.62, 'eval_steps_per_second': 0.073, 'epoch': 6.0} | |
| 20%|██ | 1000/5000 [8:28:52<31:18:37, 28.18s/it][INFO|trainer.py:2877] 2023-05-06 20:02:40,234 >> Saving model checkpoint to ./checkpoint-1000 | |
| [INFO|configuration_utils.py:458] 2023-05-06 20:02:40,239 >> Configuration saved in ./checkpoint-1000/config.json | |
| [INFO|configuration_utils.py:364] 2023-05-06 20:02:40,243 >> Configuration saved in ./checkpoint-1000/generation_config.json | |
| [INFO|modeling_utils.py:1855] 2023-05-06 20:02:43,051 >> Model weights saved in ./checkpoint-1000/pytorch_model.bin | |
| [INFO|feature_extraction_utils.py:369] 2023-05-06 20:02:43,056 >> Feature extractor saved in ./checkpoint-1000/preprocessor_config.json | |
| [INFO|feature_extraction_utils.py:369] 2023-05-06 20:02:51,152 >> Feature extractor saved in ./preprocessor_config.json | |
| Traceback (most recent call last): | |
| File "/home/local/QCRI/dizham/kanari/whisper/whisper-small-ar/run_speech_recognition_seq2seq_streaming.py", line 629, in <module> | |
| main() | |
| File "/home/local/QCRI/dizham/kanari/whisper/whisper-small-ar/run_speech_recognition_seq2seq_streaming.py", line 578, in main | |
| train_result = trainer.train(resume_from_checkpoint=checkpoint) | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 1664, in train | |
| return inner_training_loop( | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 2011, in _inner_training_loop | |
| self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval) | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 2300, in _maybe_log_save_evaluate | |
| self._save_checkpoint(model, trial, metrics=metrics) | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 2444, in _save_checkpoint | |
| self._push_from_checkpoint(output_dir) | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 3622, in _push_from_checkpoint | |
| _, self.push_in_progress = self.repo.push_to_hub( | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 1305, in push_to_hub | |
| self.git_add(auto_lfs_track=True) | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 1009, in git_add | |
| tracked_files.extend(self.auto_track_binary_files(pattern)) | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 903, in auto_track_binary_files | |
| is_binary = is_binary_file(path_to_file) | |
| File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 230, in is_binary_file | |
| with open(filename, "rb") as f: | |
| IsADirectoryError: [Errno 21] Is a directory: '/home/local/QCRI/dizham/kanari/whisper/whisper-small-ar/./wandb/latest-run' |