0%|          | 0/5000 [00:00<?, ?it/s]


Reading metadata...: 23332it [00:04, 5566.60it/s]


[INFO|trainer_utils.py:693] 2023-05-06 11:35:39,880 >> The following columns in the training set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/torch/nn/parallel/_functions.py:68: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
  warnings.warn('Was asked to gather along dimension 0, but all '
























  0%|          | 25/5000 [14:20<40:49:08, 29.54s/it]
























  1%|          | 49/5000 [25:57<39:52:38, 29.00s/it]

























  1%|▏         | 74/5000 [38:10<39:26:07, 28.82s/it]

























  2%|▏         | 99/5000 [50:13<39:03:58, 28.70s/it]

























  2%|▏         | 124/5000 [1:02:25<38:47:32, 28.64s/it]


























  3%|▎         | 150/5000 [1:15:09<40:31:35, 30.08s/it]













  3%|▎         | 163/5000 [1:20:20<21:29:00, 15.99s/it]

Reading metadata...: 22927it [00:03, 10857.89it/s]












  3%|▎         | 174/5000 [1:26:45<37:53:20, 28.26s/it]


























  4%|▍         | 200/5000 [1:38:49<36:55:58, 27.70s/it]
























  4%|▍         | 224/5000 [1:49:53<36:01:48, 27.16s/it]

























  5%|▍         | 249/5000 [2:01:25<35:37:24, 26.99s/it]

























  5%|▌         | 274/5000 [2:12:59<36:06:06, 27.50s/it]

























  6%|▌         | 299/5000 [2:24:40<37:31:10, 28.73s/it]

























  6%|▋         | 324/5000 [2:35:58<29:09:59, 22.45s/it]



  7%|▋         | 327/5000 [2:36:27<18:05:01, 13.93s/it]


Reading metadata...: 24506it [00:05, 5432.25it/s]























  7%|▋         | 349/5000 [2:48:13<35:54:20, 27.79s/it]

























  7%|▋         | 374/5000 [3:00:12<40:33:48, 31.57s/it]


























  8%|▊         | 400/5000 [3:12:45<36:54:59, 28.89s/it]

























  8%|▊         | 425/5000 [3:24:31<35:49:54, 28.20s/it]
























  9%|▉         | 449/5000 [3:35:37<34:58:52, 27.67s/it]


























 10%|▉         | 475/5000 [3:47:38<34:53:17, 27.76s/it]















 10%|▉         | 490/5000 [3:53:33<19:04:32, 15.23s/it]

Reading metadata...: 10438it [00:00, 25934.78it/s]









 10%|▉         | 499/5000 [3:58:50<37:18:31, 29.84s/it]


























 10%|█         | 525/5000 [4:10:52<34:41:42, 27.91s/it]

























 11%|█         | 550/5000 [4:22:29<34:29:35, 27.90s/it]

























 12%|█▏        | 575/5000 [4:34:05<34:32:46, 28.11s/it]

























 12%|█▏        | 600/5000 [4:45:35<34:09:51, 27.95s/it]

























 12%|█▎        | 625/5000 [4:57:10<33:31:38, 27.59s/it]

























 13%|█▎        | 650/5000 [5:08:39<31:35:25, 26.14s/it]



 13%|█▎        | 654/5000 [5:09:19<16:25:57, 13.61s/it]

Reading metadata...: 10438it [00:00, 25770.40it/s]




















 13%|█▎        | 674/5000 [5:19:54<34:45:47, 28.93s/it]


























 14%|█▍        | 700/5000 [5:31:55<32:45:38, 27.43s/it]
























 14%|█▍        | 724/5000 [5:43:03<33:14:52, 27.99s/it]


























 15%|█▌        | 750/5000 [5:55:12<34:08:13, 28.92s/it]

























 16%|█▌        | 775/5000 [6:06:46<32:23:45, 27.60s/it]

























 16%|█▌        | 800/5000 [6:18:21<32:13:15, 27.62s/it]

















Reading metadata...: 28043it [00:01, 19987.99it/s]s/it]
Reading metadata...: 10438it [00:00, 25735.72it/s]








 16%|█▋        | 825/5000 [6:29:57<33:54:47, 29.24s/it]

























 17%|█▋        | 850/5000 [6:41:29<31:44:09, 27.53s/it]

























 18%|█▊        | 875/5000 [6:53:08<31:50:31, 27.79s/it]

























 18%|█▊        | 900/5000 [7:04:43<31:41:55, 27.83s/it]

























 18%|█▊        | 925/5000 [7:16:16<31:19:18, 27.67s/it]
























 19%|█▉        | 949/5000 [7:27:24<31:16:42, 27.80s/it]


























 20%|█▉        | 975/5000 [7:39:31<30:54:27, 27.64s/it]





Reading metadata...: 28043it [00:00, 28643.05it/s]s/it]
Reading metadata...: 10438it [00:00, 24507.54it/s]


















 20%|██        | 1000/5000 [7:51:12<31:18:37, 28.18s/it][INFO|trainer.py:3138] 2023-05-06 19:25:00,699 >> ***** Running Evaluation *****
[INFO|trainer.py:3142] 2023-05-06 19:25:00,699 >>   Num examples: Unknown
[INFO|trainer.py:3143] 2023-05-06 19:25:00,699 >>   Batch size = 64
{'loss': 0.0517, 'learning_rate': 8.893333333333333e-06, 'epoch': 6.0}


[INFO|trainer_utils.py:693] 2023-05-06 19:25:16,854 >> The following columns in the evaluation set don't have a corresponding argument in `WhisperForConditionalGeneration.forward` and have been ignored: input_length. If input_length are not expected by `WhisperForConditionalGeneration.forward`,  you can safely ignore this message.
{'eval_loss': 0.43406692147254944, 'eval_wer': 54.476, 'eval_runtime': 2259.524, 'eval_samples_per_second': 4.62, 'eval_steps_per_second': 0.073, 'epoch': 6.0}
 20%|██        | 1000/5000 [8:28:52<31:18:37, 28.18s/it][INFO|trainer.py:2877] 2023-05-06 20:02:40,234 >> Saving model checkpoint to ./checkpoint-1000
[INFO|configuration_utils.py:458] 2023-05-06 20:02:40,239 >> Configuration saved in ./checkpoint-1000/config.json
[INFO|configuration_utils.py:364] 2023-05-06 20:02:40,243 >> Configuration saved in ./checkpoint-1000/generation_config.json
[INFO|modeling_utils.py:1855] 2023-05-06 20:02:43,051 >> Model weights saved in ./checkpoint-1000/pytorch_model.bin
[INFO|feature_extraction_utils.py:369] 2023-05-06 20:02:43,056 >> Feature extractor saved in ./checkpoint-1000/preprocessor_config.json
[INFO|feature_extraction_utils.py:369] 2023-05-06 20:02:51,152 >> Feature extractor saved in ./preprocessor_config.json
Traceback (most recent call last):
  File "/home/local/QCRI/dizham/kanari/whisper/whisper-small-ar/run_speech_recognition_seq2seq_streaming.py", line 629, in <module>
    main()
  File "/home/local/QCRI/dizham/kanari/whisper/whisper-small-ar/run_speech_recognition_seq2seq_streaming.py", line 578, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 1664, in train
    return inner_training_loop(
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 2011, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 2300, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 2444, in _save_checkpoint
    self._push_from_checkpoint(output_dir)
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/transformers/trainer.py", line 3622, in _push_from_checkpoint
    _, self.push_in_progress = self.repo.push_to_hub(
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 1305, in push_to_hub
    self.git_add(auto_lfs_track=True)
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 1009, in git_add
    tracked_files.extend(self.auto_track_binary_files(pattern))
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 903, in auto_track_binary_files
    is_binary = is_binary_file(path_to_file)
  File "/home/local/QCRI/dizham/miniconda3/envs/whisper/lib/python3.9/site-packages/huggingface_hub/repository.py", line 230, in is_binary_file
    with open(filename, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/home/local/QCRI/dizham/kanari/whisper/whisper-small-ar/./wandb/latest-run'