Unable to Get Decoded Output from Whisper
Author: as1078
Pull request: no

### System Info

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.2
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.11 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: Yes (nvidia a100-sxm4-40gb)
- Using distributed or parallel set-up in script?: parallel

### Who can help?

speech model: @sanchit-gandhi 
tokenizer: @ArthurZucker 
trainer: @sgugger 
PyTorch: @sgugger 

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

All preprocessing steps for the data were the same as the following notebook: https://colab.research.google.com/github/sanchit-gandhi/notebooks/blob/main/fine_tune_whisper.ipynb.
Training the data was able to yield results with proper metrics for WER, but using `trainer.evaluate()` led to an error as transcripts were unable to be generated.

```
dataset = dataset_dict['train']
dataset = dataset.train_test_split(test_size=0.25)
print(dataset)

DatasetDict({
    train: Dataset({
        features: ['audio', 'sentence'],
        num_rows: 1750
    })
    test: Dataset({
        features: ['audio', 'sentence'],
        num_rows: 584
    })
})

from transformers import Seq2SeqTrainingArguments

training_args = Seq2SeqTrainingArguments(
    output_dir="./whisper-small-hi",  # change to a repo name of your choice
    per_device_train_batch_size=16,
    gradient_accumulation_steps=1,  # increase by 2x for every 2x decrease in batch size
    learning_rate=1e-5,
    warmup_steps=500,
    max_steps=4000,
    gradient_checkpointing=True,
    fp16=True,
    evaluation_strategy="steps",
    per_device_eval_batch_size=8,
    predict_with_generate=True,
    generation_max_length=225,
    save_steps=1000,
    eval_steps=1000,
    logging_steps=25,
    report_to=["tensorboard"],
    load_best_model_at_end=True,
    metric_for_best_model="wer",
    greater_is_better=False,
    push_to_hub=True,
)

from transformers import Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=common_voice['train'],
    eval_dataset=common_voice['test'],
    data_collator=data_collator,
    tokenizer=processor.feature_extractor,
    compute_metrics=compute_metrics,
)

trainer.evaluate()

in <cell line: 1>:1                                                                              │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:2945 in evaluate                 │
│                                                                                                  │
│   2942 │   │   start_time = time.time()                                                          │
│   2943 │   │                                                                                     │
│   2944 │   │   eval_loop = self.prediction_loop if self.args.use_legacy_prediction_loop else se  │
│ ❱ 2945 │   │   output = eval_loop(                                                               │
│   2946 │   │   │   eval_dataloader,                                                              │
│   2947 │   │   │   description="Evaluation",                                                     │
│   2948 │   │   │   # No point gathering the predictions if there are no metrics, otherwise we d  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/trainer.py:3227 in evaluation_loop          │
│                                                                                                  │
│   3224 │   │   │   │   │   EvalPrediction(predictions=all_preds, label_ids=all_labels, inputs=a  │
│   3225 │   │   │   │   )                                                                         │
│   3226 │   │   │   else:                                                                         │
│ ❱ 3227 │   │   │   │   metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, lab  │
│   3228 │   │   else:                                                                             │
│   3229 │   │   │   metrics = {}                                                                  │
│   3230                                                                                           │
│ in compute_metrics:13                                                                            │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:3490 in          │
│ batch_decode                                                                                     │
│                                                                                                  │
│   3487 │   │   Returns:                                                                          │
│   3488 │   │   │   `List[str]`: The list of decoded sentences.                                   │
│   3489 │   │   """                                                                               │
│ ❱ 3490 │   │   return [                                                                          │
│   3491 │   │   │   self.decode(                                                                  │
│   3492 │   │   │   │   seq,                                                                      │
│   3493 │   │   │   │   skip_special_tokens=skip_special_tokens,                                  │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:3491 in          │
│ <listcomp>                                                                                       │
│                                                                                                  │
│   3488 │   │   │   `List[str]`: The list of decoded sentences.                                   │
│   3489 │   │   """                                                                               │
│   3490 │   │   return [                                                                          │
│ ❱ 3491 │   │   │   self.decode(                                                                  │
│   3492 │   │   │   │   seq,                                                                      │
│   3493 │   │   │   │   skip_special_tokens=skip_special_tokens,                                  │
│   3494 │   │   │   │   clean_up_tokenization_spaces=clean_up_tokenization_spaces,                │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/whisper/tokenization_whisper.py:592  │
│ in decode                                                                                        │
│                                                                                                  │
│    589 │   │   Returns:                                                                          │
│    590 │   │   │   `str`: The decoded sentence.                                                  │
│    591 │   │   """                                                                               │
│ ❱  592 │   │   text = super().decode(                                                            │
│    593 │   │   │   token_ids,                                                                    │
│    594 │   │   │   skip_special_tokens=skip_special_tokens,                                      │
│    595 │   │   │   clean_up_tokenization_spaces=clean_up_tokenization_spaces,                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py:3530 in decode   │
│                                                                                                  │
│   3527 │   │   # Convert inputs to python lists                                                  │
│   3528 │   │   token_ids = to_py_obj(token_ids)                                                  │
│   3529 │   │                                                                                     │
│ ❱ 3530 │   │   return self._decode(                                                              │
│   3531 │   │   │   token_ids=token_ids,                                                          │
│   3532 │   │   │   skip_special_tokens=skip_special_tokens,                                      │
│   3533 │   │   │   clean_up_tokenization_spaces=clean_up_tokenization_spaces,                    │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/models/whisper/tokenization_whisper.py:619  │
│ in _decode                                                                                       │
│                                                                                                  │
│    616 │   │   │   decoder_start_token_id = self.convert_tokens_to_ids("
")  │
│    617 │   │   │   token_ids = self._strip_prompt(token_ids, prompt_token_id, decoder_start_tok  │
│    618 │   │                                                                                     │
│ ❱  619 │   │   filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip  │
│    620 │   │                                                                                     │
│    621 │   │   # To avoid mixing byte-level and unicode for byte-level BPT                       │
│    622 │   │   # we need to build string separately for added tokens and byte-level tokens       │
│                                                                                                  │
│ /usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py:906 in                │
│ convert_ids_to_tokens                                                                            │
│                                                                                                  │
│   903 │   │   │   │   return self._convert_id_to_token(ids)                                      │
│   904 │   │   tokens = []                                                                        │
│   905 │   │   for index in ids:                                                                  │
│ ❱ 906 │   │   │   index = int(index)                                                             │
│   907 │   │   │   if skip_special_tokens and index in self.all_special_ids:                      │
│   908 │   │   │   │   continue                                                                   │
│   909 │   │   │   if index in self.added_tokens_decoder:                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: int() argument must be a string, a bytes-like object or a real number, not 'list'


```

### Expected behavior

I would expect `trainer.evaluate()` to return proper metrics (validation loss and WER) along with generated transcripts for each of the samples fed into the Whisper model.

Comments:
Hey @as1078 - did you pre-process your inputs according to the function `prepare_dataset`? See https://huggingface.co/blog/fine-tune-whisper#prepare-data

I observe that your dataset has two columns present: `audio` and `sentence`. These are both columns corresponding to raw audio input data and raw target text data. As explained in the blog post / Colab, we need to pre-process the (audio, text) data to (log-mel spectrograms, token ids) respectively.

You should be able to run evaluation simply by pre-processing your dataset as-per the instructions provided and then passing the pre-processed dataset to the `trainer`.

If you want a more streamlined version of a Whisper evaluation script, I recommend you check out: https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event#evaluation

You should just be able to specify your model id and dataset metadata and run evaluation directly

Yes, the data was preprocessed according to the `prepare_dataset` function. However, `trainer.evaluate()` still gave those errors. If I am running the script that you sent a link to, does the model checkpoint need to have been saved after training? Also, I have limited GPU access even with Google Colab Pro, so would saving checkpoints be a better way to save computational resources?

Hey @as1078 - could you provide an end-to-end reproducible code snippet to run your script? It would be helpful in checking that all the pre-processing steps have been applied correctly (the fact that we're seeing `audio` and `sentence` in your dataset means something has gone wrong!)

The script can either use a `model_id` for a checkpoint on the Hub (e.g. `"openai/whisper-small"` for the pre-trained small Whisper checkpoint, or the path to a locally saved checkpoint (e.g. if you set your save directory to `./my-model`, set `model_id=./my-model` in the training arguments)

Yes, I can. Here is the link to my colab file: https://colab.research.google.com/drive/10NaxWZtQgaYMN2fTnbRNNqV2baGkJVGG?usp=sharing. 
The data directory is linked here: https://drive.google.com/drive/folders/1-3WqzbKH4ZFUm0r2rYwi2f7Wyw64bYa1?usp=sharing. Here is the link to the CSV file with the data files listed:
[audio_new.csv](https://github.com/huggingface/transformers/files/12023463/audio_new.csv)


Hey @as1078 - thanks for sharing your script. It looks largely correct, but I can't run it since the data is saved in your Google Drive, so I can't link it to my Colab runtime without downloading it all. Could you perhaps load your dataset, and then push it to the Hub with:
```python
common_voice.push_to_hub("stuttering_asr")
```

This will create the dataset under your namespace, which will then allow me to run your script by streaming the data from the Hub.

If you're just interested in evaluation, there's a lightweight script [here](https://github.com/huggingface/community-events/tree/main/whisper-fine-tuning-event#evaluation) that you can use that will do all the pre-processing for you and not require the HF Trainer.

Yes, of course. The notebook has been updated with this: [https://colab.research.google.com/drive/10NaxWZtQgaYMN2fTnbRNNqV2baGkJVGG?usp=sharing](url). I can also try the evaluation script, but it would be great if you could take a look at the data and give me any feedback. Thanks so much for the help.

Hey @as1078 - I'm still not able to reproduce your script unfortunately. The dataset that you have pushed contains an `audio` column that is the absolute **path** to a local audio file, rather than an audio file itself. See the dataset viewer to inspect the first 100 examples: https://huggingface.co/datasets/amansingh203/stuttering_asr_dataset/viewer/amansingh203--stuttering_asr_dataset/train?row=0

Could you first load your dataset as an audio dataset, and then push it to the Hub? This way, the audio files will be pushed, and subsequently I'll be able to load them locally. You can follow these steps for doing so: https://huggingface.co/docs/datasets/audio_load

Once you've done this, simply push to Hub:
```python
dataset.push_to_hub("stuttering_asr")
```

Hi @sanchit-gandhi . The data was loaded into my Hugging Face data account (I pushed it to the Hub), where the audio is now stored. Let me know if you have any issues accessing it. I was able to resolve the issues when I ran `trainer.evaluate()` (I was not using a `Sequence2SequenceTrainer`). However, when I generate transcripts, some of them are not in English, even though the tokenizer is set to transcribe English. Was wondering if this was an issue with the code, or if the model needs more epochs to run.

Hey @as1078 - nice work on figuring out the issue!

> when I generate transcripts, some of them are not in English

Since you're fine-tuning on an English-only dataset, it makes sense to use an English-only checkpoint as your starting point. See the table [here](https://huggingface.co/blog/fine-tune-whisper#introduction) for details. If doing this, ensure that you **do not** specify the language or task arguments to the tokenizer and processor - these are not required for English-only fine-tuning!

In short, you can swap `openai/whisper-small` for `openai/whisper-small.en` everywhere in your script, and remove all the `language` and `task` arguments

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Going to close this one as the original issue has been resolved, and a fix for the follow-up issue proposed in my last comment. Feel free to re-open this issue if you have a problem with the Trainer @as1078, or open a new issue if your transcriptions are not looking accurate (e.g. with the language).