Whisper breaks on poor quality speech audio
Author: frankiedrake
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.19.0-32-generic-x86_64-with-glibc2.17
- Python version: 3.8.16
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I don't know if it's a bug, but it's definitely not an expected behaviour for me.
Also I saw a thread with such behaviour where @Narsil said that "the model is repeating itself", but I can't find it right now, I'll update the issue when I do.

To recognize audio file I'm using script that I found in one of the threads here on github [link](https://colab.research.google.com/drive/1Qz9hUL3Z3SxHLUt7f4vuzZG7KEgV0ofk?usp=share_link)
```
processor = WhisperProcessor.from_pretrained("openai/whisper-small")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-small")
model.config.forced_decoder_ids = processor.get_decoder_prompt_ids(task="transcribe")

input_speech, sr = audio2numpy.audio_from_file(file)

input_features = processor(input_speech, return_tensors="pt", sampling_rate=16000).input_features
predicted_ids = model.generate(input_features, max_length=model.config.max_length, repetition_penalty=1)
transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)
```
And it works okay if the speech is clear and the utterance is detected good, but when speaker talks fast or not legibly enough, or if there's little silence in audio, then the transcription becomes ugly. 
It looks like:
"Привіт, хороша погода але але але але але але але але але але але але але але але але"
Currently I'm using only ukrainian files so I'm not aware if it happens in other languages.

### Expected behavior

The text is recognized along the whole audio file without breaking

Comments:
cc @ArthurZucker and @sanchit-gandhi 

Hey! Did you get similar results using the original `Whisper` model? Most probably this is just the model not being very good. 

> Hey! Did you get similar results using the original `Whisper` model? Most probably this is just the model not being very good.

No, I don't face such problem with original `Whisper`

OK! There can be a few things to try, but if you can share an audio file with me it would help a lot! 

@frankiedrake Have you tried using `return_timestamps` ? When experimenting I found that the default for openai is using timestamps (even if not shown) and that the model seems to perform better than without.

> 

Sorry for a noob question, where should I put this parameter? I tried to add `no_timestamps=False` to `processor.get_decoder_prompt_ids` call and also to the three other methods call, but result didn't change.

Btw it works well with `medium` model, but it's too heavy for my setup

> OK! There can be a few things to try, but if you can share an audio file with me it would help a lot!

Unfortunately, I cannot share it because of NDA, but I'll try to record one myself

`generate(..., return_timestamps=True)` to get the proper generation mode.
That's correct right @ArthurZucker 

> return_timestamps

I tried this, but got an error, that model's `forward` method isn't aware of this parameter

```ValueError: The following `model_kwargs` are not used by the model: ['return_timestamps'] (note: typos in the generate arguments will also show up in this list)```

`pipe = pipeline(..., return_timestamps=True)`  should work though.

> @frankiedrake Have you tried using `return_timestamps` ? When experimenting I found that the default for openai is using timestamps (even if not shown) and that the model seems to perform better than without.

I tested all variants, result is the same

Without being able to reproduce it's really hard. Could you dive to the level of logits and figure out any potential differences ?

I'm pretty sure it should come down to a configuration difference in the end, but if *we* can't reproduce, it'd be hard to understand. 

> Without being able to reproduce it's really hard. Could you dive to the level of logits and figure out any potential differences ?
> 
> I'm pretty sure it should come down to a configuration difference in the end, but if _we_ can't reproduce, it'd be hard to understand.

Okay, thank you for suggestions, I'll try to look at it, and also will try to find an audio I could share with you

You are probably not using `main`! The timestamps were not part of the initial release 😉 

@ArthurZucker Correct! I installed the library from main branch and the transcription is now better with one file, but remains the same with another, probably I can tune the parameters further to make it work even better

Glad that I could help 😉 

> OK! There can be a few things to try, but if you can share an audio file with me it would help a lot!

Guys, I was able to record an [audio](https://github.com/frankiedrake/demo/blob/master/whisper_test.wav) that reproduces a problem, but it wasn't so easy, so the text I recorded doesn't actually exists 😁, anyway I'd expect the model gives some text instead of repeating a single word multiple times.
Also, I noticed that if I don't specify language manually the model seems not to break, in my case it detected Chinese language and emitted some readable text.
Output of Ukrainian:
`Вереси, єдине м'якості, єдине ісцемо, і два, і бере, і бере, і бере, і бере, і бере, і бере, і бере, і бере, і бере, і бере.`

The original whisper produces:
`Веріості, є рівні, є мільйорівні, є сьогодні, є два, є бере, є бере, чіжено, чіжено, нервно, нервно.`
 

> f I don't specify language manually the model seems not to break

Can you share how you specify language in both `openai` and `transformers` ? The difference is likely coming from there.




> > f I don't specify language manually the model seems not to break
> 
> Can you share how you specify language in both `openai` and `transformers` ? The difference is likely coming from there.

I really doubt that it's related because it reproduces even if I don't specify the language in other audios

Seems like it's a known [issue](https://github.com/microsoft/DialoGPT/issues/45#issuecomment-680087019) with transformers models, and tuning `temperature` and `repetition_penalty` params of `generate` method helps, I was able to stop the model from repeating a single word, text now looks much much better, but seems I noticed a small cut at the end of the audio, will investigate a bit deeper. I read about `repetition_penalty` before but I didn't succeed, maybe the conjunction with the `temperature` is significant. 
Maybe you can shed some light on why these parameters are so important?

>  Maybe you can shed some light on why these parameters are so important?

LMs are know to hallucinate by repeating tokens, or several tokens. I don't think there's a good consensus on why it's the case, but it's a very well know issue on LMs. Adding penalties does help, but it's a clutch in my very personal opinion.

Maybe the openai defaults are still different the ones we have


Do we agree the openai test you're doing is simply 

```
whisper whisper_test.wav --model small
```

And get this

```
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: Chinese
[00:00.420 --> 00:02.000] 我問他們
[00:02.000 --> 00:03.320] 他們在那邊
[00:03.320 --> 00:05.280] 順便他們等著過去
[00:05.280 --> 00:06.980] 還要去
[00:06.980 --> 00:08.260] 剛剛說他們背後
```
 
right ?


I have the following result
```
Detecting language using up to the first 30 seconds. Use `--language` to specify the language
Detected language: Chinese
[00:00.000 --> 00:01.700] 要生一年年老了
[00:01.700 --> 00:03.280] 一生不一年老
[00:03.480 --> 00:05.960] 練一陣子
[00:05.960 --> 00:06.700] 熱情不二
[00:06.700 --> 00:07.620] 謝謝大家
```


Can you share anything reproducible ? Right now it's a back and forth and we can't reproduce anything on our end.

Please share a clear (small) script, that I can copy past that should reproduce the issue on `transformers@main` and `whisper@main` otherwise, it's going to be too tedious for us to investigate.

> Can you share anything reproducible ? Right now it's a back and forth and we can't reproduce anything on our end.
> 
> Please share a clear (small) script, that I can copy past that should reproduce the issue on `transformers@main` and `whisper@main` otherwise, it's going to be too tedious for us to investigate.

But I shared a file with you, what exactly you can't reproduce?

I don't have the same output for neither `openai/whisper` nor `transformers` on your file.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.