Whisper model predicts "thank  you" or "you" on silence
Author: mirfan899
Pull request: no

### System Info

wave2vec2 models predict text even there is no speech. The model predicts "thank you" and "you" on silence or empty speech.

Python 3.10
Ubuntu 20.04
transformers==4.30.2

https://github.com/gradio-app/gradio/issues/4663#issue-1772508542

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Follow these steps:

1. install gradio and transformers.
2. use automatic speech recognition pipeline and wave2vec2 base model.
3. use a microphone and don't speak. 

It will keep predicting words on an empty stream. 

Code to reproduce it.
```python
from transformers import pipeline
import gradio as gr

p = pipeline("automatic-speech-recognition", model="openai/whisper-base")

def transcribe(audio, state=""):
    text = p(audio)["text"]
    state += text + " "
    return state, state


# Set the starting state to an empty string
gr.Interface(
    fn=transcribe,
    inputs=[
        gr.Audio(source="microphone", type="filepath", streaming=True),
        "state"
    ],
    outputs=[
        "textbox",
        "state"
    ],
    live=True).launch(share=True)
```

### Expected behavior

It should not predict words on an empty stream.

Comments:
Hmm, this code snippet involves `gradio` which is usually out of the scope of the `transformers` GitHub issue pages. But trying to have a reproducible example with only `transformers` involved seems tricky in this case.

@sgugger @amyeroberts @ArthurZucker Any general comment on how we proceed in such cases?



Hey! 
Gradio can be interfering indeed, especially given the `streaming = True`. A few questions would help:
- does this happen on silence for  multiple audio? 
- was "thank you"  said before? Seems like the example
- Can you provide an example audio file when this happens? 
- Did you try without gradio (just the pipeline on the audio)? 
This would help us a lot if you can have these informations! 

yes
no
audios (silence, noise, recording)[audios.zip](https://github.com/huggingface/transformers/files/11882755/audios.zip)
Yes.

```python
p = pipeline("automatic-speech-recognition", model="openai/whisper-base")

p("silence.wav")['text']
/home/irfan/.pyenv/versions/3.10.10/envs/WhisperDemo/lib/python3.10/site-packages/transformers/generation/utils.py:1353: UserWarning: Using `max_length`'s default (448) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.
  warnings.warn(
' you'
```

Same issue, using gradio and non-gradio wav sound sources. I've also seen the behavior in [Unity Undertone](https://leastsquares.io/docs/unity/undertone) , a Whisper package for Unity 3D. So it may be in Whisper and not the ASR pipeline. Maybe a few more switches to control returned info might help.
[Whisper: Decode with condition_on_previous_text=False](https://github.com/huggingface/transformers/issues/21467#top)](https://github.com/huggingface/transformers/issues/21467)
[A possible solution to Whisper hallucination](https://github.com/openai/whisper/discussions/679)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Pretty sure this is whisper hallucinating with the given input. Let's close this for now as it does not appear to be a bug with the whisper model implementation / pipeline.

@ArthurZucker I was guessing the same, but how do you prevent it? I was thinking of listening with another tool to the audio and if it is below certain volume (Db) ignore it. Is there a way to do it with out audio processing?