Whisper not returning last phrase from audio >25s
Author: pli66
Pull request: no

### System Info

- `transformers` version: 4.30.2
- Platform: Linux-5.4.226-129.415.amzn2.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.13
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes, don't know why pytorch was not detected as it is using it for sure
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi

### Information

- [X] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Whisper consistently does not return the last phrase in a transcription for clips >25s. 
"https://github.com/pli66/test/raw/main/personage_hardy_ah_64kbTrimmed0.mp3" is a 30s clip
"https://github.com/pli66/test/raw/main/personage_hardy_ah_64kbTrimmed7.mp3" is the first half of above clip, 15s.
"https://github.com/pli66/test/raw/main/personage_hardy_ah_64kbTrimmed8.mp3" is the second half of above clip, 15s.


```
import whisper
import torch
import tempfile
import requests
import os

from transformers import WhisperProcessor, WhisperForConditionalGeneration, WhisperTokenizer
from datasets import load_dataset, Dataset, Audio

VERSION = "medium"

def download_file(url):
    """ Download a file frome a url and save it to a named temporary file """
    response = requests.get(url)
    temp = tempfile.NamedTemporaryFile(delete=True, dir=os.getcwd(), mode='w+b')
    temp.write(response.content)
    temp.seek(0)
    return temp

urls = ["https://github.com/pli66/test/raw/main/personage_hardy_ah_64kbTrimmed0.mp3",
        "https://github.com/pli66/test/raw/main/personage_hardy_ah_64kbTrimmed7.mp3",
        "https://github.com/pli66/test/raw/main/personage_hardy_ah_64kbTrimmed8.mp3"]

processor = WhisperProcessor.from_pretrained(f"openai/whisper-{VERSION}")
generator = WhisperForConditionalGeneration.from_pretrained(f"openai/whisper-{VERSION}").to("cuda")
generator.eval()
tokenizer = WhisperTokenizer.from_pretrained(f"openai/whisper-{VERSION}")

for url in urls:
    with download_file(url) as f:
        audio = whisper.load_audio(f.name)

    processed = processor(audio, sampling_rate=16000, return_tensors="pt", return_attention_mask=True).to("cuda")

    generated = generator.generate(processed.input_features, attention_mask=processed.attention_mask, return_timestamps=True, output_scores=True, return_dict_in_generate=True, temperature=0, repetition_penalty=8.0, num_beams=2)

    predicted_ids = generated[0].to("cpu")
    batch_transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False, decode_with_timestamps=True, output_offsets=True)

    print(batch_transcription)
```

[{'text': '


<|0.00|> A Popular Personage at Home by Thomas Hardy.<|12.36|><|12.36|> I live here, Wessex is my name—I am a dog known rather well.<|19.12|><|19.12|> I guard the house, but how that climb to be my whim I cannot tell.<|25.40|><|25.40|>
', 'offsets': [{'text': ' A Popular Personage at Home by Thomas Hardy.', 'timestamp': (0.0, 12.36)}, {'text': ' I live here, Wessex is my name—I am a dog known rather well.', 'timestamp': (12.36, 19.12)}, {'text': ' I guard the house, but how that climb to be my whim I cannot tell.', 'timestamp': (19.12, 25.400000000000002)}]}]

[{'text': '


<|0.00|> A Popular Personage at Home by Thomas Hardy.<|4.76|><|4.76|> Read for LibriVox.org by Anita Hibbard, January 23rd, 2023.<|11.84|><|11.84|> I live here.<|14.04|><|14.04|> Essex is my name.<|15.04|>
', 'offsets': [{'text': ' A Popular Personage at Home by Thomas Hardy.', 'timestamp': (0.0, 4.76)}, {'text': ' Read for LibriVox.org by Anita Hibbard, January 23rd, 2023.', 'timestamp': (4.76, 11.84)}, {'text': ' I live here.', 'timestamp': (11.84, 14.040000000000001)}, {'text': ' Essex is my name.', 'timestamp': (14.040000000000001, 15.040000000000001)}]}]

[{'text': '


<|0.00|> name. I am a dog known rather well. I guard the house, but how that climb to be my whim<|7.76|>**<|7.76|> I cannot tell. With a leap and a heart elate I go at the end of an hour\'s expectancy."<|15.04|>
**', 'offsets': [{'text': ' name. I am a dog known rather well. I guard the house, but how that climb to be my whim', 'timestamp': (0.0, 7.76)}, {'text': ' I cannot tell. With a leap and a heart elate I go at the end of an hour\'s expectancy."', 'timestamp': (7.76, 15.040000000000001)}]}]



Observe that the last phrase is caught by the second half 15s clip but not the 30s one.
This behavior is also observed in pipeline. (The first half 15s clip is repeating oddly without the beam search but that is a different issue).


```
from transformers import pipeline
pipe = pipeline(task="automatic-speech-recognition", model="openai/whisper-medium")
for url in urls:
    print(pipe(url, return_timestamps=True))
```

{'text': ' A Popular Personage at Home by Thomas Hardy. I live here. Wessex is my name. I am a dog known rather well. I guard the house, but how that climb to be my whim I cannot tell.', 'chunks': [{'timestamp': (0.0, 12.36), 'text': ' A Popular Personage at Home by Thomas Hardy.'}, {'timestamp': (12.36, 14.08), 'text': ' I live here.'}, {'timestamp': (14.08, 16.2), 'text': ' Wessex is my name.'}, {'timestamp': (16.2, 19.12), 'text': ' I am a dog known rather well.'}, {'timestamp': (19.12, 25.4), 'text': ' I guard the house, but how that climb to be my whim I cannot tell.'}]}

{'text': ' A Popular Personage at Home by Thomas Hardy. Read for LibriVox.org by Anita Hibbard. January 23, 2023. I live here. Essex is my name. I live here. Essex is my name. I live here. Essex is my name. I live here. Essex is my name. I live here. Essex is my name. I live here. Essex is my name. I live here. Essex is my name. I live here. Essex is my name.', 'chunks': [{'timestamp': (0.0, 4.76), 'text': ' A Popular Personage at Home by Thomas Hardy.'}, {'timestamp': (4.76, 8.68), 'text': ' Read for LibriVox.org by Anita Hibbard.'}, {'timestamp': (8.68, 11.24), 'text': ' January 23, 2023.'}, {'timestamp': (11.24, 14.04), 'text': ' I live here.'}, {'timestamp': (14.04, 15.04), 'text': ' Essex is my name.'}, {'timestamp': (15.04, 16.04), 'text': ' I live here.'}, {'timestamp': (16.04, 17.04), 'text': ' Essex is my name.'}, {'timestamp': (17.04, 18.04), 'text': ' I live here.'}, {'timestamp': (18.04, 19.04), 'text': ' Essex is my name.'}, {'timestamp': (19.04, 20.04), 'text': ' I live here.'}, {'timestamp': (20.04, 21.04), 'text': ' Essex is my name.'}, {'timestamp': (21.04, 22.04), 'text': ' I live here.'}, {'timestamp': (22.04, 23.04), 'text': ' Essex is my name.'}, {'timestamp': (23.04, 24.04), 'text': ' I live here.'}, {'timestamp': (24.04, 25.04), 'text': ' Essex is my name.'}, {'timestamp': (25.04, 26.04), 'text': ' I live here.'}, {'timestamp': (26.04, 27.04), 'text': ' Essex is my name.'}, {'timestamp': (27.04, 28.04), 'text': ' I live here.'}, {'timestamp': (28.04, 29.04), 'text': ' Essex is my name.'}]}

{'text': " name. I am a dog known rather well. I guard the house, but how that climb to be my whim I cannot tell. With a leap and a heart elate I go, at the end of an hour's expectancy.", 'chunks': [{'timestamp': (0.0, 7.76), 'text': ' name. I am a dog known rather well. I guard the house, but how that climb to be my whim'}, {'timestamp': (7.76, 15.04), 'text': " I cannot tell. With a leap and a heart elate I go, at the end of an hour's expectancy."}]}



None of the configuration settings I change seem to affect this, including setting early_stopping to "never". This also seems to be a different issue to https://github.com/huggingface/transformers/issues/23231 as the last segment does not show up in the regular decoding output or the offsets. 

### Expected behavior

The last segment should be returned for 30s clips. A workaround would be splitting >25s clips in half but that is not so ideal since it impacts both performance and accuracy. 

Comments:
Thanks for the examples - they suggest that the model is predicting the EOS token too early within the 30s segment. Having listened to the audio, I think this is because the last sentence does not finish completely within the 30s window, so the Whisper model decides to ignore it.

What we can do is feed a larger context length to the model, so it can handle audios of arbitrary length. The last sentence now finishes within a context window, so Whisper now predicts the EOS token after this sentence. Let's also use the English variant of the medium checkpoint if working with the English language (better performance vs the multilingual one for English):
```python
from transformers import pipeline
import torch

url = "https://github.com/pli66/test/raw/main/personage_hardy_ah_64kbTrimmed0.mp3"

device = "cuda:0" if torch.cuda.is_available() else "cpu"
pipe = pipeline(task="automatic-speech-recognition", model="openai/whisper-medium.en", device=device)

print(pipe(url, return_timestamps=True, chunk_length_s=30.0))
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Should be fixed by the above comment. Feel free to re-open @pli66 if you're still experiencing issues!