Whisper transcription accuracy improves when last 1600 samples of input audio are muted
Author: jozefchutka
Pull request: no
Labels: name: bug

name: Audio

name: Whisper

`### System Info

transcformers version: 4.55.0

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When using a Whisper model in transformers, the transcription differs significantly depending on whether the last 1600 samples (~0.1s at 16 kHz) of the input audio are muted (set to 0.0).

With the original audio, some content is missing in the transcript. When last 1600 samples are muted, the transcript is complete and accurate.

This suggests that Whisper in transformers might mishandle small trailing noise or non-speech at the very end of an audio file.

```py
import torch
import numpy as np
from transformers import pipeline

asr = pipeline(
    task="automatic-speech-recognition",
    model="openai/whisper-base",
    torch_dtype=torch.float32,
    device=-1
)

audio = np.fromfile("ch.pcm", dtype=np.float32)
#audio[-1600:] = 0.0

result = asr(
    audio,
    generate_kwargs={"language": "en"}
)

print(result)
```

Transcript with the original audio:

```
 The schoolbooks say it can't be here again chocolate rain.
```

Transcript when muting last samples `audio[-1600:] = 0.0` : 
```
 Chocolate Rain Some stay dry and others feel a pain chocolate rain Our baby porn will die before the same chocolate rain The school books say it can't 
be here again chocolate rain The prison
```

I am using first 30 seconds of the following audio as input https://www.youtube.com/watch?v=EwTZ2xpQwpA , extracting as pcm using ffmpeg (attached [ch.zip](https://github.com/user-attachments/files/21696489/ch.zip) ) :

```sh
ffmpeg -ss 0 -t 30 -i ch.mp4 -filter_complex:a "[0:1]aformat=channel_layouts=mono,aresample=16000[aout]" -map "[aout]" -c:a pcm_f32le -f data ch.pcm -y
```

### Expected behavior

Small trailing silence/noise at the end of the audio should not cause the model to omit words from the transcript.



Comments:
maybe @eustlb or @vasqu can help?

Thanks @jozefchutka, can you confirm this differs from the behavior you get using the original whisper implementation?

For visibility: #40704 is not a fix. I’ll dig into this if it’s confirmed that the problem comes from our implementation and not the model itself 😊

Hi @eustlb please have a look here https://github.com/huggingface/transformers.js/issues/1358#issuecomment-3084350855 where there is output from original/python lib for comparision. ~~Please consider if these two tasks can be merged together - the other one contains more info.~~

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I think this still needs to be addressed

@jozefchutka the comment you've pinged above uses `transformers` so I am not sure to get what you mean. 
What is needed before digging in this issue is for you to confirm that you get a different output using [the original whisper inference library](https://github.com/openai/whisper) please 🙏

@eustlb sorry for confusion. I am not experienced original whisper lib user, but tried following steps: 

```sh
./venv/bin/pip install openai-whisper
ffmpeg -f f32le -ar 16000 -ac 1 -i ch.pcm ch.wav # using ch.zip asset uploaded in this issue
whisper ch.wav --model base --language en --verbose False --output_format txt
```

The output ch.txt contains:

```
Music
Chocolate rain
Some stay dry and others feel the pain chocolate rain
Our baby born will die before the same chocolate rain
The schoolbooks say it can't be here again chocolate rain
The prison
```

compared to the `transformers` output (as originally reported):

```
 The schoolbooks say it can't be here again chocolate rain.
```

This seems to confirm the output does differ.

Okay issue confirmed then, thanks! looking into it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Please keep open, this still needs to be addressed

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Please keep open, this still needs to be addressed

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Please keep open, this still needs to be addressed

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.