[Whisper] Fix whisper integration tests
Author: eustlb
Pull request: yes
Labels: name: run-slow

# What does this PR do?

This PR fixes multiple errors in Whisper integration tests and expected outputs. 

To compute the correct excepted outputs, it is necessary to work from a [very simple fork of the original OpenAI Whisper implementation](https://github.com/eustlb/whisper/tree/transcribe-from-mel-spectrogram). Indeed, the extraction of the mel spectrogram in `WhisperFeatureExtractor` diverges slightly from OpenAI's one: we pad the audio array to 30sec/ to longest with 0.0s and then extract our spectrogram through batched STFT while OpenAI's one will add 30sec of 0.0s to the audio array (and not pad _to_ 30sec). This way, the are sure that model inputs for our and OpenAI's implementations are exactly the same. 

With this, we can use the following protocol to compute the expected outputs for the tests:
1. extract mel inputs using the test's implementation (so using the WhisperFeatureExtractor)
2. infer OpenAI's model through the above explained fork directly from the mel input

> [!IMPORTANT] 
> Code to reproduce the outputs for each of the verified tests can be found [here](https://github.com/eustlb/reproduce-whisper-expected-outputs). 

# Edit: some more details about why we work from a whisper fork 

In Transformers, we have two inputs possibilities for Whisper:
1. mel spectrogram with 3000 frames →  audio is first padded **to** 30sec with 0.0s and then we extract the mel
2. mel spectrogram with more than 3000 frames →  no need for padding 

--- 
**case 1**
With an audio <=30sec, the difference between our implementation and OAI is that we first pad to 30sec with 0.0s, then extract features and this will be the input to the model's forward, while OAI pads audio with adding 30sec 0.0s, extract features, slice the exact number of frames and then pads the mel spectrogram to 3000 frames with 0.0s.

To understand better, for an audio of 10secs:
**Transformers**: audio + 20sec of 0.0s →  mel spectrogram of shape [80, 3000] where **[2000:] frames are close but not exactly 0.0s** 
**OAI**: audio + 30sec of 0.0s →  mel spectrogram of shape [80, 4000] → sliced to the duration of the audio (so until frame 1000) and then padded with 0.0s: **[2000:] frames are exactly 0.0s.**

--- 
**case 2**
No differences (other than numerical difference due to STFT implementation).

---

About the implementation in the [simple whisper fork](https://github.com/eustlb/whisper/tree/transcribe-from-mel-spectrogram):
We just take the mel spectrogram and concat with 3000 frames of 0.0s. This emulates the 30sec of 0.0s added originally. 
For case 1, the duration considered by OAI is 30sec (see [this line](https://github.com/openai/whisper/blob/25639fc17ddc013d56c594bfbf7644f2185fad84/whisper/transcribe.py#L134C9-L134C46)) and therefore the audio segment that will be given to the forward is the exact mel input that was given. 
For case 2, likewise the duration considered is the one of the given mel input. 

With inferring OAI directly on the mel spectrogram (so either of exactly 3000 frames, either on more than 3000 frames), we ensure that each pass of the forward of OAI whisper and our gets the exact same mel spectrogram. This ensures that the expected result we have in the test are indeed results that should be expected given the same input mel with OAI implementation.

> [!NOTE]  
> For tests that required batched inference which is not supported by OAI implementation, I simply run it sequentially to get the outputs 

# TODO

## Tests to be verified and eventually corrected 
✅ for a correct test
❌ for an incorrect one

- [x] **test_tiny_timestamp_generation** ❌
- [x] **test_large_timestamp_generation** ❌
- [x] **test_default_multilingual_transcription_short_form** ✅
- [x] **test_default_multilingual_transcription_long_form** ❌ →  here we wan't the model to return input token ids and eos  
- [x] **test_whisper_shortform_single_batch_prev_cond** ❌
- [x] **test_whisper_shortform_multi_batch_hard_prev_cond** ❌
- [x] **test_whisper_longform_single_batch** ✅
- [x] **test_whisper_longform_prompt_ids** ✅ →  partially verified, `"all-segments"` has no equivalent in OAI implem
- [x] **test_whisper_longform_multi_batch** ✅
- [x] **test_whisper_longform_single_batch_prev_cond** ✅
- [x] **test_whisper_longform_multi_batch_prev_cond** ✅
- [x] **test_whisper_longform_multi_batch_hard** ❌
- [x] **test_whisper_longform_multi_batch_hard_prev_cond** ❌
- [x] **test_whisper_longform_single_batch_beam** ❌
- [x] **test_tiny_generation** ❌
- [x] **test_tiny_en_generation** ❌
- [x] **test_tiny_en_batched_generation** ❌
- [x] **test_tiny_longform_timestamps_generation** ❌
- [x] **test_large_generation** ❌
- [x] **test_large_batched_generation** ❌
- [x] **test_large_generation_multilingual** ❌
- [x] **test_small_longform_timestamps_generation** ✅



Comments:
> These are the expected results when generating with OpenAI code, right? Are these the results when doing padding up until 30s (like we do) or adding a 30s zero padded audio (as OpenAI does)?

Indeed, these are the expected results generated with OAI code inferred on mel input features extracted through `WhisperFeatureExtractor`. I feel like here more details are required (also updating the PR's description).

In Transformers, we have two inputs possibilities for Whisper:
1. mel spectrogram with 3000 frames →  audio is first padded **to** 30sec with 0.0s and then we extract the mel
2. mel spectrogram with more than 3000 frames →  no need for padding 

--- 
**case 1**
With an audio <=30sec, the difference between our implementation and OAI is that we first pad to 30sec with 0.0s, then extract features and this will be the input to the model's forward, while OAI pads audio with adding 30sec 0.0s, extract features, slice the exact number of frames and then pads the mel spectrogram to 3000 frames with 0.0s.

To understand better, for an audio of 10secs:
**Transformers**: audio + 20sec of 0.0s →  mel spectrogram of shape [80, 3000] where [2000:] frames are close but not exactly 0.0s 
**OAI**: audio + 30sec of 0.0s →  mel spectrogram of shape [80, 4000] → sliced to the duration of the audio (so until frame 1000) and then padded with 0.0s: [2000:] frames are exactly 0.0s.

--- 
**case 2**
No differences (other than numerical difference due to STFT implementation).

---

About the implementation in the [simple whisper fork](https://github.com/eustlb/whisper/tree/transcribe-from-mel-spectrogram):
We just take the mel spectrogram and concat with 3000 frames of 0.0s. This emulates the 30sec of 0.0s added originally. 
For case 1, the duration considered by OAI is 30sec (see [this line](https://github.com/openai/whisper/blob/25639fc17ddc013d56c594bfbf7644f2185fad84/whisper/transcribe.py#L134C9-L134C46)) and therefore the audio segment that will be given to the forward is the exact mel input that was given. 
For case 2, likewise the duration considered is the one of the given mel input. 

With inferring OAI directly on the mel spectrogram (so either of exactly 3000 frames, either on more than 3000 frames), we ensure that each pass of the forward of OAI whisper and our gets the exact same mel spectrogram. This ensures that the expected result we have in the test are indeed results that should be expected given the same input mel with OAI implementation.

**Note:** For tests that required batched inference which is not supported by OAI implementation, I simply run it sequentially to get the outputs 



The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34111). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

As mentioned [here](https://github.com/huggingface/transformers/pull/34135#issuecomment-2422712988), we won't return special tokens anymore with generate for Whisper. Let's adapt the tests a bit for that.

One can argue that for efficiency, if we are going to remove the features of the zero padding anyway, then there is no need to calculate them from the beginning, padding with `HOP_LENGTH` is sufficient to get the exact numerical values and we calculate 1 extra value instead of 3000 

The features resulting from the two padding strategies should be identical
```python
from whisper import log_mel_spectrogram
import torch

SAMPLING_RATE = 16000
N_SAMPLES = SAMPLING_RATE * 30
HOP_LENGTH = 160
N_FRAMES = N_SAMPLES // HOP_LENGTH


for i in range(100):
    audio_n_samples = torch.randint(low=1, high=30, size=(1,)).item() * SAMPLING_RATE

    audio = torch.rand(audio_n_samples)

    features_pad_hop_length = log_mel_spectrogram(audio, padding=HOP_LENGTH)

    features_pad_30s_zeros = log_mel_spectrogram(audio, padding=N_SAMPLES)

    assert torch.allclose(
        features_pad_hop_length[:, :-1],
        features_pad_30s_zeros[:, :-N_FRAMES],
    )
```

Hey @MahmoudAshraf97,

Thanks for your comment! It is a valid point. Nevertheless, this PR does not aim to address it: it is only checking and correcting Whisper tests, with a fixed feature extraction strategy meaning that we are feeding our implementation and OpenAI's one with the exact same features. Feel free to open a PR to modify `WhisperFeatureExtractor` :) !