Support generating with fallback for short form audio in Whisper
Author: kamilakesbi
Pull request: yes
Labels: name: Audio

# What does this PR do?

The aim of this PR is to refacto the Whisper `generate` method to handle both short form and long form audio generation similarly. It will support short form audio generation with fallback (as requested in #29508). 

Here's what I've done: 

### Removed previous short-form scripts: 

I've removed the part of the code used for short form generation. This involve lines [562 to 603 ](https://github.com/huggingface/transformers/blob/348e2294ac1a3cc744a93f04ab1f886439ee90d3/src/transformers/models/whisper/generation_whisper.py#L562) and lines [498 to 505](https://github.com/huggingface/transformers/blob/348e2294ac1a3cc744a93f04ab1f886439ee90d3/src/transformers/models/whisper/generation_whisper.py#L498) in main. Now when a short form audio (or a batched short form of audio) is passed to `generate`, it's processed by the part of the code previously used for long form generation.

### Use is_shortform to still distinguish between short form and long form in some cases: 

- In the `_postprocess_outputs` method we only return `past_key_values` if the audios are short form. For long form audios it is too expensive. (cf. this [line](https://github.com/huggingface/transformers/blob/80c5bc6a212a39e8a3ccb340ef5a7a4c2609f5e6/src/transformers/models/whisper/generation_whisper.py#L932)). 

-  In `_retrieve_max_frames_and_seek `:  For long form audios, we necessarily need to pass an attention mask but not for short form audios. We can thus compute `max_frames` and `seek` without relying on the attention mask for short form audios.

- I've also updated the `split_by_batch_index` method:  the previous method was broken when return_dict_in_generate was set to True for different short form audio cases. Now it handles both short form and long form audios. 

- I've removed the `is_shortform` parameter from the inputs to the `_retrieve_logit_processors ` method to allow the use of `generation_config.no_speech_threshold` for short form audios.  

- I've removed `is_shortfrom` parameter from the inputs to the `_set_return_outputs ` method to allow the use of `logprob_threshold` for short form audios. 

### Make num_return_sequences>1 compatible with generate_with_fallback: 

- This is a bit tricky because `generate_with_fallback` can't handle num_return_sequences>1 by design. I've added a new method, called `_expand_variables_for_generation` , which expands the different variables before passing into `generate_with_fallback` when `generation_config.num_return_sequences>1`. After expansion it will set `generation_config.num_return_sequences` to 1 for compatibility with `generate_with_fallback`. 

### Ensure that the output format for short form audio is compatible with the output format in main: 

The output format for long-form audio is different from that for short-form audio. In order to ensure that the output is similar to that obtained in main when processing short form audio, we need to add a few post-processing steps: This is what is done in lines [721 to 765](https://github.com/huggingface/transformers/blob/80c5bc6a212a39e8a3ccb340ef5a7a4c2609f5e6/src/transformers/models/whisper/generation_whisper.py#L721). In particular here: 

- We add an `EOS` token to the output sequence as it was removed during generation with fallback. 
-  We return the token timestamps if `return_token_timestamps` is True in the correct format ([see here](https://github.com/huggingface/transformers/blob/80c5bc6a212a39e8a3ccb340ef5a7a4c2609f5e6/src/transformers/models/whisper/generation_whisper.py#L741)). 
- If `return_dict_in_generate` is True, we use the new method `_stack_split_outputs` to get the output dict (containing all attributes (scores, encoder_attentions, etc.)) in the right format. `_stack_split_outputs` basically performs the opposite operations to `split_by_batch_index `.

### Make failing slow tests to pass: 

- I've updated some failing slow tests and made them pass ([see here](https://github.com/huggingface/transformers/pull/30984#issuecomment-2148028809)). 

### Add new tests to make sure generation with fallback works for short form audios: 

I've added two tests:  `test_whisper_shortform_single_batch_prev_cond ` and `test_whisper_shortform_multi_batch_hard_prev_cond`. 


## Who can review: 

@sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30984). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I have run `test_modeling_whisper` slow tests. Here are the currently failing tests (which pass on main): 

- test_assisted_decoding_matches_greedy_search_0_random
- test_speculative_decoding_non_distil
- test_generate_continue_from_past_key_values
- test_tiny_en_generation
- test_tiny_generation
- test_tiny_en_batched_generation
- test_tiny_timestamp_generation 
- test_tiny_token_timestamp_batch_generation
- test_large_timestamp_generation

@sanchit-gandhi I have taken all your reviews into account. The CI and slow test pass :) 
This PR should be ready for final review!

cc @amyeroberts. 

Added two slow tests (`test_whisper_shortform_single_batch_prev_cond` and `test_whisper_shortform_multi_batch_hard_prev_cond`) and made a few modifications to make them pass. 

Hi @ArthurZucker, could you please do a review on this PR ? 
Failing tests on the CI are unrelated to this work I think :) 

@kamilakesbi As this is quite big PR - is it possible to split up? In particular, could we split up the enabling of speculative decoding and the unification of the long and short form audio logic? 

Hi @amy, in this PR we only focus on the unification of the long and short form audio logic! we don't change how speculative decoding is handled here :) 

> I've adapted the code to make it compatible with speculative decoding, return_tokens_timestamps, return_timestamps, return_dict_in_generate, num_beams>=1 and num_return_sequences>=1, max_new_tokens and max_length parameters.

@kamilakesbi Could you update the PR description to reflect the current state (either disambiguate for confused people like me or remove :) )? 

@amyeroberts, sorry for that. We've indeed worked on many points with Sanchit to make this PR pass. 
I've updated the PR description to show the main changes in the current state. Hope this makes it clearer for you! 

I'll take care of reviewing it in a bit! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Gentle ping @ArthurZucker 

@ArthurZucker thanks for your review! I took your remarks into account :) 

Failing tests are unrelated to this PR. If this is ok for you we can perhaps merge or wait for the CI to be green... 

Let's wait for the full CI seems alright now!

Also a question ont answered!

The CI is green yes :) if it's ok for you I can merge!