[Whisper] Refactor forced_decoder_ids & prompt ids
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR refactors `forced_decoder_ids` making sure that we now always pass prompted ids as `decoder_input_ids` into generate for Whisper. The whole idea of forcing ids instead of just passing them as initial tokens was a bad design choice and we should try to move away from it.

In addition, Whisper prompting is improved by:
- Not allowing `prompt_ids` to be passed as a numpy array
- Enable `prompt_ids` for long-form generation with two modes:
  - a) prompt only the first segment
  - b) prompt every segment

While a) is the only supported case in the original Whisper repo b) can be very useful as can be seen in the added slow test [here](https://github.com/huggingface/transformers/pull/28687/files#r1467376608).


This is the final code PR regarding Whisper for Transformers. In the next weeks focus will be put on writing nice docs, tutorials and blog posts.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28687). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Still need to make sure that "Whisper default language detection behavior" stays the same. See: https://huggingface.co/openai/whisper-large-v3/discussions/71#65b78d5a34297095bcecfe78

**Update**:

This PR now does two additonal things:
- `generation_config` is always preferred over `config.json`
- By default multilingual Whisper does language detection followed by transcription

cc @sanchit-gandhi 

- [x] TODO(Patrick) Make sure to run all slow tests

Hi there @patrickvonplaten  I started getting the following error when I tried finetune whisper-tiny model, which points  me to this PR  here :

> Due to a bug fix in https://github.com/huggingface/transformers/pull/28687 transcription using a multilingual Whisper will default to language detection followed by transcription instead of translation to English.This might be a breaking change for your use case. If you want to instead always translate your audio to English, make sure to pass `language='en'`.
> 
>  ValueError: Multiple languages detected when trying to predict the most likely target language for transcription. It is currently not supported to transcribe to different languages in a single batch. Please make sure to either force a single language by passing `language='...'` or make sure all input audio is of the same language.


I am simply following the instructions here by @sanchit-gandhi :
https://huggingface.co/blog/fine-tune-whisper#prepare-environment

I used exactly the same code  here ( https://colab.research.google.com/drive/1Ffd2ZnaaRfM02A0QiW08blufw0qFdlIu?usp=sharing ) previously to push same model (https://huggingface.co/ckandemir/whisper-tiny-tr) ,and this week I wanted to experiment with different parameters started getting the the error above,  I was wondering if you could help me understand the issue here. 


Hey @canberk17 - this can be fixed by specifying the language in the `generation_config` as follows:
```python
model.generation_config.language = "
"  # for hindi
model.generation_config.task = "transcribe"
```
Provided you run these two lines before calling `trainer.train()`, you'll de-activate automatic language detection during inference and be able to run the code as before. I'll update the blog post + Colab to reflect these changes!

Hi @sanchit-gandhi , thank you so much for clarifying !

I just started hitting the
```
ValueError: Multiple languages detected when trying to predict the most likely target language for transcription. It is currently not supported to transcribe to different languages in a single batch. Please make sure to either force a single language by passing language='...' or make sure all input audio is of the same language.
```

error as well due to the change in behavior. Setting `language='en'` seems to fix this, however when using `return_language=True` I now get all the returned languages as English, and not the source language(s) that I was getting previously. Is that expected?

Okay I think I get it now, since you're tracking `init_tokens` as a list of tokens, you're just requiring all items in a batch to have the same language somewhat arbitrarily. Any plans to address that? That's a pretty major breaking change that prevents batched inference on mixed language audio, especially when Whisper is our method for detecting the audio language. Can the language tokens just be tracked as a batch tensor instead of a single token, and that would fix the whole issue?

What I can't figure out is exactly how this worked before. I see the `forced_decoder_ids` gets a `(1, None)` entry for the language token, but I can't see where the `None` is handled when creating the `init_tokens`. I see it might get handled in the `ForceTokensLogitsProcessor`? But it seems like it would err out when creating the `decoder_input_ids`.

> Hey @canberk17 - this can be fixed by specifying the language in the `generation_config` as follows:
> 
> ```python
> model.generation_config.language = "
"  # for hindi
> model.generation_config.task = "transcribe"
> ```
> 
> Provided you run these two lines before calling `trainer.train()`, you'll de-activate automatic language detection during inference and be able to run the code as before. I'll update the blog post + Colab to reflect these changes!

Hi @sanchit-gandhi , I tried implementing this method, but when I ran `trainer.train()`, whenever it reaches the evaluation step, it tells me this message.

>You have passed task=transcribe, but also have set `forced_decoder_ids` to [[1, None], [2, 50359]] which creates a conflict. `forced_decoder_ids` will be ignored in favor of task=transcribe.

Any idea why this is happening?

ValueError: A custom logits processor of type <class 'transformers.generation.logits_process.ForceTokensLogitsProcessor'> with values <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f4230cfac50> has been passed to `.generate()`, but it has already been created with the values <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f422829c510>. <transformers.generation.logits_process.ForceTokensLogitsProcessor object at 0x7f422829c510> has been created by passing the corresponding arguments to generate or by the model's config default values. If you just want to change the default values of logits processor consider passing them as arguments to `.generate()` instead of using a custom logits processor.


what's the way to use it now?

Hey! Could you open a new issue with a repro? 🤗