[Whisper] ASR pipeline ignores/ rejects generate_kwargs on inference
Author: Vaibhavs10
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.1+cu116 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

pipeline: @Narsil 
whisper/ speech: @ArthurZucker / @sanchit-gandhi 


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

The failure mode is specific to `Whisper` model in the ASR pipeline. Potentially introduced after the introduction of the `return_timestamps` argument.

Repro can be found in the [colab here](https://colab.research.google.com/drive/1koqO7Tjos5vGlFBCugGeCRKJm5gf2hfj?usp=sharing) or on [GitHub here](https://github.com/Vaibhavs10/scratchpad/blob/main/Whisper_return_timestamp_bug_report.ipynb)

Essentially, when the pipeline is invoked and inferred with:
```python
pipe = pipeline("automatic-speech-recognition", model="openai/whisper-small", generate_kwargs={"task": "transcribe", "language": "german"})
pipe(test_sample, chunk_length_s=30)
```

It rejects to accept the `generate_kwargs` and throws an error:
```
ValueError: The following `model_kwargs` are not used by the model: ['task', 'language'] (note: typos in the generate arguments will also show up in this list)
```

The interesting bit, if I run inference whilst explicitly setting `return_timestamps=True` it does work however, 'translates' instead of `transcribe`.
```python
pipe(test_sample, return_timestamps=True, chunk_length_s=30, stride_length_s=[6,0])
```

I went more step further and tried with loading a vanilla pipeline and ran the pipeline by explicitly setting the `forced_decoder_ids` and it worked well:

```python
pipe = pipeline("automatic-speech-recognition", model="openai/whisper-small")
pipe.model.config.forced_decoder_ids = (
    pipe.tokenizer.get_decoder_prompt_ids(
        language="de", task="transcribe"
    )
)
pipe(test_sample, chunk_length_s=30)
```

However, if I now pass the `return_timestamps=True` it again rejects the original decoder_ids and `translates` in English:
```python
pipe(test_sample, return_timestamps=True, chunk_length_s=30, stride_length_s=[6,0])
```

As said above, the [colab](https://colab.research.google.com/drive/1koqO7Tjos5vGlFBCugGeCRKJm5gf2hfj?usp=sharing) or on [GitHub](https://github.com/Vaibhavs10/scratchpad/blob/main/Whisper_return_timestamp_bug_report.ipynb) has a proper repro, do give it a look.

### Expected behavior

The expected behaviour of the pipeline would be to respect the `generate_kwargs` and throw potentially a more meaningful error message.

Comments:
You should use the main branch! 
The following 
```python 
    def test_simple_whisper_translation(self):
        speech_recognizer = pipeline(
            task="automatic-speech-recognition",
            model="openai/whisper-large",
            framework="pt",
        )
        ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation").sort("id")
        filename = ds[40]["file"]
        output = speech_recognizer(filename)
        self.assertEqual(output, {"text": " A man said to the universe, Sir, I exist."})

        model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large")
        tokenizer = AutoTokenizer.from_pretrained("openai/whisper-large")
        feature_extractor = AutoFeatureExtractor.from_pretrained("openai/whisper-large")

        speech_recognizer_2 = AutomaticSpeechRecognitionPipeline(
            model=model, tokenizer=tokenizer, feature_extractor=feature_extractor
        )
        output_2 = speech_recognizer_2(filename)
        self.assertEqual(output, output_2)

        # either use generate_kwargs or set the model's generation_config
        # model.generation_config.task = "transcribe"
        # model.generation_config.lang = "
"
        speech_translator = AutomaticSpeechRecognitionPipeline(
            model=model,
            tokenizer=tokenizer,
            feature_extractor=feature_extractor,
            generate_kwargs={"task": "transcribe", "language": "
"},
        )
        output_3 = speech_translator(filename)
        self.assertEqual(output_3, {"text": " Un uomo ha detto all'universo, Sir, esiste."})
```
is used in our testing suit and can confirm that it works.

In my case, your script outputs the folliwing: 
```python 
│ /home/arthur_huggingface_co/transformers/src/transformers/models/whisper/modeling_whisper.py:134 │
│ 5 in generate                                                                                    │
│                                                                                                  │
│   1342 │   │                                                                                     │
│   1343 │   │   if hasattr(generation_config, "is_multilingual") and generation_config.is_multil  │
│   1344 │   │   │   if hasattr(generation_config, "language"):                                    │
│ ❱ 1345 │   │   │   │   forced_decoder_ids.append((1, generation_config.lang_to_id[generation_co  │
│   1346 │   │   │   else:                                                                         │
│   1347 │   │   │   │   forced_decoder_ids.append((1, None))                                      │
│   1348                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'german'
```
Because 
```python 
In [7]: pipe.model.generation_config.language
Out[7]: 'german'
```
It should be `
`. There was an issue opened to make it so that the actual language code is used here, but we have not adressed it yet. 

Ha! I am invoking the `pipeline` the same way as you are in your first comment. The only difference is that on `4.26.1` it doesn't work and neither does it raise any errors.
It works perfectly on `main`: https://github.com/Vaibhavs10/scratchpad/blob/main/Whisper_return_timestamp_bug_report_w_main.ipynb

Do we have any ETA on when this would make it to a release?

Think it was part of the patch release see [here ](https://github.com/huggingface/transformers/releases/tag/v4.26.1)