Insanely-Fast-Whisper model: large - Cantonese KeyError "yue"
Author: asusdisciple
Pull request: no

### System Info

Using Newest transformers from Main. Ubuntu Linux 22.04

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Load Insanely Fast Whisper with large-v1 in HF
2. Try to transcribe something in Cantonese, "yue"
3. KeyError |yue| not found.

### Expected behavior

I think the problem is Cantonese was added as a language with whisper large-v3. So when insanely fast whisper is used it somehow tries to use this language instead of ignoring it, because yue is not a valid language.

Comments:
Hey could you share a full reproducer, with a traceback and the output of `transformers-cli env`

```
from transformers import pipeline
import torch

model = "openai/whisper-large"

obj = pipeline(model=model,
               torch_dtype=torch.float16,
               device="cuda:0",  # or mps for Mac devices
               chunk_length_s=30,
               batch_size=8,
               return_timestamps=False,
               model_kwargs={"use_flash_attention_2": False},

                       )
obj.model = obj.model.to_bettertransformer()

audio = obj("/.../csmd024.wav", generate_kwargs={"language": "yue"})

print(audio)
```

Error log:
```
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
The BetterTransformer implementation does not support padding during training, as the fused kernels do not support attention masks. Beware that passing padded batched data during training may result in unexpected outputs. Please refer to https://huggingface.co/docs/optimum/bettertransformer/overview for more details.
Traceback (most recent call last):
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/.../Whisper/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 924, in device_placement
    yield
  File "/.../Whisper/venv/lib/python3.10/site-packages/transformers/pipelines/base.py", line 1046, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
  File "/.../Whisper/venv/lib/python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 569, in _forward
    tokens = self.model.generate(
  File "/.../Whisper/venv/lib/python3.10/site-packages/transformers/models/whisper/modeling_whisper.py", line 2050, in generate
    forced_decoder_ids.append((1, generation_config.lang_to_id[language_token]))
KeyError: '
'
```


That is expected you are not using `whisper-large-v3`. Other version do not support this language

I think the behaviour is still strange because usually if the language is not supported it gives you an error with the languages that are supported.

That's because it's supported in the actual code but not in the lang to id. But for sure the error message should be improved! 

The issue is with this:
```python 
                if generation_config.language in generation_config.lang_to_id.keys():
                    language_token = generation_config.language
                elif generation_config.language in TO_LANGUAGE_CODE.keys():
                    language_token = f"<|{TO_LANGUAGE_CODE[generation_config.language]}|>"
                elif generation_config.language in TO_LANGUAGE_CODE.values():
                    language_token = f"<|{generation_config.language}|>"
```
the language token is in the supported language but not in the lang to id.