Repository: huggingface/transformers (Python)
Date: 2023-05-03 22:47:37+00:00
Task type: Bug Fix

Issue:
Whisper generation support for passing acronym to language arg
### System Info

- `transformers` version: 4.29.0.dev0
- Platform: macOS-13.0-arm64-arm-64bit
- Python version: 3.9.16
- Huggingface_hub version: 0.12.0
- Safetensors version: 0.2.8
- PyTorch version (GPU?): 1.13.1 (False)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@hollance @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```py
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]["array"]
input_features = processor.feature_extractor(sample, return_tensors="pt").input_features

pred_ids = model.generate(input_features, language="de")
```

Throws this error:
<img width="778" alt="Screenshot 2023-05-03 at 6 29 38 PM" src="https://user-images.githubusercontent.com/78612354/236067028-ee7ab371-e9a2-44eb-9895-b5c8f3a2fcdd.png">

Then this error when that's fixed:
<img width="1198" alt="Screenshot 2023-05-03 at 6 30 34 PM" src="https://user-images.githubusercontent.com/78612354/236067052-8f1ae574-db51-44e4-800c-aa4f38b0200e.png">


### Expected behavior

Should recognize and use language passed in acronym format as per the docstring


Issue discussion:

Solution patch:
diff --git a/src/transformers/models/whisper/modeling_whisper.py b/src/transformers/models/whisper/modeling_whisper.py
--- a/src/transformers/models/whisper/modeling_whisper.py
+++ b/src/transformers/models/whisper/modeling_whisper.py
@@ -1562,6 +1562,7 @@ def generate(
             generation_config.return_timestamps = False
 
         if language is not None:
+            language = language.lower()
             generation_config.language = language
         if task is not None:
             generation_config.task = task
@@ -1573,10 +1574,13 @@ def generate(
                     language_token = generation_config.language
                 elif generation_config.language in TO_LANGUAGE_CODE.keys():
                     language_token = f"<|{TO_LANGUAGE_CODE[generation_config.language]}|>"
+                elif generation_config.language in TO_LANGUAGE_CODE.values():
+                    language_token = f"<|{generation_config.language}|>"
                 else:
+                    is_language_code = len(generation_config.language) == 2
                     raise ValueError(
-                        f"Unsupported language: {self.language}. Language should be one of:"
-                        f" {list(TO_LANGUAGE_CODE.keys()) if generation_config.language in TO_LANGUAGE_CODE.keys() else list(TO_LANGUAGE_CODE.values())}."
+                        f"Unsupported language: {generation_config.language}. Language should be one of:"
+                        f" {list(TO_LANGUAGE_CODE.values()) if is_language_code else list(TO_LANGUAGE_CODE.keys())}."
                     )
                 forced_decoder_ids.append((1, generation_config.lang_to_id[language_token]))
             else:


Test patch:
diff --git a/tests/models/whisper/test_modeling_whisper.py b/tests/models/whisper/test_modeling_whisper.py
--- a/tests/models/whisper/test_modeling_whisper.py
+++ b/tests/models/whisper/test_modeling_whisper.py
@@ -414,6 +414,21 @@ def test_generate_fp16(self):
         model.generate(input_features)
         model.generate(input_features, num_beams=4, do_sample=True, early_stopping=False, num_return_sequences=3)
 
+    def test_generate_language(self):
+        config, input_dict = self.model_tester.prepare_config_and_inputs()
+        input_features = input_dict["input_features"]
+        model = WhisperForConditionalGeneration(config).to(torch_device)
+        # Hack to keep the test fast and not require downloading a model with a generation_config
+        model.generation_config.__setattr__("lang_to_id", {"
": 1})
+        model.generation_config.__setattr__("task_to_id", {"transcribe": 2})
+
+        # test language code
+        model.generate(input_features, language="en")
+        # test tokenizer code
+        model.generate(input_features, language="
")
+        # test language name
+        model.generate(input_features, language="English")
+
     def test_forward_signature(self):
         config, _ = self.model_tester.prepare_config_and_inputs_for_common()