Repository: huggingface/transformers (Python)
Date: 2023-09-14 14:02:14+00:00
Task type: Bug Fix

Issue:
Whisper Prompting max_new_tokens
### System Info

- `transformers` version: 4.31.0
- Platform: Linux-5.15.109+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.0.1+cu118 (False)
- Tensorflow version (GPU?): 2.12.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.7.1 (cpu)
- Jax version: 0.4.14
- JaxLib version: 0.4.14
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@sanchit-gandhi

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

## Bug Related
We keep `model.config.max_length=448`. The error happens when:
1. `len(prompt_ids) + max_new_tokens > model.config.max_length  + 1`
2. We fix `max_new_tokens` in `model.generate()`
3.  The length of the generated new tokens reaches its maximum. This mainly occurs when Whisper fails to predict the `eos` token and starts repeating some sequence of tokens.

```python
from transformers import (WhisperFeatureExtractor, WhisperProcessor, WhisperForConditionalGeneration)
from datasets import load_dataset

# Load dataset
fleurs_fr = load_dataset("google/fleurs", "fr_fr", split="test")

# Load Processor + Model
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny")

# Chosen a sample that causes repetition
i = 512
input_speech = fleurs_fr[i]["audio"]["array"]
sr = fleurs_fr[i]["audio"]["sampling_rate"]

# Create big enough prompt text
# It should be sliced inside generate anyway
prompt_text = " bien," * 113
prompt_ids = processor.get_prompt_ids(prompt_text)

# Generate
input_features = processor(input_speech, return_tensors="pt",
                            sampling_rate=16e3).input_features

output_with_prompt = model.generate(input_features,
                                    language="fr",
                                    task="transcribe",
                                    prompt_ids= prompt_ids,
                                    max_new_tokens=224)
```
Output:

```
IndexError                                Traceback (most recent call last)
[<ipython-input-4-3420d576291f>](https://localhost:8080/#) in <cell line: 4>()
      2                             sampling_rate=16e3).input_features
      3 
----> 4 output_with_prompt = model.generate(input_features,
      5                                     language="fr",
      6                                     task="transcribe",

3 frames
[/usr/local/lib/python3.10/dist-packages/transformers/models/whisper/modeling_whisper.py](https://localhost:8080/#) in generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, return_timestamps, task, language, is_multilingual, prompt_ids, return_token_timestamps, **kwargs)
   1747                 )
   1748 
-> 1749         outputs = super().generate(
   1750             inputs,
   1751             generation_config,

[/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py](https://localhost:8080/#) in decorate_context(*args, **kwargs)
    113     def decorate_context(*args, **kwargs):
    114         with ctx_factory():
--> 115             return func(*args, **kwargs)
    116 
    117     return decorate_context

[/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py](https://localhost:8080/#) in generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, **kwargs)
   1536 
   1537             # 11. run greedy search
-> 1538             return self.greedy_search(
   1539                 input_ids,
   1540                 logits_processor=logits_processor,

[/usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py](https://localhost:8080/#) in greedy_search(self, input_ids, logits_processor, stopping_criteria, max_length, pad_token_id, eos_token_id, output_attentions, output_hidden_states, output_scores, return_dict_in_generate, synced_gpus, streamer, **model_kwargs)
   2370                 continue  # don't waste resources running the code we don't need
   2371 
-> 2372             next_token_logits = outputs.logits[:, -1, :]
   2373 
   2374             # pre-process distribution

IndexError: index -1 is out of bounds for dimension 1 with size 0
```

The bug might be caused by no condition set on `max_new_tokens` inside the `generate()` function, which might be a general bug for generation and not only for prompting.

## Note
Also, as I was reading the code I noticed [this line](https://github.com/huggingface/transformers/blob/d0c1aebea467af499331234e7b285a6bf91ea073/src/transformers/models/whisper/modeling_whisper.py#L1726C1-L1726C82):
`text_prompt_ids = text_prompt_ids[-self.config.max_length // 2 - 1 :]`

It slices the text prompt ids and takes `(self.config.max_length // 2 + 1)` tokens instead of `(self.config.max_length // 2 - 1)` as taken in the original code of Whisper [here](https://github.com/openai/whisper/blob/c09a7ae299c4c34c5839a76380ae407e7d785914/whisper/decoding.py#L599).



### Expected behavior

- Clear warning or error about surpassing the `model.max_length`. 
- Being able to set `max_new_tokens=224 ( = max_length // 2)` during prompting.







Issue discussion:
Hi @Helene-Maxcici! Thanks for writing this issue, there’s definitely an out of bounds issue here. 

Appreciate you catching the precedence issue that the slicing doesn’t quite match OpenAI’s, we should change that in the fix PR so its slicing one less than half the max_length instead one one more than half. Ultimately it’s not at the root of this problem since the prompt isn’t competing for space with anything else, like a prefix, and we could just decrement the max_new_tokens param by 1 and this script would run, or alternatively after updating the slicing to match OpenAI’s we could still increment max_new_tokens by 2 to 226 and it would still have this error.

Instead, I think the issue is that the length stopping criteria warning [here](https://github.com/huggingface/transformers/blob/d0c1aebea467af499331234e7b285a6bf91ea073/src/transformers/generation/stopping_criteria.py#L64-L69) doesn’t capture the out of bounds issue for this model since the it looks [here](https://github.com/huggingface/transformers/blob/d0c1aebea467af499331234e7b285a6bf91ea073/src/transformers/generation/utils.py#L1019-L1025) for `max_position_embeddings` in the generation_config, but the value is named `max_target_positions` for Whisper. Not sure if Hugging Face would prefer that we rename the value in Whisper’s generation config to `max_position_embeddings` or add a second config attribute check for `max_target_positions` to determine what to pass to the stopping criteria, or something else but @sanchit-gandhi could say more
I'm not sure if this will help or not but I faced the same error running 
```python
generated_tokens = (
                model.generate(
                    input_features=batch["input_features"].to("cuda"),
                    decoder_input_ids=batch["labels"][:, :4].to("cuda"),
                    max_new_tokens=448,
                )

```
However if I use PEFT model as in 

```python
model = WhisperForConditionalGeneration.from_pretrained(
  peft_config.base_model_name_or_path, device_map="auto", load_in_8bit=True)
  model = PeftModel.from_pretrained(model, evaluate_model)

```

I don't face this issue if I set the `max_new_tokens` to 224 in either case (PEFT or without)


Thanks for the excellent issue description @Helene-Maxcici and for the astute remarks @connor-henderson! IMO each of the findings deserves a PR of its own:
* For the max length issue, I think the best thing we can do is throw a warning in the `.generate` method for Whisper when the model's max length is exceeded. Probably, this can be placed after we determine the correct `max_length` / `max_new_tokens` with prompting: https://github.com/huggingface/transformers/blob/5e5fa0d88c293e6d5be2517b4f45680ba3bb5df2/src/transformers/models/whisper/modeling_whisper.py#L1730 I would be against changing the `config`/`generation_config` for the model, since this is very difficult to do without breaking changes. Since Whisper is quite unique in its approach to prompting, I think we're safe to just add a check in the Whisper model's `.generate` method, rather than the more generic one (cc @gante)
* Agree with your spot and @connor-henderson's remarks with the slicing difference: this would be a quick PR to fix!

Would you like to open a PR for one or both of these issues @Helene-Maxcici? Happy to help guide the integration process, or answer any questions / queries along the way!
Hi @sanchit-gandhi , thank you for your response! I would be happy to open a PR for each. 
Thank you for opening a well-explained issue, @Helene-Maxcici! 🤗 

Since this issue is particular to Whisper, which modifies `max_new_tokens` in its `generate` function, I agree -- we should add a warning in Whisper's generate (cc @sanchit-gandhi)

Solution patch:
diff --git a/src/transformers/models/whisper/modeling_whisper.py b/src/transformers/models/whisper/modeling_whisper.py
--- a/src/transformers/models/whisper/modeling_whisper.py
+++ b/src/transformers/models/whisper/modeling_whisper.py
@@ -1719,13 +1719,22 @@ def generate(
             decoder_start_token_id, *text_prompt_ids = prompt_ids
             # Slicing the text prompt ids in a manner consistent with the OpenAI implementation
             # to accomodate context space for the prefix (see https://github.com/openai/whisper/blob/c09a7ae299c4c34c5839a76380ae407e7d785914/whisper/decoding.py#L599)
-            text_prompt_ids = text_prompt_ids[-self.config.max_length // 2 - 1 :]
+            text_prompt_ids = text_prompt_ids[-self.config.max_target_positions // 2 - 1 :]
             # Set the decoder_start_token_id to 

             kwargs.update({"decoder_start_token_id": decoder_start_token_id})
 
             # If the user passes `max_new_tokens`, increase its number to account for the prompt
             if kwargs.get("max_new_tokens", None) is not None:
                 kwargs["max_new_tokens"] += len(text_prompt_ids)
+                if kwargs["max_new_tokens"] >= self.config.max_target_positions:
+                    raise ValueError(
+                        f"The length of the sliced `prompt_ids` is {len(text_prompt_ids)}, and the `max_new_tokens` "
+                        f"{kwargs['max_new_tokens'] - len(text_prompt_ids)}. Thus, the combined length of the sliced "
+                        f"`prompt_ids` and `max_new_tokens` is: {kwargs['max_new_tokens']}. This exceeds the "
+                        f"`max_target_positions` of the Whisper model: {self.config.max_target_positions}. "
+                        "You should either reduce the length of your prompt, or reduce the value of `max_new_tokens`, "
+                        f"so that their combined length is less that {self.config.max_target_positions}."
+                    )
 
             # Reformat the forced_decoder_ids to incorporate the prompt
             non_prompt_forced_decoder_ids = (


Test patch:
diff --git a/tests/models/whisper/test_modeling_whisper.py b/tests/models/whisper/test_modeling_whisper.py
--- a/tests/models/whisper/test_modeling_whisper.py
+++ b/tests/models/whisper/test_modeling_whisper.py
@@ -1075,6 +1075,29 @@ def test_generate_with_prompt_ids_and_forced_decoder_ids(self):
         for row in output.tolist():
             self.assertListEqual(row[: len(expected_output_start)], expected_output_start)
 
+    def test_generate_with_prompt_ids_max_length(self):
+        config, input_dict = self.model_tester.prepare_config_and_inputs_for_common()
+        config.max_target_positions = 5
+
+        model = WhisperForConditionalGeneration(config).eval().to(torch_device)
+        input_features = input_dict["input_features"]
+        prompt_ids = np.asarray(range(4))
+        sliced_prompt_ids = prompt_ids[1:]
+        sliced_prompt_ids = sliced_prompt_ids[-config.max_target_positions // 2 - 1 :]
+        max_new_tokens = 5
+
+        with self.assertRaisesRegex(
+            ValueError,
+            f"The length of the sliced `prompt_ids` is {len(sliced_prompt_ids)}, and the `max_new_tokens` "
+            f"{max_new_tokens}. Thus, the combined length of the sliced `prompt_ids` and `max_new_tokens` is: "
+            f"{len(sliced_prompt_ids) + max_new_tokens}. This exceeds the `max_target_positions` of the Whisper model: "
+            f"{config.max_target_positions}. You should either reduce the length of your prompt, or reduce the "
+            f"value of `max_new_tokens`, so that their combined length is less that {config.max_target_positions}.",
+        ):
+            model.generate(input_features, max_new_tokens=max_new_tokens, prompt_ids=prompt_ids)
+
+        model.generate(input_features, max_new_tokens=1, prompt_ids=prompt_ids)
+
 
 @require_torch
 @require_torchaudio