[whisper] setting `prompt_condition_type="all-segments"` results in generation errors when `prompt_ids` is set
Author: benniekiss
Pull request: no
Labels: name: bug

name: Audio

name: Whisper

### System Info

- `transformers` version: 4.44.0
- Platform: Linux-6.9.12-201.fsync.fc40.x86_64-x86_64-with-glibc2.35
- Python version: 3.12.4
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.4
- Accelerate version: 0.33.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3080

### Who can help?

@ArthurZucker @Sanch

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
import torch
from transformers import AutoProcessor, WhisperForConditionalGeneration
from datasets import load_dataset, Audio

processor = AutoProcessor.from_pretrained(
    "openai/whisper-large-v3", 
    attn_implementation="flash_attention_2", 
    low_cpu_mem_usage=True, 
    use_safetensors=True,
)
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3")
model.to("cuda", torch.float16)

# load audios > 30 seconds
ds = load_dataset("distil-whisper/meanwhile", "default")["test"]
# resample to 16kHz
ds = ds.cast_column("audio", Audio(sampling_rate=16000))
# take first 8 audios and retrieve array
audio = ds[:8]["audio"]
audio = [x["array"] for x in audio]

vocab = [
    "aberration",
    "accolade",
    "acumen",
    "alacrity",
    "anomaly",
    "antithesis",
    "aplomb",
    "apocryphal",
    "arduous",
    "assiduous",
    "austere",
    "bellicose",
    "benevolent",
    "brevity",
    "cacophony",
    "capricious",
    "cogent",
    "colloquial",
    "conundrum",
    "credence",
    "cryptic",
    "deference",
    "delineate",
    "demagogue",
    "diatribe",
    "dichotomy",
    "disparage",
    "ebullient",
    "eclectic",
    "efficacy",
    "egregious",
    "elucidate",
    "ephemeral",
    "equivocate",
    "esoteric",
    "euphemism",
    "exacerbate",
    "exonerate",
    "expedite",
    "facetious",
    "fallacy",
    "fervent",
    "fortuitous",
    "gregarious",
    "harbinger",
    "idiosyncrasy",
    "imminent",
    "inchoate",
    "ineffable",
    "inimical"
]

prompt = f"Glossary: {'; '.join(vocab)}."

prompt_ids = processor.get_prompt_ids(prompt, return_tensors="pt")
prompt_ids = prompt_ids.to("cuda")
# make sure to NOT truncate the input audio, to return the `attention_mask` and to pad to the longest audio
inputs = processor(audio, return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000)

inputs = inputs.to("cuda", torch.float16)

# transcribe audio to ids
generated_ids = model.generate(
    **inputs, 
    prompt_ids=prompt_ids, 
    condition_on_prev_tokens=True,
    temperature=[0.0, 0.2, 0.4, 0.6, 0.8, 1.0],
    logprob_threshold=-1.0,
    compression_ratio_threshold=2.4,
    no_speech_threshold=None,
    ## COMMENT OUT TO TEST
    prompt_condition_type="all-segments",
)

transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)
transcription[0]
```

Expected result:
> " Folks, if you watch the show, you know I spend a lot of time right over there. Patiently and astutely scrutinizing the boxwood and mahogany chess set of the day's biggest stories. Developing the central headline pawns. Deftly maneuvering an oh-so-topical knight to F6. Feigning a classic Sicilian Neidorf variation on the news. All the while seeing eight moves deep and patiently marshalling the latest press releases into a Fischer-Schosen-Lipnitsky attack that culminates in the elegant, lethal, slow-played en passant checkmate that is my nightly monologue. But sometimes, sometimes, folks, I... Sometimes I startle awake upside down in the monkey bars of a condemned playground on a Superfund site. Get all hepped up on goofballs rummaged through a discarded tag bag of defective toys. Yank out a fistful of disembodied doll limbs. Toss them on a stained kid's placemat from a defunct dentist. Set up a table inside a rusty cargo container down by the wharf. And challenge toothless drifters to the godless bug house blitz of tournament that is my segment... Meanwhile..."

Actual results (with `prompt_condition_type="all-segments"`):
> " Folks, if you watch the show, you know I spend a lot of time right over there, patiently and astutely scrutinizing the boxwood and mahogany chess set of the day's biggest stories, developing the central headline pawns, deftly maneuvering an oh-so-topical knight to f6, feigning a classic Sicilian Neidorf variation on the news, all the while seeing eight moves deep and patiently marshaling the latest press releases into a Fischer-Chosin-Lipnitsky attack that culminates in the elegant, lethal, slow-played en passant chess. I'm not a chess-mack mate that is my nightly monologue, but sometimes, sometimes, folks, I... sometimes I... startle awake upside down in the last blitz of tournament that is my segment..."

### Expected behavior

Setting `prompt_condition_type="all-segments"` results in a significant chunk of the transcription being left off the end.

Semi-relatedly, I was experiencing other generation errors when setting `prompt_condition_type="all-segments"` in my own pipeline, but I haven't been able to reproduce those in an example. The errors I'm receiving are actually when I'm processing a shortform audio segment of about 2 seconds. I'm using a similar prompt as the above example, and the pipeline uses the same generation parameters. This is the error I get:

> Input length of decoder_input_ids is 448, but `max_length` is set to 448. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`.

However, setting `max_new_tokens` to something like `124` or even as low as `62` results in this error on other audio segments:

> The length of `decoder_input_ids` equal `prompt_ids` plus special start tokens is 342, and the `max_new_tokens` is 448. Thus, the combined length of `decoder_input_ids` and `max_new_tokens` is: 790. This exceeds the `max_target_positions` of the Whisper model: 448. You should either reduce the length of your prompt, or reduce the value of `max_new_tokens`, so that their combined length is less than 448.

Setting `prompt_condition_type="first-segment"` allows it to process all of the audio segments just fine without error without having to set `max_new_tokens`.

Is this just a side effect of `prompt_condition_type`? Ideally, I want to use this setting since many of the spoken vocab words do not appear in the first 30 seconds of speech.

Comments:
cc @sanchit-gandhi @ylacombe 

Hey @benniekiss, thanks for opening this issue!
I believe you're using a way too-long token condition. Your vocab is 189 token-long, which is pre-pended to the decoder input ids at each generation.

If you remove a few words (say half of it or more), the model predict the expected answer!

Hope that helps!



Ok that makes sense. Re-reading the error message, I see what it is saying now. And the reason it was failing on different segments is because of the added length of the previously decoded tokens.

The [first line of that error message](https://github.com/huggingface/transformers/blob/d70347726577e9823e35c11883e98c5b2c520b37/src/transformers/models/whisper/generation_whisper.py#L1692-L1703) is a little confusing, and I think it could be clarified to say something like, `The length of decoder_input_ids, including special start tokens, prompt_ids, and previous tokens, ...`.

I noticed that in order for `prompt_condition_type="all-segments"` to be set, `condition_on_prev_tokens` has to be set as well. Is there a structural reason for this? If not, I think it would be helpful to be able to set `prompt_condition_type="all-segments"` without setting `condition_on_prev_tokens`. 

It would also be helpful to be able to control the number of tokens used when setting `condition_on_prev_tokens=True`, so that a fixed number of tokens could be specified.

I'm willing to contribute any of these changes if they seem worthwhile!

Hey @benniekiss, indeed a clearer message would be beneficial! Would you like to open a PR to make it clearer ?

Regarding `prompt_condition_type` and `condition_on_prev_tokens`, the reason is that if `condition_on_prev_tokens=False`, no conditioning is done at all and thus `prompt_condition_type=all-segments` doesn't matter. Does that make sense?

> It would also be helpful to be able to control the number of tokens used when setting condition_on_prev_tokens=True, so that a fixed number of tokens could be specified.

Not sure to understand what you mean there tbh ! Let me know !


I'll definitely open a PR for the warning!

That makes sense regarding `condition_on_prev_tokens`, thanks for explaining.

 > Not sure to understand what you mean there tbh ! Let me know !
 
I was a unclear-What would be helpful is if there was an option, like `condition_on_num_prev_tokens: int = N`, where only `N` tokens from the previous window were included.

Since whisper runs over a sliding window of 30 seconds, the generated tokens per window can vary. When conditioning on previous tokens, it means that the number of tokens added to the beginning is unpredictable. So when creating a vocab prompt, in my case, it makes it a guessing game as to how to maximize the length of the prompt. By being able to configure the number of previous tokens used in conditioning, it makes the length of decoder_ids more predictable, making it easier to maximize prompt length.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

closing since this was addressed already :) last comment is a tangential request that could be its own issue