Fix Qwen-VL family with prompt tuning
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/pull/41173#issuecomment-3586424032.

TL;DR: PEFT pop position ids and cache position when doing prompt tuning, and expects that they will be reconstructed in forward using past cache length. However Qwen assumes that `cache_position == None` is always a prefill stage which will not be correct with PEFT

This PR changes the is-prefill check in qwen forward pass to be more explicit and rely on cache length. Checked with the below script

cc @BenjaminBossan . Btw, there are some more issues if we install "rc0" and run PEFT. I guess one of the breaking PR affected it, lmk if you need help on that as well

```python
import torch
from transformers import AutoTokenizer, Qwen2_5_VLForConditionalGeneration
from peft import get_peft_model, PromptTuningConfig, PromptTuningInit, TaskType

max_new_tokens = 40
model_id =  "Qwen/Qwen2.5-VL-3B-Instruct"
text = "Discuss the most important work by Mary Shelley."
tokenizer = AutoTokenizer.from_pretrained(model_id)
inputs = tokenizer(text, return_tensors="pt").to(0)

model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_id, device_map=0)
prompt_tune_config = PromptTuningConfig(
    task_type=TaskType.CAUSAL_LM,
    prompt_tuning_init=PromptTuningInit.RANDOM,
    num_virtual_tokens=5,
)
model = get_peft_model(model, prompt_tune_config)
model.prompt_encoder.default.embedding.weight.data.zero_()  # make peft almost no-op

torch.manual_seed(0)
with torch.no_grad():
    generated_ids = model.generate(
        # seq len should be 5 virtual + 9 normal tokens
        **inputs,
        max_new_tokens=max_new_tokens,
        # use_cache=False,  # without cache, it works
    )
print(tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0])

```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_42508). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks @zucchini-nlp this indeed solves the issue. The generations with cache enabled are still bad compared to cache being disabled, but that was already true before #41173 so it must be something else. 

```
With cache:
Discuss the most important work by Mary Shelley. Mary Shelley,,, and, and, and and and and the novel Shelley W., and and and, and the, and Mary Shelley, and and Mary Shelley Shelley, and Mary Shelley,
```

```
Without cache:
Discuss the most important work by Mary Shelley. Mary Shelley was a British author who wrote "Frankenstein" in 1818. She was born in 1797 and died in 1851. She was a
```

> Btw, there are some more issues if we install "rc0" and run PEFT. I guess one of the breaking PR affected it, lmk if you need help on that as well

We have a backlog of a couple of things that are broken since then, but maybe not everything. What do you mean specifically?

> The generations with cache enabled are still bad compared to cache being disabled,

Hmm, I ran the script with 4.57.3 and got a normal generation. Smth might have been broken after that in peft

> We have a backlog of a couple of things that are broken since then, but maybe not everything. What do you mean specifically?

Nothing specifically, just saw a few issues which I think are related to recent breaking changes I introduced in v5. If peft might requires fixes or changes on transformers side re these, lmk

run-slow: qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe

This comment contains `run-slow`, running the specified jobs: 

models: ["models/qwen2_5_omni", "models/qwen2_5_vl", "models/qwen2_vl", "models/qwen3_omni_moe", "models/qwen3_vl", "models/qwen3_vl_moe"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19822280975)

✅ No failing test specific to this PR 🎉 !


> Hmm, I ran the script with 4.57.3 and got a normal generation. Smth might have been broken after that in peft

Hmm, how? Did you apply your patch from this PR to 4.57.3? Also, what PEFT version did you use? I tried a few PEFT versions back and each time, the generation was just gibberish.

> Nothing specifically, just saw a few issues which I think are related to recent breaking changes I introduced in v5. If peft might requires fixes or changes on transformers side re these, lmk

Thanks for the offer, I'll ping you if something comes up. The remaining issues I'm aware of are related to weight conversion or tied weights/gradient checkpointing.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe

> Hmm, how? Did you apply your patch from this PR to 4.57.3? Also, what PEFT version did you use? I tried a few PEFT versions back and each time, the generation was just gibberish.

@BenjaminBossan oh you're right, my mind made sense of gibberish and assumed it was normal haha. The issue is again related to popping cache position because the fallback was initialize position ids of all zeros. I fixed it in my last commit to rely on cache length as it is equivalent to `cache_position[0]`


run-slow: qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe



> The issue is again related to popping cache position because the fallback was initialize position ids of all zeros. I fixed it in my last commit to rely on cache length as it is equivalent to `cache_position[0]`

Oh really nice, now I get the same generations with and without cache, thank you so so much!