Qwen-2-VL generates inconsistent logits between `generate()` and `__call__ ()` for multi-modal queries
Author: waffoo
Pull request: no
Labels: name: bug

name: VLM

### System Info

- `transformers` version: 4.49.0.dev0  (commit: 62db3e, for Qwen2.5-VL)
- Platform: Linux-5.4.0-204-generic-x86_64-with-glibc2.31
- Python version: 3.10.14
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	- compute_environment: LOCAL_MACHINE
	- distributed_type: NO
	- mixed_precision: bf16
	- use_cpu: False
	- debug: False
	- num_processes: 1
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: all
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- enable_cpu_affinity: False
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: NO
- Using GPU in script?: YES
- GPU type: NVIDIA A100 80GB PCIe

### Who can help?

@ArthurZucker @zucchini-nlp

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When using Qwen2-VL or Qwen2.5-VL with multi-modal queries, logits generated inside `generate()` have non-negligible error from the ones directly generated from `__call__()` method.

The code follows the example introduced in [Qwen2-VL docs on Huggingface](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct#using-%F0%9F%A4%97--transformers-to-chat).
```python
import torch

from qwen_vl_utils import process_vision_info
from transformers import (Qwen2_5_VLForConditionalGeneration,
                        Qwen2_5_VLProcessor)

model_name = "Qwen/Qwen2.5-VL-7B-Instruct"
processor = Qwen2_5_VLProcessor.from_pretrained(model_name)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(model_name,
                                                            torch_dtype=torch.bfloat16,
                                                            device_map="auto")
messages = [{
    "role": "user",
    "content": [
        {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
        {"type": "text", "text": "Describe this image."},
    ]
}]

text = processor.apply_chat_template(messages,
                                    tokenize=False,
                                    add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")
```

`generate()` produces the same logits as `__call__()` with `use_cache=True` and an appropriate `cache_position` similar to what is done in `generate()` method.  
However, the logits from `__call__()` without cache, which are usually used for training, are different from the ones used in `generate()`.

```python
with torch.no_grad():
  logits_in_generate = model.generate(**inputs, max_new_tokens=5,
                                      output_logits=True,
                                      return_dict_in_generate=True).logits
  logits_from_call = model(**inputs).logits
  logits_from_call_with_cache = model(**inputs, use_cache=True,
                                      cache_position=torch.arange(inputs.input_ids.shape[1])).logits

# Check logits of the first generated tokens (First 8 vocabularies only)
print(f'generate():\n{logits_in_generate[0][0, :8]}')
# tensor([ 9.5625, 14.3750, 11.1875,  9.9375,  9.7500,  5.8125,  8.2500, 12.0625],
#        device='cuda:0')

print(f'__call__():\n{logits_from_call[0, -1, :8]}')
# tensor([11.0000, 16.1250, 12.8125, 11.5625, 11.1875,  6.9375,  9.5625, 13.2500],
#        device='cuda:0', dtype=torch.bfloat16)

print(f'__call__() with cache:\n{logits_from_call_with_cache[0, -1, :8]}')
# tensor([ 9.5625, 14.3750, 11.1875,  9.9375,  9.7500,  5.8125,  8.2500, 12.0625],
#        device='cuda:0', dtype=torch.bfloat16) 
```

If a query contains no image, the logits from `generate()` and `__call__()` are the same.
```python
messages = [{
    "role": "user",
    "content": [
        {"type": "text", "text": "Describe this image."},
    ]
}]

# ... same as above ...

image_inputs = None

# ... same as above ...

# These are the same
print(f'generate():\n{logits_in_generate[0][0, :8]}')
print(f'__call__(): {logits_from_call[0, -1, :8]}')
print(f'__call__() with cache: {logits_from_call_with_cache[0, -1, :8]}')
```

It is unclear if this is purely a precision problem or involves other issues.  
A related discussion can be found in #25420, in particular [this comment](https://github.com/huggingface/transformers/issues/25420#issuecomment-1775317535), but in my case, switching to higher precision does not resolve the issue, and visual tokens appear to affect the logits unexpectedly.

### Expected behavior

Consistent logits are used in `generate()` and `__call__()`.

Comments:
Hey @waffoo !

I believe it is related to calculating the correct `RoPE deltas`, which we usually calculate once per generation when the inputs are passed the first time. Then we simply cache the results. Since you already called `generate()`, the `rope_deltas` got cached and were used in afterwards `forward` calls because we couldn't infer that it was the fist call from `cache_position`

This indeed needs a fix, I will submit a PR

https://github.com/huggingface/transformers/blob/62db3e6ed67a74cc1ed1436acd9973915c0a4475/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1648-L1657

The PR resolved my issue, thank you so much!