Qwen 2.5 VL Batch Inference Error: tensors not on the same device
Author: YiqunChen1999
Pull request: no
Labels: name: bug

### System Info

```
Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.51.3
- Platform: Linux-5.15.0-127-generic-x86_64-with-glibc2.31
- Python version: 3.11.11
- Huggingface_hub version: 0.30.2
- Safetensors version: 0.5.3
- Accelerate version: 1.6.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: True
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3090

```

### Who can help?

@zucchini-nlp 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying batch inference following the demo from https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct with `transformers==4.51.3`. I can successfully run the single-sample demo, but fail with batch inference:

```
Traceback (most recent call last):
  File "/PATH/TO/DEMO/DIR/demo.py", line 96, in <module>
    generated_ids = model.generate(**inputs, max_new_tokens=128)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
... # ignored
File "MY_CONDA_ENV/lib/python3.11/site-packages/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 1334, in _prepare_4d_causal_attention_mask_with_cache_position
    diagonal_attend_mask = torch.arange(target_length, device=device) > cache_position.reshape(-1, 1)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!
```


How I run the script:
```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 python demo.py
```


My code (copied from the above url):

<details>

```python
from transformers import Qwen2_5_VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

# default: Load the model on the available device(s)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2.5-VL-3B-Instruct", torch_dtype="auto", device_map="auto"
)

# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
# model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
#     "Qwen/Qwen2.5-VL-3B-Instruct",
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

# default processer
processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct")

# The default range for the number of visual tokens per image in the model is 4-16384.
# You can set min_pixels and max_pixels according to your needs, such as a token range of 256-1280, to balance performance and cost.
# min_pixels = 256*28*28
# max_pixels = 1280*28*28
# processor = AutoProcessor.from_pretrained("Qwen/Qwen2.5-VL-3B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

# Sample messages for batch inference
messages1 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"},
            {"type": "text", "text": "What are the common elements in these pictures?"},
        ],
    }
]
messages2 = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who are you?"},
]
# Combine messages for batch processing
messages = [messages1, messages2]

# Preparation for batch inference
texts = [
    processor.apply_chat_template(msg, tokenize=False, add_generation_prompt=True)
    for msg in messages
]
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=texts,
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Batch Inference
generated_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_texts = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_texts)

```
</details>


### Expected behavior

Forward the batched inputs and output normally.

Comments:
Thanks for reporting. I can reproduce it locally and will make a fix soon

> Thanks for reporting. I can reproduce it locally and will make a fix soon

I have noticed the same issue working with the video inference, can you confirm that there is a problem there too. Thanks

@zucchini-nlp  I encounter the same issue when I was not using batch inference

```
import torch
from transformers import Qwen2_5_VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info

model_path_vl = "/jindofs_temp/users/huggingface/model/Qwen/Qwen2.5-VL-7B-Instruct"

# We recommend enabling flash_attention_2 for better acceleration and memory saving, especially in multi-image and video scenarios.
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_path_vl,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",
    device_map="auto",
)

# default processor
processor = AutoProcessor.from_pretrained(model_path_vl)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "video",
                "video": "./data/space_woaudio.mp4",
                "min_pixels": 4 * 28 * 28,
                "max_pixels": 256 * 28 * 28,
                "total_pixels": 20480 * 28 * 28,
                "fps":0.5
            },
            {"type": "text", "text": "Describe this video."},
        ],
    }
]
# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

image_inputs, video_inputs, video_kwargs = process_vision_info(messages, return_video_kwargs=True)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
    **video_kwargs,
)
inputs = inputs.to("cuda")
generated_ids = model.generate(**inputs, max_new_tokens=128)
```
the error is:
`RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!`

It's on A100 80G, torch 2.2.2, transformers github main branch

more about the traceback, i think it is caused by the impl of the rope in the model

```bash
File [~/transformers-main/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py:1823](https://fis-ingress-100.alibaba-inc.com/nb/notebook-12813263/lab/tree/nlp_solutions/LiveRangerX/notebooks/~/transformers-main/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#line=1822), in Qwen2_5_VLForConditionalGeneration.forward(self, input_ids, attention_mask, position_ids, past_key_values, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict, pixel_values, pixel_values_videos, image_grid_thw, video_grid_thw, rope_deltas, cache_position, second_per_grid_ts)
   1816 if position_ids is None and (attention_mask is None or attention_mask.ndim == 2):
   1817     # calculate RoPE index once per generation in the pre-fill stage only
   1818     if (
   1819         (cache_position is not None and cache_position[0] == 0)
   1820         or self.rope_deltas is None
   1821         or (past_key_values is None or past_key_values.get_seq_length() == 0)
   1822     ):
-> 1823         position_ids, rope_deltas = self.get_rope_index(
   1824             input_ids,
   1825             image_grid_thw,
   1826             video_grid_thw,
   1827             second_per_grid_ts,
   1828             attention_mask,
   1829         )
   1830         self.rope_deltas = rope_deltas
   1831     # then use the prev pre-calculated rope-deltas to get the correct position ids
   1832     else:

File [~/transformers-main/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py:1666](https://fis-ingress-100.alibaba-inc.com/nb/notebook-12813263/lab/tree/nlp_solutions/LiveRangerX/notebooks/~/transformers-main/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#line=1665), in Qwen2_5_VLForConditionalGeneration.get_rope_index(self, input_ids, image_grid_thw, video_grid_thw, second_per_grid_ts, attention_mask)
   1663 range_tensor = torch.arange(llm_grid_t).view(-1, 1)
   1664 expanded_range = range_tensor.expand(-1, llm_grid_h * llm_grid_w)
-> 1666 time_tensor = expanded_range * second_per_grid_t * self.config.vision_config.tokens_per_second
   1668 time_tensor_long = time_tensor.long()
   1669 t_index = time_tensor_long.flatten()

```