Qwen2-VL used to work with `inputs_embeds` instead of `input_ids`, but no more
Author: minostauros
Pull request: no
Labels: name: bug

name: VLM

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-4.18.0-513.18.1.el8_9.x86_64-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: Yes
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@zucchini-nlp 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

### Preparation

```python
import torch
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2VLForConditionalGeneration.from_pretrained(
 "Qwen/Qwen2-VL-7B-Instruct",
 torch_dtype=torch.bfloat16,
 attn_implementation="eager", # flash_attention_2 also produces the same error
 device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")
```

### Working example

```python
generated_ids = model.generate(**inputs, max_new_tokens=128)
```

### Used to work
Worked in 9470d6532436e9db2951a196effd6f8841befb76 but not in v4.47.1 [[comparison]](https://github.com/huggingface/transformers/compare/9470d6532436e9db2951a196effd6f8841befb76...241c04d36867259cdf11dbb4e9d9a60f9cb65ebc)

```python
input_ids = inputs["input_ids"]
attention_mask = inputs["attention_mask"]
pixel_values = inputs["pixel_values"]
image_grid_thw = inputs["image_grid_thw"]

inputs_embeds = model.model.embed_tokens(input_ids)
if pixel_values is not None:
    pixel_values = pixel_values.type(model.visual.get_dtype())
    image_embeds = model.visual(pixel_values, grid_thw=image_grid_thw)
    n_image_tokens = (input_ids == model.config.image_token_id).sum().item()
    n_image_features = image_embeds.shape[0]
    if n_image_tokens != n_image_features:
        raise ValueError(
            f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
        )
    image_mask = (
        (input_ids == model.config.image_token_id)
        .unsqueeze(-1)
        .expand_as(inputs_embeds)
        .to(inputs_embeds.device)
    )
    image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
    inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)

if attention_mask is not None:
    attention_mask = attention_mask.to(inputs_embeds.device)

generated_ids = model.generate(inputs_embeds=inputs_embeds, attention_mask=attention_mask, max_new_tokens=128)
```

### Expected behavior

The latter should work the same as the former.

The latter's error message example

```
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 578, in forward
    attn_weights = attn_weights + causal_mask
RuntimeError: The size of tensor a (2362) must match the size of tensor b (1182) at non-singleton dimension 3
```

Comments:
> ### System Info
> * `transformers` version: 4.47.1
> * Platform: Linux-4.18.0-513.18.1.el8_9.x86_64-x86_64-with-glibc2.35
> * Python version: 3.10.12
> * Huggingface_hub version: 0.27.0
> * Safetensors version: 0.4.5
> * Accelerate version: 1.2.1
> * Accelerate config:    not found
> * PyTorch version (GPU?): 2.5.1+cu121 (True)
> * Tensorflow version (GPU?): not installed (NA)
> * Flax version (CPU?/GPU?/TPU?): not installed (NA)
> * Jax version: not installed
> * JaxLib version: not installed
> * Using distributed or parallel set-up in script?: No
> * Using GPU in script?: Yes
> * GPU type: NVIDIA H100 80GB HBM3
> 
> ### Who can help?
> [@zucchini-nlp](https://github.com/zucchini-nlp)
> 
> ### Information
> * [ ]  The official example scripts[x]  My own modified scripts
> 
> ### Tasks
> * [ ]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)[x]  My own task or dataset (give details below)
> 
> ### Reproduction
> ### Preparation
> import torch
> from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
> from qwen_vl_utils import process_vision_info
> 
> model = Qwen2VLForConditionalGeneration.from_pretrained(
>  "Qwen/Qwen2-VL-7B-Instruct",
>  torch_dtype=torch.bfloat16,
>  attn_implementation="eager", # flash_attention_2 also produces the same error
>  device_map="auto",
> )
> processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")
> 
> messages = [
>     {
>         "role": "user",
>         "content": [
>             {
>                 "type": "image",
>                 "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
>             },
>             {"type": "text", "text": "Describe this image."},
>         ],
>     }
> ]
> 
> text = processor.apply_chat_template(
>     messages, tokenize=False, add_generation_prompt=True
> )
> image_inputs, video_inputs = process_vision_info(messages)
> inputs = processor(
>     text=[text],
>     images=image_inputs,
>     videos=video_inputs,
>     padding=True,
>     return_tensors="pt",
> )
> inputs = inputs.to("cuda")
> ### Working example
> generated_ids = model.generate(**inputs, max_new_tokens=128)
> ### Used to work
> Worked in [9470d65](https://github.com/huggingface/transformers/commit/9470d6532436e9db2951a196effd6f8841befb76) but not in v4.47.1 [[comparison]](https://github.com/huggingface/transformers/compare/9470d6532436e9db2951a196effd6f8841befb76...241c04d36867259cdf11dbb4e9d9a60f9cb65ebc)
> 
> input_ids = inputs["input_ids"]
> attention_mask = inputs["attention_mask"]
> pixel_values = inputs["pixel_values"]
> image_grid_thw = inputs["image_grid_thw"]
> 
> inputs_embeds = model.model.embed_tokens(input_ids)
> if pixel_values is not None:
>     pixel_values = pixel_values.type(model.visual.get_dtype())
>     image_embeds = model.visual(pixel_values, grid_thw=image_grid_thw)
>     n_image_tokens = (input_ids == model.config.image_token_id).sum().item()
>     n_image_features = image_embeds.shape[0]
>     if n_image_tokens != n_image_features:
>         raise ValueError(
>             f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
>         )
>     image_mask = (
>         (input_ids == model.config.image_token_id)
>         .unsqueeze(-1)
>         .expand_as(inputs_embeds)
>         .to(inputs_embeds.device)
>     )
>     image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
>     inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
> 
> if attention_mask is not None:
>     attention_mask = attention_mask.to(inputs_embeds.device)
> 
> generated_ids = model.generate(inputs_embeds=inputs_embeds, attention_mask=attention_mask, max_new_tokens=128)
> ### Expected behavior
> The latter should work the same as the former.
> 
> The latter's error message example
> 
> ```
>   File "/usr/local/lib/python3.10/dist-packages/transformers/models/qwen2_vl/modeling_qwen2_vl.py", line 578, in forward
>     attn_weights = attn_weights + causal_mask
> RuntimeError: The size of tensor a (2362) must match the size of tensor b (1182) at non-singleton dimension 3
> ```

Hello, I'm interested in this work and tried experimenting with the two methods myself. However, after making the changes, the results are noticeably worse than before. Do you know why this might happen? @minostauros 

@justairr hey! You mean the quality of generation degraded after the PR was merged? Does that happen with input ids or input embeddings? A more clear small reproducer would help us a ton to find the issue

> [@justairr](https://github.com/justairr) hey! You mean the quality of generation degraded after the PR was merged? Does that happen with input ids or input embeddings? A more clear small reproducer would help us a ton to find the issue

Performance would drop a lot using inputs_embed. I used the inference method above for evaluation. Thank you for your help! @zucchini-nlp 

```python
import torch
from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
from qwen_vl_utils import process_vision_info

model = Qwen2VLForConditionalGeneration.from_pretrained(
 "Qwen/Qwen2-VL-7B-Instruct",
 torch_dtype=torch.bfloat16,
 attn_implementation="eager", # flash_attention_2 also produces the same error
 device_map="auto",
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

input_ids = inputs["input_ids"]
attention_mask = inputs["attention_mask"]
pixel_values = inputs["pixel_values"]
image_grid_thw = inputs["image_grid_thw"]

inputs_embeds = model.model.embed_tokens(input_ids)
if pixel_values is not None:
    pixel_values = pixel_values.type(model.visual.get_dtype())
    image_embeds = model.visual(pixel_values, grid_thw=image_grid_thw)
    n_image_tokens = (input_ids == model.config.image_token_id).sum().item()
    n_image_features = image_embeds.shape[0]
    if n_image_tokens != n_image_features:
        raise ValueError(
            f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
        )
    image_mask = (
        (input_ids == model.config.image_token_id)
        .unsqueeze(-1)
        .expand_as(inputs_embeds)
        .to(inputs_embeds.device)
    )
    image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
    inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)

if attention_mask is not None:
    attention_mask = attention_mask.to(inputs_embeds.device)

generated_ids = model.generate(inputs_embeds=inputs_embeds, attention_mask=attention_mask, max_new_tokens=128)
```

> Performance would drop a lot using inputs_embed. I used the inference method above for evaluation. 

Hi, what kind of evaluation? what's the difference of the performance? what is the metric are you using?
How can I see the difference? Can you provide the full code?

> > Performance would drop a lot using inputs_embed. I used the inference method above for evaluation.
> 
> Hi, what kind of evaluation? what's the difference of the performance? what is the metric are you using? How can I see the difference? Can you provide the full code?




I used VLMEvalKit for evaluation. [comparison](https://github.com/justairr/VLMEvalKit/commit/9d27064c8f75849f061f1ff6d13f9c639450ad3b) I have tested on InfoVQA_VAL, using the script below:
```bash
AUTO_SPLIT=1 python run.py --data InfoVQA_VAL --model Qwen2-VL-7B-Instruct --verbose
```

Performance would drop a lot under MAX_PIXELS=1280, MIN_PIXELS=768 settings, from 72.7621 to 53.341.

Thanks for the detail!
The thing is, though I cannot reveal the company's internal code, I achieved better performance than VLMEvalKit in MVBench using `inputs_embeds` (did not test without using `inputs_embeds`).

As I do not have used InfoVQA_VAL, we might need some inspection...

If InfoVQA_VAL is a video benchmark, you may change image related codes like `image_grid_thw` to video related variables.

Thank you for your reply! Could you please provide an example demonstrating how you use the `inputs_embeds` code for evaluation? I am unable to find more usage examples of `inputs_embeds`.
BTW, InfoVQA is a single-image VQA dataset. @minostauros 

For now, it doesn't seem to be `transformers`'s problem, and sorry I cannot fully write a code to assist you.
instead of using processor's outputs, try the following

1. `apply_chat_template` to the input json
2. replace processor.image_token with `'
'`
3. put the text into tokenizer, not processor
4. embed `input_ids` with `model.embed_tokens`
5. masked_scatter visual info
6. generate
7. voila

> For now, it doesn't seem to be `transformers`'s problem, and sorry I cannot fully write a code to assist you. instead of using processor's outputs, try the following
> 
> 1. `apply_chat_template` to the input json
> 2. replace processor.image_token with `'
'`
> 3. put the text into tokenizer, not processor
> 4. embed `input_ids` with `model.embed_tokens`
> 5. masked_scatter visual info
> 6. generate
> 7. voila

OK, thanks for your help anyway!

 Yep, I might be wrong with "it doesn't seem to be `transformers`'s problem", so please share if you make any progress, thanks!

> > [@justairr](https://github.com/justairr) hey! You mean the quality of generation degraded after the PR was merged? Does that happen with input ids or input embeddings? A more clear small reproducer would help us a ton to find the issue
> 
> Performance would drop a lot using inputs_embed. I used the inference method above for evaluation. Thank you for your help! [@zucchini-nlp](https://github.com/zucchini-nlp)
> 
> import torch
> from transformers import Qwen2VLForConditionalGeneration, AutoTokenizer, AutoProcessor
> from qwen_vl_utils import process_vision_info
> 
> model = Qwen2VLForConditionalGeneration.from_pretrained(
>  "Qwen/Qwen2-VL-7B-Instruct",
>  torch_dtype=torch.bfloat16,
>  attn_implementation="eager", # flash_attention_2 also produces the same error
>  device_map="auto",
> )
> processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")
> 
> messages = [
>     {
>         "role": "user",
>         "content": [
>             {
>                 "type": "image",
>                 "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
>             },
>             {"type": "text", "text": "Describe this image."},
>         ],
>     }
> ]
> 
> text = processor.apply_chat_template(
>     messages, tokenize=False, add_generation_prompt=True
> )
> image_inputs, video_inputs = process_vision_info(messages)
> inputs = processor(
>     text=[text],
>     images=image_inputs,
>     videos=video_inputs,
>     padding=True,
>     return_tensors="pt",
> )
> inputs = inputs.to("cuda")
> 
> input_ids = inputs["input_ids"]
> attention_mask = inputs["attention_mask"]
> pixel_values = inputs["pixel_values"]
> image_grid_thw = inputs["image_grid_thw"]
> 
> inputs_embeds = model.model.embed_tokens(input_ids)
> if pixel_values is not None:
>     pixel_values = pixel_values.type(model.visual.get_dtype())
>     image_embeds = model.visual(pixel_values, grid_thw=image_grid_thw)
>     n_image_tokens = (input_ids == model.config.image_token_id).sum().item()
>     n_image_features = image_embeds.shape[0]
>     if n_image_tokens != n_image_features:
>         raise ValueError(
>             f"Image features and image tokens do not match: tokens: {n_image_tokens}, features {n_image_features}"
>         )
>     image_mask = (
>         (input_ids == model.config.image_token_id)
>         .unsqueeze(-1)
>         .expand_as(inputs_embeds)
>         .to(inputs_embeds.device)
>     )
>     image_embeds = image_embeds.to(inputs_embeds.device, inputs_embeds.dtype)
>     inputs_embeds = inputs_embeds.masked_scatter(image_mask, image_embeds)
> 
> if attention_mask is not None:
>     attention_mask = attention_mask.to(inputs_embeds.device)
> 
> generated_ids = model.generate(inputs_embeds=inputs_embeds, attention_mask=attention_mask, max_new_tokens=128)

I met some problem when i tried to generate with input_embeds, have you solve this ？

Might be related to how Qwen2-VL computes RoPE deltas, as it relies on input ids to infer at which positions we have pixel values. I can investigate this a bit later, and if that's true it means we can't use the model with only embeddings. You can try passing both embeddings and ids, so the model has access to infer pixel values' positions