Qwen2.5-VL-7B-Instruct: Significant accuracy regression on MMMU benchmark with transformers >=4.54.0
Author: rahul-tuli
Pull request: no
Labels: name: bug

### System Info

## Description

We've observed a significant accuracy regression when evaluating Qwen2.5-VL-7B-Instruct on the MMMU Literature benchmark after upgrading from transformers 4.53.3 to 4.54.0. The accuracy drops from **86.67%** to **73.33%**, representing a **13.34 percentage point decrease** (~15.4% relative regression).

## Environment

- **Model**: Qwen/Qwen2.5-VL-7B-Instruct
- **Evaluation framework**: lm-eval
- **Task**: mmmu_val_literature (30 samples)
- **Hardware**: CUDA-enabled GPU
- **Python**: 3.10
- **Other dependencies**: torch, torchvision, accelerate, Pillow


### Transformers 4.53.3 Output
```
hf-multimodal (pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 8
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|Literature|      0|none  |     0|acc   |↑  |0.8667|±  |0.0631|
```

### Transformers 4.54.0 Output
```
hf-multimodal (pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 8
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|Literature|      0|none  |     0|acc   |↑  |0.7333|±  |0.0821|
```

## Summary

| Version | Accuracy | Stderr | Difference from baseline |
|---------|----------|--------|-------------------------|
| 4.53.3  | 0.8667   | ±0.0631| -                       |
| 4.54.0  | 0.7333   | ±0.0821| -0.1334 (-15.4%)        |



## Possible causes

The regression may be related to:
- Changes in the default image processor (slow → fast) 
- Changes in the vision model loading mechanism
- Modifications to the multimodal pipeline

## Impact

This regression affects production systems using Qwen2.5-VL models for visual question answering and multimodal understanding tasks. Users upgrading to 4.54.0 may experience unexpected accuracy drops.

## Additional context

- The evaluation uses a fixed random seed for reproducibility
- Both tests were run on the same hardware with identical configurations
- The regression is statistically significant given the standard errors


## Steps to Reproduce

### 1. Install dependencies
```bash
pip install lm-eval torch torchvision accelerate Pillow
```

### 2. Test with transformers 4.53.3 (baseline)
```bash
pip install transformers==4.53.3

lm_eval \
    --model hf-multimodal \
    --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True" \
    --tasks mmmu_val_literature \
    --num_fewshot 0 \
    --batch_size 8 \
    --verbosity INFO
```

### 3. Test with transformers 4.54.0
```bash
pip install transformers==4.54.0

lm_eval \
    --model hf-multimodal \
    --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True" \
    --tasks mmmu_val_literature \
    --num_fewshot 0 \
    --batch_size 8 \
    --verbosity INFO
```

## Expected behavior

The model accuracy should remain consistent or improve. A 15% relative performance regression on a standard benchmark is unexpected and concerning.

Comments:
@rahul-tuli Thanks for raising! I'm looking into it. Looks like the change to fast image processor might not be the only culprit/ not the culprit, as I'm getting different scores on main when forcing processing to slow compared to v4.53.3

I've identified two other PRs that impacted the eval accuracy on this benchmark, not sure these are intended/expected, could you maybe help on this @zucchini-nlp?

This makes the accuracy goes from 0.9 to 0.8667:
`Fix SDPA attention precision issue in Qwen2.5-VL` https://github.com/huggingface/transformers/pull/37363

And this one from 0.8667 to 0.8:
`[qwen2 vl] fix packing with all attentions` https://github.com/huggingface/transformers/pull/39447

Could you confirm that you get similar accuracy changes on your end after these PRs @rahul-tuli ?

I believe this PR is actually breaking the vision:


And this one from 0.8667 to 0.8:
[qwen2 vl] fix packing with all attentions https://github.com/huggingface/transformers/pull/39447

Let me make a new PR discussing what I've found:   [#40154](https://github.com/huggingface/transformers/issues/40154)

Hmm, is it possible to get one example input where the generations are different? I don't think it is only in vision, because in above PRs one updates vision and the other text attentions


I'm seeing even worse results in the latest release, opened a new issue for that #41180