Qwen2.5-VL-7B-Instruct Accuracy Regression Still Persists in v4.56.2
Author: rahul-tuli
Pull request: no
Labels: name: bug

## Summary

Despite issue #40136 being marked as resolved, the significant accuracy regression in `Qwen2.5-VL-7B-Instruct` model persists in the latest Transformers version `4.56.2`. Our testing shows a more significant drop ~26% relative accuracy drop on MMMU Literature benchmark that was reported in the original issue. 

## Problem Description

The `Qwen2.5-VL-7B-Instruct model shows` inconsistent and degraded performance on multimodal evaluation benchmarks when using recent Transformers versions (4.54.0+), despite PR #40490 claiming to fix this issue.

### Observed Results
```bash
| Transformers Version | MMMU Literature Accuracy | Relative Change |
|---------------------|--------------------------|-----------------|
| 4.53.3              | 93.33% ± 4.63%          | Baseline        |
| 4.56.2              | 70.00% ± 8.51%          | -25.0% relative |
```

## Reproduction Steps

### Environment Setup
```bash
uv pip install lm-eval torch torchvision accelerate Pillow transformers==4.56.2
```

### Evaluation Command
```bash
lm_eval \
    --model hf-multimodal \
    --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True" \
    --tasks mmmu_val_literature \
    --num_fewshot 0 \
    --batch_size 8 \
    --verbosity INFO
```


## Impact

This regression affects:
- Production systems using Qwen2.5-VL for visual question answering
- Research benchmarks and evaluations
- Any multimodal applications relying on Qwen2.5-VL models

The ~25% relative accuracy drop represents a significant degradation that makes the model substantially less reliable for downstream applications.

## Expected Behavior

The model should maintain consistent accuracy across Transformers versions, as observed with v4.53.3 (93.33% accuracy).

## Actual Behavior

The model shows degraded performance in v4.56.2 (70.00% accuracy), indicating the original issue was not fully resolved.

## Additional Context

- **Original Issue**: #40136 (marked as resolved)
- **Claimed Fix**: PR #40490
- **Hardware**: NVIDIA GPU with CUDA support H100
- **Framework**: lm-eval with hf-multimodal backend
- **Model**: Qwen/Qwen2.5-VL-7B-Instruct
- **Task**: MMMU Literature benchmark
- **Evaluation Consistency**: Fixed random seeds used across all tests

## Request

Please reopen investigation into this regression as the issue appears to persist despite the claimed resolution. The consistent ~25% accuracy drop indicates a systematic issue that needs addressing.

---

**Test Environment Details:**
- Python 3.12
- CUDA-enabled environment
- Multiple test runs with identical configurations
- Fixed random seeds for reproducibility

## Raw Test Logs

<details>
<summary>Transformers 4.56.2 Results</summary>

```
hf-multimodal (pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 8
|  Tasks   |Version|Filter|n-shot|Metric|   |Value|   |Stderr|
|----------|------:|------|-----:|------|---|----:|---|-----:|
|Literature|      0|none  |     0|acc   |↑  |  0.7|±  |0.0851|
```
</details>

<details>
<summary>Transformers 4.53.3 Results</summary>

```
hf-multimodal (pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True), gen_kwargs: (None), limit: None, num_fewshot: 0, batch_size: 8
|  Tasks   |Version|Filter|n-shot|Metric|   |Value |   |Stderr|
|----------|------:|------|-----:|------|---|-----:|---|-----:|
|Literature|      0|none  |     0|acc   |↑  |0.9333|±  |0.0463|
```
</details>

### System Info

- `transformers` version: 4.56.2
- Platform: Linux-5.14.0-611.el9.x86_64-x86_64-with-glibc2.34
- Python version: 3.12.11
- Huggingface_hub version: 0.35.1
- Safetensors version: 0.6.2
- Accelerate version: 1.10.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA H100 80GB HBM3


### Environment Setup
```bash
uv pip install lm-eval torch torchvision accelerate Pillow transformers==4.56.2
```

### Evaluation Command
```bash
lm_eval \
    --model hf-multimodal \
    --model_args "pretrained=Qwen/Qwen2.5-VL-7B-Instruct,dtype=bfloat16,add_bos_token=True,convert_img_format=True" \
    --tasks mmmu_val_literature \
    --num_fewshot 0 \
    --batch_size 8 \
    --verbosity INFO
```

### Expected behavior

The model should maintain consistent accuracy across Transformers versions, as observed with v4.53.3 (93.33% accuracy).

Comments:
cc @zucchini-nlp 

Can attest that the issue exists in [ChartQA](https://huggingface.co/datasets/HuggingFaceM4/ChartQA) benchmark as well.
Interestingly, using batch_size=1 fixes the accuracy to benchmark numbers.

> Interestingly, using batch_size=1 fixes the accuracy to benchmark numbers.

Then with high probability it's the 3D rope issue or FA2 varying length attention

@rahul-tuli Thank you so much for pointing a transformers version that works!

Taking a look 👀 

Oke, I got 0.9 on the eval (not 0.933) on my setup in v4.53 and using the latest main branch also results in the same accuracy. So I believe we had an issue that appeared periodically, then we fix it back. AFAIK we had a lot if changes in FA2 recently with some refactors and fixes by @vasqu . I think those changes should have made the model robust to accidental regressions

I verified that the regression between `4.53.2` and `4.56.2` happened due to different reasons, one of the kwargs was not being used when creating rope positions and was fixed in https://github.com/huggingface/transformers/pull/41153/files 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@rahul-tuli Can you check on the current main branch and confirm? It seems to be resolved there but would like confirmation, otherwise we will take another look 👀  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.