[Qwen2VL] Image processor backend refactor (#43514) introduces pixel_values precision drift, causing bbox coordinate shift
Author: chenxi971215
Pull request: no

## Title

[Qwen2VL] Image processor backend refactor (#43514) introduces pixel_values precision drift, causing bbox coordinate shift

## Labels

bug, Vision

## Body

### System Info

- `transformers` 4.57.1 vs 5.5.4
- `torchvision` 0.22.0
- `torch` 2.7.0
- Model: `Qwen3-VL-32B-Instruct`
- GPU: NVIDIA H20 (baseline) vs PPU (comparison)

### Who can help?

@yonigozlan @ArthurZucker

### Information

After the image processor backend refactor in #43514, `Qwen2VLImageProcessorFast` (removed in 5.5.4) was replaced by `Qwen2VLImageProcessor(TorchvisionBackend)`. Both use torchvision BICUBIC resize and identical normalization parameters, but they produce slightly different `pixel_values`.

### Reproduction

1. Deploy `Qwen3-VL-32B-Instruct` with SGLang using `transformers==4.57.1` and `transformers==5.5.4` separately
2. Send the same image (medical report) with the same prompt requesting bbox coordinates (0-1000 normalized)
3. Compare `pixel_values` tensor statistics and model output

### Observed behavior

**pixel_values difference** (same image, same preprocessing config):

| Version | Processor Class | pixel_values mean |
|---------|----------------|-------------------|
| 4.57.1 | `Qwen2VLImageProcessorFast` | 0.5669328... |
| 5.5.4 | `Qwen2VLImageProcessor` (TorchvisionBackend) | 0.5669392... |

The difference appears at the 5th-6th decimal place. After passing through the ViT encoder and language model, this gets amplified into ~20-unit coordinate shifts in 0-1000 normalized bbox space.

**Example bbox output comparison** (same image, same prompt):

| Field | transformers 4.57.1 | transformers 5.5.4 |
|-------|--------------------|--------------------|
| Patient name bbox | [100, 265, 268, 278] | [95, 245, 258, 258] |
| Doctor name bbox | [765, 750, 908, 778] | [768, 725, 908, 755] |
| Patient ID bbox | [673, 265, 900, 278] | [666, 245, 900, 258] |

The y-coordinates are systematically shifted by ~20 units, and bbox heights are reduced. This significantly impacts downstream tasks like sensitive information masking.

### Root cause analysis

Both versions declare `resample = PILImageResampling.BICUBIC` and use `torchvision.transforms.v2.functional.resize`. The code paths diverge in the parent class:

- **4.57.1**: `BaseImageProcessorFast._further_process_kwargs()` converts `resample` → `interpolation` kwarg, passes to `_preprocess(interpolation=...)` → `self.resize(interpolation=...)`
- **5.5.4**: `BaseImageProcessor._standardize_kwargs()` keeps `resample` kwarg, passes to `_preprocess(resample=...)` → `self.resize(resample=...)` → internally converts via `pil_torch_interpolation_mapping`

Additionally, `group_images_by_shape` and `reorder_images` were refactored between versions, potentially affecting tensor stacking order and memory layout, which can introduce floating-point precision differences.

The `image_mean`/`image_std` handling also differs: 4.57.1 converts `list → tuple` in `_further_process_kwargs`, while 5.5.4 does not, which may affect the normalize computation path.

### Expected behavior

The backend refactor should produce numerically identical (or near-identical) `pixel_values` for the same input image when using the same interpolation method and normalization parameters. A precision drift at the 5th decimal place may seem small, but it gets amplified through the model and causes observable output differences for coordinate-sensitive tasks.

### Workaround

Lock `transformers==4.57.1` to use the original `Qwen2VLImageProcessorFast` implementation.

Comments:
Taking a look 👁️ 

@zucchini-nlp  @chenxi971215  Reproduced and investigated on CPU (deterministic mode). Tested all three comparison points: 4.57.1 fast vs 5.5.4, 4.57.1 fast vs current main, 5.5.4 vs current main -  across single images triggering actual BICUBIC resize AND batches of 4 mixed-size images exercising [group_images_by_shape] in /src/transformers/image_transforms.py. CPU max_diff = 0.0 in every case.

The suspected causes (resample→interpolation kwarg path, list→tuple for image_mean/std, group_images_by_shape ordering) are all provably equivalent on the same hardware. I hypothesize that the observed ~6e-6 drift is hardware-induced: your baseline ran on H20 and your comparison ran on PPU ,different GPU architectures can produce slightly different floating-point results for BICUBIC interpolation.

Could you reproduce both 4.57.1 and 5.5.4 on the same hardware to confirm whether the drift persists? If it does, I'd be happy to dig deeper.

In the meantime, I've written two regression tests that guard the processor's numerical determinism and cross-backend shape/grid equivalence. @ArthurZucker  @yonigozlan happy to share the test PR if that's useful.

Oh nice, thanks for checking that without external libraries. In that case I'll wait for @chenxi971215 to reply and re-checl

Re the tests: we have a few tests already to check backend equivalence, compilation and there aren't any equivalence issues. The hardware might be the culprit if @chenxi971215 confirms that the diff disappears when launching the model under same conditions  

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.