Fix Pan and Scan on batched images Gemma3
Author: yonigozlan
Pull request: yes
Labels: name: for patch

# What does this PR do?
Currently, inputs such as this one:
```python
from transformers import (
    AutoModelForImageTextToText,
    Gemma3Processor,
    Gemma3ForConditionalGeneration,
)
import torch

model_id = "gg-hf-g/gemma-3-4b-it"
torch_device = "cuda" if torch.cuda.is_available() else "cpu"
processor = Gemma3Processor.from_pretrained(
    model_id, padding_side="left", use_fast=False
)

model = Gemma3ForConditionalGeneration.from_pretrained(
    model_id, low_cpu_mem_usage=True, torch_dtype=torch.bfloat16
).to(torch_device)
crop_config = {
    "images_kwargs": {
        "do_pan_and_scan": True,
        "pan_and_scan_max_num_crops": 448,
        "pan_and_scan_min_crop_size": 32,
        "pan_and_scan_min_ratio_to_activate": 0.3,
    }
}


messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}],
    },
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/cow_beach_1.png",
            },
            {"type": "text", "text": "What is shown in this image?"},
        ],
    },
]
messages_2 = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}],
    },
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/cow_beach_1.png",
            },
            {
                "type": "image",
                "url": "https://www.ilankelman.org/stopsigns/australia.jpg",
            },
            {"type": "text", "text": "Are these images identical?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    [messages, messages_2],
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
    padding=True,
    add_generation_prompt=True,
    **crop_config,
).to(torch_device)

output = model.generate(**inputs, max_new_tokens=30, do_sample=False)
output_text = processor.batch_decode(output, skip_special_tokens=True)
print(output_text)
```
will crash with both slow and fast image processor.

Non-batched inputs with pan and scan will also fail with fast image processors.

This PR fixes both issue, and simplify the image processing by processing flattened images instead of nested ones.
This PR also introduces some changes to take better advantages of batch processing in the fast image processor, with a batched pan_and_scan method.

Also add some tests.


Cc @zucchini-nlp @RyanMullins 


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

> Hm, interesting, since we had a test for PaS in image-processors which was green. Or was the error in processing code?

I don't think PaS was tested with batched inputs was it?

Edit: I see it was, but the problem is really with num_crops, where for example fi you have image inputs like [[image1, image2], [image3]], the num_crops returned will be [[2, 2], [2]] which will crash when trying to return pt tensors.
Modified the test to account for this case!


> ➕ on spending time to write simpler code! We can find some magic here and there.
In this specific case, I don't necessarily know. But, sometimes padding then unpadding can lead to better perfs.

True! In general I think now that we have several techniques for fast processing, I need to make a new benchmark to compare those for each models (batched vs unbatched, padded vs unpadded, different techniques for splitting images into patches etc.).
@ArthurZucker  Do you thinks it would be interesting to add these benchmarks to Transformers as a util or something? Or better to keep them separate?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36864). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: gemma3

This comment contains run-slow, running the specified jobs: This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

@zucchini-nlp CI won't run the integration tests because of require_read_token

It worked, we have asked access to gemma-3 already. All green, cool

Ah great, merging then