VLMs Processors are not fully consistent in the inputs formats they accept
Author: yonigozlan
Pull request: no
Labels: name: WIP

name: bug

name: Processing

### Summary of accepted inputs for image-text-to-text-models
| Model | Uses Image Tokens | Accepts URL | Accepts PIL | Accepts Path | Accepts Non-Nested Batched Images (one per text) | Accepts Nested Images (one per text) | Accepts Nested Images (Multiple images per text, same number for each text) | Accepts Nested Images (Multiple images per text, different number for each text) |
|---|---|---|---|---|---|---|---|---|
| Pix2Struct | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| BLIP | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Donut | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| UDOP | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| GIT | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| KOSMOS-2 | ❌ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Idefics | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| FUYU | ❌ | ❌ | ✅ | ❌ | ✅ | ✅ | ❌ | ❌ |
| Llava | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| VipLlava | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| BLIP2 | ✅ | ❌ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ |
| Qwen2-VL | ✅ | ❌ | ✅ | ❌ | ✅* crash when padding set to True | ❌ | ✅ | ✅ |
| Chameleon | ✅ | ❌ | ✅ | ❌ | ❌* works when padding manually set to True | ✅ | ✅ | ✅ |
| Pixtral | ✅ | ✅ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ |
| Paligemma | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
| Llava-Next | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
| LLaVA-OneVision | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
| Mllama | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
| InstructBLIP | ✅ | ❌ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
| Idefics2 | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |
| Idefics3 | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ |

### Reproduction

For instance, with these inputs:
```python
import requests
from PIL import Image

image_ny = "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
image_chicago = "https://cdn.britannica.com/59/94459-050-DBA42467/Skyline-Chicago.jpg"
url_pokemon = "https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/pokemon.png"

image_pokemon = Image.open(requests.get(url_pokemon, stream=True).raw)
text_batched_image_tokens = [
    "<image> <image> Are these the same cities? If not what cities are these?",
    "<image> Describe what you see in this image. I see",
]

```

This will crash at the processor level:

```python
from transformers import pipeline

pipe = pipeline(task="image-text-to-text", model="llava-hf/llava-interleave-qwen-0.5b-hf")

pipe(
    images=[[image_ny, image_chicago], [image_pokemon]],
    text=text_batched_image_tokens,
)

```

But this will work:

```python
from transformers import pipeline

pipe = pipeline(task="image-text-to-text", model="llava-hf/llava-onevision-qwen2-0.5b-ov-hf")

pipe(
    images=[[image_ny, image_chicago], [image_pokemon]],
    text=text_batched_image_tokens,
)
```

### Expected behavior

Except for model-specific limitations or features, vlms used for similar tasks should accept a similar set of inputs format

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

cc @zucchini-nlp - I think this slipped under the radar the first time!

it is a tracker issue from Yoni 😄 

My bad! I was clicking through a lot of issues this morning, lol

Hi @zucchini-nlp and @Rocketknight1!

I'm picking this issue back up as I've started working on it. I've added a table that summarizes which image-text-to-text processors accept which inputs.

Regarding accepting URLs, PiL/arrays/tensors, and file paths as inputs: I was wondering if we should aim to make supporting all three standard? Right now, most processors accept only PiL/arrays/tensors; some also accept URLs, but I don't think any accept file paths. Is there a particular reason for that?

As for accepting batches of nested images: it makes sense that models that don't use image tokens also don't accept nested images (although they could accept one image per text input, like FUYU, and then raise an error if there are more). But it seems like models that do use image tokens should be capable of accepting any number of nested images. It looks like models such as LLaVA and VipLlava are being limited by their processors, which seem to be raising errors when they shouldn't be.

I might have gotten some of this wrong, so please feel free to correct me if that's the case 🤗.

Otherwise, I'm happy to start standardizing the processors, with the goal of getting everything a fully green line for models that use image tokens.

Yay, cool! Nice table to track the differences between processors. 

> Regarding accepting URLs, PiL/arrays/tensors, and file paths as inputs: I was wondering if we should aim to make supporting all three standard?

Personally, I don't think so. Seems it was enabled in the very first Idefics as part of a very pre-mature chat template similar to OpenAI format and got replicated in subsequent releases. I think we should support files/urls/etc but as part of the chat template

> As for accepting batches of nested images: it makes sense that models that don't use image tokens also don't accept nested images

Agreed, the older llava models had to support added for that and it is only the latest llava-ov which supports both. So we can make all processors accept images as nested dict as well as flat lists. 

Let me know if you need any help :)

> Personally, I don't think so. Seems it was enabled in the very first Idefics as part of a very pre-mature chat template similar to OpenAI format and got replicated in subsequent releases. I think we should support files/urls/etc but as part of the chat template

Any specific reason why we shouldn't support URL and file path in processor? Would it be to encourage users to use chat templates? 

> Would it be to encourage users to use chat templates?

Yes, that is the intuition I had, but I am open to supporting it without templates if we see user requests. WDYT?

> Yes, that is the intuition I had, but I am open to supporting it without templates if we see user requests. WDYT?

I haven't seen any user requests so I'm ok with keeping things as they are now to encourage the use of chat templates :) !

@yonigozlan btw, "Accepts Non-Nested Batched" means smth like `images=[im1, im2, im3], text=[text_for_im1, text_for_im2, text_for_im3]` only or any possible combinations like `images=[im1, im2, im3], text=[text_for_im1+im2, text_no_im, text_for_im3]`?