IdeficsImageProcessor raises unexpected ValueError
Author: Kamichanw
Pull request: no
Labels: name: bug

name: Vision

### System Info

- `transformers` version: 4.47.1
- Platform: Linux-5.15.0-126-generic-x86_64-with-glibc2.35
- Python version: 3.13.1
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.4.5
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

@amyeroberts @qubvel @zucchini-nlp 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I was running the following code:
```python
# ignore imports...

checkpoint = "HuggingFaceM4/idefics-9b-instruct"
model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
processor = AutoProcessor.from_pretrained(checkpoint)
images = [
    PIL.Image.open(...),
    PIL.Image.open(...),
]
batched_text = ["<image><image>", "<image><image>"] 
batched_image = [images, images]
processor(batched_image, batched_text)
```
I got the following exception:
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 5
      3 batched_text =  ["<image><image>", "<image><image>"] 
      4 batched_images = [images, images]
----> 5 processor(batched_images, batched_text)

File ~/miniconda3/envs/icl/lib/python3.13/site-packages/transformers/utils/deprecation.py:165, in deprecate_kwarg.<locals>.wrapper.<locals>.wrapped_func(*args, **kwargs)
    161 elif minimum_action in (Action.NOTIFY, Action.NOTIFY_ALWAYS):
    162     # DeprecationWarning is ignored by default, so we use FutureWarning instead
    163     warnings.warn(message, FutureWarning, stacklevel=2)
--> 165 return func(*args, **kwargs)

File ~/miniconda3/envs/icl/lib/python3.13/site-packages/transformers/models/idefics/processing_idefics.py:430, in IdeficsProcessor.__call__(self, images, text, audio, videos, **kwargs)
    427 if add_eos_token:
    428     full_text += self.tokenizer.eos_token
--> 430 image_objects = self.image_processor(image_objects, **output_kwargs["images_kwargs"])
    432 all_prompts.append(full_text)
    433 all_images.append(image_objects)

File ~/miniconda3/envs/icl/lib/python3.13/site-packages/transformers/image_processing_utils.py:41, in BaseImageProcessor.__call__(self, images, **kwargs)
     39 def __call__(self, images, **kwargs) -> BatchFeature:
     40     """Preprocess an image or a batch of images."""
---> 41     return self.preprocess(images, **kwargs)

File ~/miniconda3/envs/icl/lib/python3.13/site-packages/transformers/models/idefics/image_processing_idefics.py:134, in IdeficsImageProcessor.preprocess(self, images, image_num_channels, image_size, image_mean, image_std, transform, return_tensors, **kwargs)
    131 if isinstance(images, list) and len(images) == 0:
    132     return []
--> 134 images = make_list_of_images(images)
    136 if not valid_images(images):
    137     raise ValueError(
    138         "Invalid image type. Must be of type PIL.Image.Image, numpy.ndarray, "
    139         "torch.Tensor, tf.Tensor or jax.ndarray."
    140     )

File ~/miniconda3/envs/icl/lib/python3.13/site-packages/transformers/image_utils.py:206, in make_list_of_images(images, expected_ndims)
    201         raise ValueError(
    202             f"Invalid image shape. Expected either {expected_ndims + 1} or {expected_ndims} dimensions, but got"
    203             f" {images.ndim} dimensions."
    204         )
    205     return images
--> 206 raise ValueError(
    207     "Invalid image type. Expected either PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or "
    208     f"jax.ndarray, but got {type(images)}."
    209 )

ValueError: Invalid image type. Expected either PIL.Image.Image, numpy.ndarray, torch.Tensor, tf.Tensor or jax.ndarray, but got <class 'list'>.
```
Note that this bug is not exist in v4.45.2.

### Expected behavior

Even I have followed instruction provided [here](https://github.com/huggingface/transformers/blob/8f38f58f3de5a35f9b8505e9b48985dce5470985/src/transformers/models/idefics/processing_idefics.py#L354-L358), I still got such a bug, why?

Comments:
@Kamichanw Before v4.45.2 Idefics did not accept images/text as two separate kwargs and thus was working only with inputs formatted as chat by interleaving text and image within one list. See below as an example. We've added an option to pass images and text separately, but the nested input is not supported yet indeed

https://github.com/huggingface/transformers/blob/8f38f58f3de5a35f9b8505e9b48985dce5470985/src/transformers/models/idefics/processing_idefics.py#L291-L298

There is a tracker PR here (https://github.com/huggingface/transformers/issues/34545) to track progress on making all VLM processors standard and thus accept different input types. @yonigozlan will be working on that and in the meanwhile you can still pass your inputs as interleaved prompts, same way you were doing before v4.45. Backward compatibility should have been preserved