idefics2 fails when first element of batch contains no images
Author: maximilianmordig
Pull request: no

### System Info

- `transformers` version: 4.41.2
- Platform: Linux-6.2.0-36-generic-x86_64-with-glibc2.35
- Python version: 3.10.11
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.31.0
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: fp16
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
        - dynamo_config: {'dynamo_backend': 'INDUCTOR', 'dynamo_mode': 'default', 'dynamo_use_dynamic': False, 'dynamo_use_fullgraph': False}
- PyTorch version (GPU?): 2.3.1.post300 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no/not yet

### Who can help?

Path: `transformers/models/idefics2/image_processing_idefics2.py`

make_list_of_images accesses `images[0][0]`, test cases:
```
make_list_of_images([[], []])
make_list_of_images([[], [Image.new("RGB", (100, 100))]])
```
example patch:
```
def patched_image_processing_idefics2(images: ImageInput) -> List[List[np.ndarray]]:
    """
    Convert a single image or a list of images to a list of numpy arrays.

    Args:
        images (`ImageInput`):
            A single image or a list of images.

    Returns:
        A list of numpy arrays.
    """
    # fixes bug when first element in image list does not contain an image, by searching for the first non-empty idx
    
    # If it's a single image, convert it to a list of lists
    if is_valid_image(images):
        images = [[images]]
    # If it's a list of images, it's a single batch, so convert it to a list of lists
    else:
        first_non_empty_idx = next((i for i, imgs in enumerate(images) if len(imgs) > 0), None)
        if first_non_empty_idx is not None:
            if isinstance(images, (list, tuple)) and len(images) > 0 and is_valid_image(images[first_non_empty_idx]):
                images = [images]
            # If it's a list of batches, it's already in the right format
            elif (
                isinstance(images, (list, tuple))
                and len(images) > 0
                and isinstance(images[first_non_empty_idx], (list, tuple))
                and is_valid_image(images[first_non_empty_idx][0])
            ):
                pass
            else:
                raise ValueError(
                    "Invalid input type. Must be a single image, a list of images, or a list of batches of images."
                )
    return images
```

Similarly for `is_scaled_image(images_list[0][0])` in `transformers/models/idefics2/image_processing_idefics2.py`
There may be more issues.

For a temporary workaround, what other models text+vision -> text models do you recommend?

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```
from transformers.models.idefics2.image_processing_idefics2 import make_list_of_images
from PIL import Image

make_list_of_images([[], []])
make_list_of_images([[], [Image.new("RGB", (100, 100))]])


# similarly for the processor
# processor(text=..., images=[[], []])
```
### Expected behavior

should not lead to exception

Comments:
Hi @maximilianmordig, thanks for opening this issue! 

Just to make sure I've understood correctly, you want to be able to pass in a batch of inputs to the model, some of which will not have any images? 

Yes, the number of images in a batch may be any number >= 0, so no images at all is possible.

Btw, `make_list_of_images` is not really supposed to be used on it's own (not part of the public API), and using the processor with text only should already work 

@maximilianmordig Would you like to open a PR with the suggested fix? cc @yonigozlan 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.