Repository: huggingface/transformers
Pull request: [image-text-to-text pipeline] Accept a chat as a positional arg

# What does this PR do?

See title. 

Calling `pipe(chat)` is not working, but `pipe(text=chat)` is. Intuitivelly, if we prepare a chat message containing the full data, text and images, it should work as a positional argument.

### Example

```py
import torch
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="google/gemma-3-4b-it", torch_dtype=torch.bfloat16)
messages = [
    {
        "role": "system",
        "content": [{"type": "text", "text": "You are a helpful assistant."}]
    },
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"},
            {"type": "text", "text": "Describe this image in detail."}
        ]
    }
]
print(pipe(messages, max_new_tokens=100))
```

before:
```
...
File "/home/joao/transformers/src/transformers/pipelines/image_text_to_text.py", line 351, in preprocess
    images = load_images(inputs["images"], timeout=timeout)
KeyError: 'images'
```

this PR:
```
[{'input_text': [{'role': 'system', 'content': [{'type': 'text', 'text': 'You are a helpful assistant.'}]}, {'role': 'user', 'content': [{'type': 'image', 'image': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg'}, {'type': 'text', 'text': 'Describe this image in detail.'}]}], 'generated_text': [{'role': 'system', 'content': [{'type': 'text', 'text': 'You are a helpful assistant.'}]}, {'role': 'user', 'content': [{'type': 'image', 'image': 'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg'}, {'type': 'text', 'text': 'Describe this image in detail.'}]}, {'role': 'assistant', 'content': "Here's a detailed description of the image:\n\n**Overall Impression:**\n\nThe image is a close-up, vibrant photograph of a garden scene featuring several pink cosmos flowers and a bumblebee. It has a slightly shallow depth of field, which brings the bee and the central cosmos flower into sharp focus, while the background is softer.\n\n**Foreground:**\n\n*   **Cosmos Flower:** The main focus is a large, bright pink cosmos flower. It has five delicate, slightly ruffled petals"}]}]
```

Changed files:
filename: src/transformers/pipelines/image_text_to_text.py
patch: @@ -54,7 +54,9 @@ class Chat:
     to this format because the rest of the pipeline code tends to assume that lists of messages are
     actually a batch of samples rather than messages in the same conversation."""
 
-    def __init__(self, messages: Dict, images: Union[str, List[str], "Image.Image", List["Image.Image"]]):
+    def __init__(
+        self, messages: Dict, images: Optional[Union[str, List[str], "Image.Image", List["Image.Image"]]] = None
+    ):
         for message in messages:
             if not ("role" in message and "content" in message):
                 raise ValueError("When passing chat dicts as input, each dict must have a 'role' and 'content' key.")
@@ -241,7 +243,15 @@ def _sanitize_parameters(
     def __call__(
         self,
         images: Optional[
-            Union[str, List[str], List[List[str]], "Image.Image", List["Image.Image"], List[List["Image.Image"]]]
+            Union[
+                str,
+                List[str],
+                List[List[str]],
+                "Image.Image",
+                List["Image.Image"],
+                List[List["Image.Image"]],
+                List[dict],
+            ]
         ] = None,
         text: Optional[Union[str, List[str], List[dict]]] = None,
         **kwargs,
@@ -250,21 +260,22 @@ def __call__(
         Generate a text given text and the image(s) passed as inputs.
 
         Args:
-            images (`str`, `List[str]`, `PIL.Image or `List[PIL.Image]`):
+            images (`str`, `List[str]`, `PIL.Image, `List[PIL.Image]`, `List[Dict[str, Union[str, PIL.Image]]]`):
                 The pipeline handles three types of images:
 
                 - A string containing a HTTP(s) link pointing to an image
                 - A string containing a local path to an image
                 - An image loaded in PIL directly
 
-                The pipeline accepts either a single image or a batch of images.
+                The pipeline accepts either a single image or a batch of images. Finally, this pipeline also supports
+                the chat format (see `text`) containing images and text in this argument.
             text (str, List[str], `List[Dict[str, Union[str, PIL.Image]]]`):
-                The text to be used for generation. If a list of strings is passed, the length of the list should be the
-                same as the number of images. Text can also follow the chat format: a list of dictionaries where each
-                dictionary represents a message in a conversation. Each dictionary should have two keys: 'role' and
-                'content'. 'role' should be one of 'user', 'system' or 'assistant'. 'content' should be a list of dictionary
-                containing the text of the message and the type of the message. The type of the message can be either
-                'text' or 'image'. If the type is 'image', no text is needed.
+                The text to be used for generation. If a list of strings is passed, the length of the list should be
+                the same as the number of images. Text can also follow the chat format: a list of dictionaries where
+                each dictionary represents a message in a conversation. Each dictionary should have two keys: 'role'
+                and 'content'. 'role' should be one of 'user', 'system' or 'assistant'. 'content' should be a list of
+                dictionary containing the text of the message and the type of the message. The type of the message
+                can be either 'text' or 'image'. If the type is 'image', no text is needed.
             return_tensors (`bool`, *optional*, defaults to `False`):
                 Returns the tensors of predictions (as token indices) in the outputs. If set to
                 `True`, the decoded text is not returned.
@@ -281,8 +292,8 @@ def __call__(
                 `False` otherwise, but you can manually override that behaviour by setting this flag.
 
         Return:
-            A list or a list of list of `dict`: Each result comes as a dictionary with the following key (cannot return a combination
-            of both `generated_text` and `generated_token_ids`):
+            A list or a list of list of `dict`: Each result comes as a dictionary with the following key (cannot
+            return a combination of both `generated_text` and `generated_token_ids`):
 
             - **generated_text** (`str`, present when `return_text=True`) -- The generated text.
             - **generated_token_ids** (`torch.Tensor`, present when `return_tensors=True`) -- The token
@@ -291,17 +302,11 @@ def __call__(
         """
         if images is None and text is None:
             raise ValueError("You must at least provide either text or images.")
-        if images is not None and text is None and not valid_images(images):
-            """
-            Supports the following format
-            - {"image": image, "text": text}
-            - [{"image": image, "text": text}]
-            - Generator and datasets
-            This is a common pattern in other multimodal pipelines, so we support it here as well.
-            """
-            return super().__call__(images, **kwargs)
 
-        if isinstance(text, (list, tuple, KeyDataset)) and isinstance(text[0], (list, tuple, dict)):
+        def _is_chat(arg):
+            return isinstance(arg, (list, tuple, KeyDataset)) and isinstance(arg[0], (list, tuple, dict))
+
+        if _is_chat(text):
             # We have one or more prompts in list-of-dicts format, so this is chat mode
             if isinstance(text[0], dict):
                 return super().__call__(Chat(text, images), **kwargs)
@@ -311,11 +316,32 @@ def __call__(
                 chats = [Chat(chat, image) for chat, image in zip(text, images)]  # 🐈 🐈 🐈
                 return super().__call__(chats, **kwargs)
 
+        # Same as above, but the `images` argument contains the chat. This can happen e.g. is the user only passes a
+        # chat as a positional argument.
+        elif text is None and _is_chat(images):
+            # We have one or more prompts in list-of-dicts format, so this is chat mode
+            if isinstance(images[0], dict):
+                return super().__call__(Chat(images), **kwargs)
+            else:
+                chats = [Chat(image) for image in images]  # 🐈 🐈 🐈
+                return super().__call__(chats, **kwargs)
+
+        elif images is not None and text is None and not valid_images(images):
+            """
+            Supports the following format
+            - {"image": image, "text": text}
+            - [{"image": image, "text": text}]
+            - Generator and datasets
+            This is a common pattern in other multimodal pipelines, so we support it here as well.
+            """
+            return super().__call__(images, **kwargs)
+
         # encourage the user to use the chat format if supported
         if getattr(self.processor, "chat_template", None) is not None:
             logger.warning_once(
-                "The input data was not formatted as a chat with dicts containing 'role' and 'content' keys, even though this model supports chat. "
-                "Consider using the chat format for better results. For more information, see https://huggingface.co/docs/transformers/en/chat_templating"
+                "The input data was not formatted as a chat with dicts containing 'role' and 'content' keys, even "
+                "though this model supports chat. Consider using the chat format for better results. For more "
+                "information, see https://huggingface.co/docs/transformers/en/chat_templating"
             )
 
         # support text only generation