Add support for OpenAI api "image_url" input in chat for image-text-to-text pipeline
Author: yonigozlan
Pull request: yes

# What does this PR do?
The huggingface inference API uses an OpenAI like chat template for image_url input: https://huggingface.co/docs/api-inference/tasks/image-text-to-text#using-the-api

This adds support for such image inputs in chats in the image-text-to-text pipeline.
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->


## Who can review?

Cc @zucchini-nlp @NielsRogge 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34562). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the review! Agreed that the Transformers chat template looks cleaner and easier to use, and nice to see it's getting even better with your PR. 
However it does look like the pipelines are the one place where we need support for both transformers and TGI chat templates format, so I can improve this PR to do so.

@zucchini-nlp I think modifications inside `apply_chat_template` to do image loading are a good idea! We should ensure that the input to Jinja doesn't change though.

For the formats of content keys with PIL images vs. image paths etc. my preference is always to be close to the OpenAI API, and then add extra features on top of that if we need.

Thanks, i think in that case we can remove the extra nesting in dicts which will still be close to OpenAI format. And yeah, we can consider passing load time params through the template dict, currently we dont have any options except for number of frames in loading videos or sampling rate in audios, which are usually same for all video/audio in the conversation

@yonigozlan wdyt? we can update the PR with these in mind

> WDYT if we remove extra nesting but leave the keys to define "url" or "path"? That is what I had in mind before simplifying it further in linked PR
> 
> Otherwise the format looks a bit weird to me as follows:
> 
> URL: content: ["type": "image_url", "image_url": {"url": 'https:// or "data:base64-image"'}]
> PIL Image: content: ["type": "image", "image": {"image": PIL.Image}]
> Path but not here yet?: content: ["type": "image_path", "image_path": {"path": "my-local-path.png"}]

Looking again at the OpenAI API doc, it seems that the only way to pass in images is through "image_url", even if the image is local, as shown [here ](https://platform.openai.com/docs/guides/vision#uploading-base64-encoded-images).
Same for TGI and API inference.

I also agree that the simplified format of transformers using only "image", whether we have an url, PIL image or path makes more sense and is more user friendly. So I don't think we need to add support for anything else other than the openai/TGI nested "image_url" format in this pipeline. It seems to me that we should keep the nesting in that specific case, because the goal is to be fully compatible with the openai/tgi format.

Just to sync after the offsite so we have same formats with https://github.com/huggingface/transformers/pull/34275, do I remember correctly that we will:
1. use unnested dict but with typing so each image might be a url, path, base64, or just PIL/array/tensor
2. can support OpenAI format which is only for URLs with a bit preprocessing, but only as a possible format for pipelines/endpoints while still keeping the general chat template format with unnested dicts 

I agree OpenAI format is kinda standard for most users and we use it in TGI/endpoints etc., but sadly it is very inconvenient and is made to support only URLs

Sounds good to me @zucchini-nlp!
Just to make sure about 1. , we would still have:
```
 {
     "role": "user",
     "content": [
         {
             "type": "image",
             "image": image, # where image can be a url, path, base64, or PIL/array/tensor
         },
         {"type": "text", "text": "Describe this image."},
     ],
 },
```

Or are we explicitly typing `"image"`?

@yonigozlan no, let's keep the explicit typing only and OpenAI format (which will be available only through pipelines for now)

> @yonigozlan no, let's keep the explicit typing only and OpenAI format (which will be available only through pipelines for now)

Ok I see, I made some changes to reflect this in this PR then :)