AutoModelForConditionalGeneration
Author: Dartvauder
Pull request: no
Labels: name: Feature request

### Feature request

A function that will do the same as AutoModelForCausalLM but only for models that also work with images

### Motivation

I've noticed more vLLM models that use the "ForConditionalGeneration" prefix in their models. I have code that works with any text models. Because of AutoModelForCausalLM, people can use any chat model. I'd like to do the same for models that work with images

### Your contribution

-

Comments:
We actually have the `MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES` to use with `AutoModelForVision2Seq` 🤗  should work for:
```python
MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES = OrderedDict(
    [
        ("blip", "BlipForConditionalGeneration"),
        ("blip-2", "Blip2ForConditionalGeneration"),
        ("chameleon", "ChameleonForConditionalGeneration"),
        ("git", "GitForCausalLM"),
        ("idefics2", "Idefics2ForConditionalGeneration"),
        ("idefics3", "Idefics3ForConditionalGeneration"),
        ("instructblip", "InstructBlipForConditionalGeneration"),
        ("instructblipvideo", "InstructBlipVideoForConditionalGeneration"),
        ("kosmos-2", "Kosmos2ForConditionalGeneration"),
        ("llava", "LlavaForConditionalGeneration"),
        ("llava_next", "LlavaNextForConditionalGeneration"),
        ("llava_next_video", "LlavaNextVideoForConditionalGeneration"),
        ("llava_onevision", "LlavaOnevisionForConditionalGeneration"),
        ("mllama", "MllamaForConditionalGeneration"),
        ("paligemma", "PaliGemmaForConditionalGeneration"),
        ("pix2struct", "Pix2StructForConditionalGeneration"),
        ("qwen2_vl", "Qwen2VLForConditionalGeneration"),
        ("video_llava", "VideoLlavaForConditionalGeneration"),
        ("vipllava", "VipLlavaForConditionalGeneration"),
        ("vision-encoder-decoder", "VisionEncoderDecoderModel"),
    ]
)
```

Several models, namely the following:

* `ChameleonForConditionalGeneration`
* `Idefics2ForConditionalGeneration`
* `Idefics3ForConditionalGeneration`
* `LlavaForConditionalGeneration`
*  `LlavaNextForConditionalGeneration`
* `LlavaNextVideoForConditionalGeneration`
* `LlavaOnevisionForConditionalGeneration`
* `PaliGemmaForConditionalGeneration`
* `VideoLlavaForConditionalGeneration`
* `VipLlavaForConditionalGeneration`
* `MllamaForConditionalGeneration` 
* `Qwen2VLForConditionalGeneration`

should not be added to this auto mapping, as they are `image-text-to-text`, not` image-to-text`, and also don't work with the `image-to-text` pipeline. They should be added to a new `AutoModelForImageTextToText` mapping which is WIP at https://github.com/huggingface/transformers/pull/32472.

cc @zucchini-nlp @yonigozlan 

Yep 👍🏻 

cc @zucchini-nlp curious why all new models are being added to this list, I'd prefer adding them to [this mapping](https://github.com/huggingface/transformers/blob/e3775539c845a2bb1c3d4283da73eae3ea05a55f/utils/check_repo.py#L165) until there's a dedicated autoclass for it. This way we avoid a deprecation warning when switching models to the new mapping

I think we are confused ourselveves / it's easy to overlook this! Let's refacto to make it better!

> We actually have the `MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES` to use with `AutoModelForVision2Seq` 🤗 should work for:
> 
> ```python
> MODEL_FOR_VISION_2_SEQ_MAPPING_NAMES = OrderedDict(
>     [
>         ("blip", "BlipForConditionalGeneration"),
>         ("blip-2", "Blip2ForConditionalGeneration"),
>         ("chameleon", "ChameleonForConditionalGeneration"),
>         ("git", "GitForCausalLM"),
>         ("idefics2", "Idefics2ForConditionalGeneration"),
>         ("idefics3", "Idefics3ForConditionalGeneration"),
>         ("instructblip", "InstructBlipForConditionalGeneration"),
>         ("instructblipvideo", "InstructBlipVideoForConditionalGeneration"),
>         ("kosmos-2", "Kosmos2ForConditionalGeneration"),
>         ("llava", "LlavaForConditionalGeneration"),
>         ("llava_next", "LlavaNextForConditionalGeneration"),
>         ("llava_next_video", "LlavaNextVideoForConditionalGeneration"),
>         ("llava_onevision", "LlavaOnevisionForConditionalGeneration"),
>         ("mllama", "MllamaForConditionalGeneration"),
>         ("paligemma", "PaliGemmaForConditionalGeneration"),
>         ("pix2struct", "Pix2StructForConditionalGeneration"),
>         ("qwen2_vl", "Qwen2VLForConditionalGeneration"),
>         ("video_llava", "VideoLlavaForConditionalGeneration"),
>         ("vipllava", "VipLlavaForConditionalGeneration"),
>         ("vision-encoder-decoder", "VisionEncoderDecoderModel"),
>     ]
> )
> ```

Thank you very much! This is what I needed.

Btw now that all the image-text-to-text models have had their processor standardized, I think we can safely merge this PR https://github.com/huggingface/transformers/pull/32472 !

> cc @zucchini-nlp curious why all new models are being added to this list, I'd prefer adding them to [this mapping](https://github.com/huggingface/transformers/blob/e3775539c845a2bb1c3d4283da73eae3ea05a55f/utils/check_repo.py#L165) until there's a dedicated autoclass for it. This way we avoid a deprecation warning when switching models to the new mapping

Yes, we are planning to move them to their respective auto-class. At the time when models were added, image-text-to-text was not even close to being shipped. And we were adding models in Vision2Seq for general consistency and as to not use "not functioning" auto-class. 
Thanks to @yonigozlan we'll be able to move to a fully functioning auto-class with a pipeline