Add ImageTextToText pipeline
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds a deprecation warning when users pass a text prompt to the `image-to-text` pipeline, and recommends to leverage a new `image-text-to-text` pipeline instead.

This way, we keep `image-to-text` for only that said task, which means image as input and text as output. Example tasks include image captioning or optical character recognition (OCR). 

The new image-text-to-text pipeline assumes input = image + text, output = text.

## Usage

Usage is as follows:

```python
from transformers import pipeline


# OK:
# model_id = "microsoft/git-base-coco"
model_id = "Salesforce/blip-image-captioning-base"
# model_id = "Salesforce/blip2-opt-2.7b" ok, although it doesn't include the text prompt in the output
# model_id = "Salesforce/instructblip-flan-t5-xl" ok, although it doesn't include the text prompt in the output
# model_id = "llava-hf/llava-1.5-7b-hf"
# model_id = "adept/fuyu-8b"
model_id = "google/pix2struct-textcaps-base"
# model_id = "microsoft/udop-large"
# model_id = "naver-clova-ix/donut-base-finetuned-docvqa"
# model_id = "microsoft/kosmos-2-patch14-224"

pipe = pipeline(task="image-text-to-text", model=model_id)

outputs = pipe(
    images="http://images.cocodataset.org/val2017/000000039769.jpg",
    text="USER: <image>\nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT:",
    max_new_tokens=200,
)

print(outputs)
```

## Supported models

This has been tested on:

- [x] GIT
- [x] BLIP
- [x] BLIP-2
- [x] IDEFICS
- [x] InstructBLIP
- [x] LLaVa
- [x] Fuyu
- [x] Pix2Struct
- [x] UDOP
- [x] Donut
- [x] KOSMOS-2

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29572). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I just tested it, thanks a lot for the PR!

The pipeline is returning weird output for KOSMOS-2 and Fuyu-8B (and raises an error for Llava) and the output doesn't seem to be postprocessed.
```python
from transformers import pipeline
import torch 

model_id = "microsoft/kosmos-2-patch14-224"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
pipe = pipeline(task="image-text-to-text", model=model_id, device=device)

outputs = pipe(
    images="http://images.cocodataset.org/val2017/000000039769.jpg",
    text="<grounding> Explain what the cats in the image are doing.",
    max_new_tokens=40
)

print(outputs)
# [{'generated_text': '<image>. the, to and of as in I that\' for is was- on' it with The as at bet he have from by are " you his " this said not has an ( but had we her they will my or were their): up about out who one all been she can more would It</image><grounding> Explain what the cats in the image are doing.<phrase> The two cats</phrase><object><patch_index_0049><patch_index_0799></delimiter_of_multi_objects/><patch_index_0096><patch_index_1007></object> are laying on a pink blanket, sleeping next to each other. One cat is on its back, while the other is laying on its side'}]
```

I can reproduce it with Fuyu-8B as well, I feel like something around postprocessing might be causing it. 
```python
from transformers import pipeline
import torch 

model_id = "adept/fuyu-8b"
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
pipe = pipeline(task="image-text-to-text", model=model_id, device=device)

outputs = pipe(
    images="http://images.cocodataset.org/val2017/000000039769.jpg",
    text="<grounding> Explain what the cats in the image are doing.",
    max_new_tokens=40
)

print(outputs)

# [{'generated_text': '|SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||SPEAKER||NEWLINE|
 <grounding> Explain what the cats in the image are doing.\x04 In the image, two cats are lying on a pink blanket, seemingly sleeping or resting.\n'}]
```

Llava-1.5B raises an error. 
```
from transformers import pipeline
import torch 

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
mm_pipeline = pipeline("image-text-to-text", model="llava-hf/llava-1.5-7b-hf", device=device)

mm_pipeline("https://huggingface.co/spaces/llava-hf/llava-4bit/resolve/main/examples/baklava.png", text="How to make this pastry?")
```
`ValueError: The input provided to the model are wrong. The number of image tokens is 0 while the number of image given to the model is 1. This prevents correct indexing and breaks batch generation.`

@merveenoyan yes the outputs are as expected, they output what you would also get by running `processor.batch_decode(generated_ids)` for these models.

The reason KOSMOS-2 outputs a set of random tokens first is because it also outputs tokens for the image patches, which can be discarded.

Similarly, Fuyu-8B first outputs a large set of |SPEAKER| tokens before the actual answer.

The reason you're getting an error for Llava-1.5B is because you're not including an \<image> token in the prompt, it's recommended to leverage this prompt template: `"USER: <image>\nWhat are these?\nASSISTANT:"`.

@amyeroberts I'd like to assign you for review, but the CI keeps failing with unrelated changes (cc @ydshieh):
```python
TypeError: snapshot_download() got an unexpected keyword argument '_from_pipeline'
```
I've rebased with main already several times. Other than that, the PR is ready for review.



Thanks for working on this! 

Just skimming the PR, there's code which shouldn't be be pushed - please make sure to look at the diff in the PR to catch this before asking for review. 

Once it's ready for review, I'll give a more in-depth look. At a high-level, I'm a bit concerned about the addition of the auto model, when from the preprocessing and processing steps there isn't complete unity in the inputs and outputs e.g. adding `decoder_input_ids` for donut.

As discussed a week or two ago, it would be good to have a method for all the model's processors which handle the postprocessing of generation outputs for these models. cc @molbap.

Removed the script, PR should be ready for review now

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Is this PR still planned? It would be useful, see e.g. these internal threads ([thread1](https://huggingface.slack.com/archives/C01EDEAV0NT/p1720004277653459?thread_ts=1711469424.532149&cid=C01EDEAV0NT), [thread2](https://huggingface.slack.com/archives/C01N44FJDHT/p1719998786039139?thread_ts=1719997770.767279&cid=C01N44FJDHT)) @NielsRogge 

Yes it's still planned, I just don't have the bandwidth to work on it now. I hope @molbap @zucchini-nlp have.

Although it would require to have the refactor of all multimodal processors first before this can land.