[`Generate`] Add conditional generation for multimodal models
Author: younesbelkada
Pull request: yes

# Motivation

Some multi-modal models (precisely, image to text models) can perform better if conditional text is passed. This simply means that `input_ids` created by `_prepare_decoder_input_ids_for_generation` is concatenated with `input_ids` that is passed along `model_kwargs`.

This PR aims to add the support for this feature for `VisionEncoderDecoderModel`, precisely now this script should be able to run without any problem:

```python
import torch
import requests
from PIL import Image
from transformers import ViTFeatureExtractor, AutoTokenizer, VisionEncoderDecoderModel


loc = "ydshieh/vit-gpt2-coco-en"

feature_extractor = ViTFeatureExtractor.from_pretrained(loc)
tokenizer = AutoTokenizer.from_pretrained(loc)
model = VisionEncoderDecoderModel.from_pretrained(loc)
model.eval()


def predict(image, text):
    pixel_values = feature_extractor(images=image, return_tensors="pt").pixel_values
    input_ids = tokenizer(text, return_tensors="pt").input_ids

    with torch.no_grad():
        output_ids = model.generate(pixel_values, input_ids=input_ids, max_length=16, num_beams=4, return_dict_in_generate=True).sequences

    preds = tokenizer.batch_decode(output_ids, skip_special_tokens=True)
    preds = [pred.strip() for pred in preds]

    return preds


# We will verify our results on an image of cute cats
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
text = "an image of"
with Image.open(requests.get(url, stream=True).raw) as image:
    preds = predict(image, text)

print(preds)
>>> ['an image of two cats sleeping on a bed']
```

cc @gante 

Related: https://github.com/huggingface/transformers/pull/22423

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As this is slightly experimental, I ran blip slow tests that also includes conditional generation tests and they all pass, will merge!

Hi @younesbelkada , I get a following error during the training stage when providing decoder_input_ids argument. Does this modification only works for inference stage or training too ?

```
-> 3029 return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
ValueError: Expected input batch_size (18) to match target batch_size (512).
```
I used batch_size 2 and beam_size 10

For a batch of examples (during training or inference), does the **input_ids** has to be same shape with padding even though each example prefix can be different length ? 

Does `input_ids = tokenizer(text, return_tensors="pt").input_ids` has to exclude special_tokens by giving the argument `add_special_tokens=False`



@younesbelkada @gante @sgugger  When I inspected the intermediate outputs during the training, `decoder_input_ids` being shape [2, 8] and `logits` being shape [2, 8, 64002], where batch_size is 2 and prefix length is 8. Looks like `decoder_outputs = self.decoder()` is not predicting anything else right after prefix given `decoder_input_ids`

Hey @cramraj8 -- would you be able to open a new issue, containing a short self-contained script so we can reproduce it? :)