`RuntimeError` when running batched inference for `Salesforce/blip2-opt-2.7b`
Author: rsomani95
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.27.0.dev0
- Platform: Linux-5.15.0-57-generic-x86_64-with-glibc2.10
- Python version: 3.8.12
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.11.0a0+b6df043 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

I'm trying to run batched inference using BLIP v2 with a slight modification from the [example script](https://huggingface.co/docs/transformers/main/en/model_doc/blip-2#transformers.Blip2ForConditionalGeneration.forward.example). Anything above a batch size of produces a `RuntimeError` due to a shape mismatch in a `torch.cat` operation. The error stems from the `.generate()` function inside the language model componenet of BLIP v2.

cc  @NielsRogge @gante


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

## Code To Reproduce

```python
from PIL import Image
import requests
from transformers import Blip2Processor, Blip2ForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"

processor = Blip2Processor.from_pretrained("Salesforce/blip2-opt-2.7b")
model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-opt-2.7b", torch_dtype=torch.float16
)
model.to(device)
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# ---------------------- Change made here ---------------------- #
# Passing `images=[image, image]` instead of `images=image` for testing batched inference
inputs = processor(images=[image, image], return_tensors="pt").to(device, torch.float16)
# ---------------------------------------------------------------- #

generated_ids = model.generate(**inputs)
```


## Error Stack Trace

```python
╭──────────────────────────── Traceback (most recent call last) ────────────────────────────╮
│                                                                                           │
│ /tmp/ipykernel_1000/1893006406.py:18 in <module>                                          │
│                                                                                           │
│ [Errno 2] No such file or directory: '/tmp/ipykernel_1000/1893006406.py'                  │
│ /opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py:28 in decorate_context │
│                                                                                           │
│    25 │   │   @functools.wraps(func)                                                      │
│    26 │   │   def decorate_context(*args, **kwargs):                                      │
│    27 │   │   │   with self.__class__():                                                  │
│ ❱  28 │   │   │   │   return func(*args, **kwargs)                                        │
│    29 │   │   return cast(F, decorate_context)                                            │
│    30 │                                                                                   │
│    31 │   def _wrap_generator(self, func):                                                │
│                                                                                           │
│ /home/synopsis/git/transformers/src/transformers/models/blip_2/modeling_blip_2.py:1421 in │
│ generate                                                                                  │
│                                                                                           │
│   1418 │   │   inputs_embeds = self.language_model.get_input_embeddings()(input_ids)      │
│   1419 │   │   inputs_embeds = torch.cat([language_model_inputs, inputs_embeds], dim=1)   │
│   1420 │   │                                                                              │
│ ❱ 1421 │   │   outputs = self.language_model.generate(                                    │
│   1422 │   │   │   inputs_embeds=inputs_embeds,                                           │
│   1423 │   │   │   attention_mask=attention_mask,                                         │
│   1424 │   │   │   **generate_kwargs,                                                     │
│                                                                                           │
│ /opt/conda/lib/python3.8/site-packages/torch/autograd/grad_mode.py:28 in decorate_context │
│                                                                                           │
│    25 │   │   @functools.wraps(func)                                                      │
│    26 │   │   def decorate_context(*args, **kwargs):                                      │
│    27 │   │   │   with self.__class__():                                                  │
│ ❱  28 │   │   │   │   return func(*args, **kwargs)                                        │
│    29 │   │   return cast(F, decorate_context)                                            │
│    30 │                                                                                   │
│    31 │   def _wrap_generator(self, func):                                                │
│                                                                                           │
│ /home/synopsis/git/transformers/src/transformers/generation/utils.py:1386 in generate     │
│                                                                                           │
│   1383 │   │   │   │   )                                                                  │
│   1384 │   │   │                                                                          │
│   1385 │   │   │   # 11. run greedy search                                                │
│ ❱ 1386 │   │   │   return self.greedy_search(                                             │
│   1387 │   │   │   │   input_ids,                                                         │
│   1388 │   │   │   │   logits_processor=logits_processor,                                 │
│   1389 │   │   │   │   stopping_criteria=stopping_criteria,                               │
│                                                                                           │
│ /home/synopsis/git/transformers/src/transformers/generation/utils.py:2224 in              │
│ greedy_search                                                                             │
│                                                                                           │
│   2221 │   │   │   │   next_tokens = next_tokens * unfinished_sequences + pad_token_id *  │
│   2222 │   │   │                                                                          │
│   2223 │   │   │   # update generated ids, model inputs, and length for next step         │
│ ❱ 2224 │   │   │   input_ids = torch.cat([input_ids, next_tokens[:, None]], dim=-1)       │
│   2225 │   │   │   model_kwargs = self._update_model_kwargs_for_generation(               │
│   2226 │   │   │   │   outputs, model_kwargs, is_encoder_decoder=self.config.is_encoder_d │
│   2227 │   │   │   )                                                                      │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size
2 for tensor number 1 in the list.
```

### Expected behavior

I'd expect being able to run batched inference for a batch size > 1

Comments:
Hey there @rsomani95 👋  I believe I know what the root cause is, on it :)

@rsomani95 if you install from `main`, it should be working now :) 

Note that batched generation + beam search is still not working, and is being tracked [here](https://github.com/huggingface/transformers/issues/21599)

@gante great, thanks a bunch for the quick work!