One Question about BlipForConditionalGeneration
Author: Yingshu-Li
Pull request: no

### System Info

I create a Blip model by: 
`BlipModel = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")`

I want to get the hidden state of the model output. I got this problem:

I just used the `BlipModel(pixel_values = ipt)` for inference part. 
I create a dummy input by `ipt = torch.randn((1,3,384,384))`. 
When the input batch size is 1, everything works fine.

However, when i tried to change the input's batch size to other number, like 2, `ipt = torch.randn((2,3,384,384))`. Then i got this kind of error:

**ValueError: Expected input batch_size (2) to match target batch_size (1).**


### Who can help?

@younesbelkada,  @ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. create a model by `BlipModel = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")`
2. create a dummy input by `ipt = torch.randn((2,3,384,384))`
3. get the output of the model by `BlipModel(ipt)`

It will get the error.

### Expected behavior

I want to get the hidden state of the model output. I give you the correct output when i set the batch as 1. When I set the batch to other number, it will get error.


```
BlipForConditionalGenerationModelOutput(loss=tensor(nan, grad_fn=<AddBackward0>), decoder_logits=tensor([[[-2.4062, -2.4062, -2.4062,  ..., -2.4061, -2.4062, -2.4062]]],
       grad_fn=<ViewBackward0>), image_embeds=tensor([[[-0.7771, -0.0999,  0.0320,  ..., -0.6212,  0.8770, -0.1978],
         [-0.9081, -0.1407,  0.1390,  ..., -0.4231,  0.5914, -0.1464],
         [-1.0026,  0.0212,  0.4119,  ..., -0.5520,  0.5102, -0.1100],
         ...,
         [-1.2060, -0.0290,  0.0165,  ..., -0.5280,  0.3483, -0.0130],
         [-1.0668,  0.4398,  0.3717,  ..., -0.7589,  0.0796,  0.1294],
         [-1.0077, -0.2549, -0.1857,  ..., -0.5054,  0.6910, -0.2062]]],
       grad_fn=<NativeLayerNormBackward0>), last_hidden_state=tensor([[[-0.7771, -0.0999,  0.0320,  ..., -0.6212,  0.8770, -0.1978],
         [-0.9081, -0.1407,  0.1390,  ..., -0.4231,  0.5914, -0.1464],
         [-1.0026,  0.0212,  0.4119,  ..., -0.5520,  0.5102, -0.1100],
         ...,
         [-1.2060, -0.0290,  0.0165,  ..., -0.5280,  0.3483, -0.0130],
         [-1.0668,  0.4398,  0.3717,  ..., -0.7589,  0.0796,  0.1294],
         [-1.0077, -0.2549, -0.1857,  ..., -0.5054,  0.6910, -0.2062]]],
       grad_fn=<NativeLayerNormBackward0>), hidden_states=None, attentions=None)
```

Comments:
Hi @Yingshu97 
Thanks for the issue! 
there are few things to keep in mind here:

1- If you want to use `BlipForConditionalGeneration` as a standalone model to retrieve the hidden states and loss value, you need to also pass `input_ids` values, as Blip uses cross attention between textual and visual input. In the provided snippet you did not pass any `input_ids`. If I correctly pass pixel values with a batch size of 2 together with random input ids that have a batch size of 2 it works as expected. The only way to generate captions without having to pass `input_ids` is to call `.generate` method that will initialize the `input_ids` with `decoder_input_ids` and `eos_token_id`.
2- Make sure to use at least the latest release of `transformers`. `pip install --upgrade transformers`

The snippet I used is:
```python
import torch
from transformers import BlipForConditionalGeneration

ipt = torch.randn((2, 3, 384, 384))
input_ids = torch.LongTensor([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 11, 12, 13, 14, 15, 16, 17, 18, 19]])
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")

out = model(pixel_values=ipt, input_ids=input_ids)
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.