Suspected Blip2ForConditionalGeneration attention mask not working properly 
Author: nnethercott
Pull request: no

### System Info

- `transformers` version: 4.34.1
- Platform: Linux-6.2.0-35-generic-x86_64-with-glibc2.37
- Python version: 3.10.13
- Huggingface_hub version: 0.17.3
- Safetensors version: 0.4.0
- Accelerate version: 0.24.0
- Accelerate config: 	- compute_environment: LOCAL_MACHINE
	- distributed_type: MULTI_GPU
	- mixed_precision: no
	- use_cpu: False
	- debug: False
	- num_processes: 4
	- machine_rank: 0
	- num_machines: 1
	- rdzv_backend: static
	- same_network: False
	- main_training_function: main
	- downcast_bf16: False
	- tpu_use_cluster: False
	- tpu_use_sudo: False
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import Blip2Processor, Blip2ForConditionalGeneration
import requests
from PIL import Image

base_model_id = "Salesforce/blip2-opt-2.7b"
model = Blip2ForConditionalGeneration.from_pretrained(base_model_id, device_map = "auto")
processor = Blip2Processor.from_pretrained(base_model_id)

# from provided huggingface example -> https://huggingface.co/Salesforce/blip2-opt-2.7b
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
question1 = "how many dogs are in the picture?"
question2 = "is there a dog here?"

# batching
inputs = processor([raw_image,raw_image], [question1, question2], padding = True, return_tensors="pt")

# both separate 
i0 = processor([raw_image], [question1], return_tensors="pt")
i1 = processor([raw_image], [question2], return_tensors="pt")

with torch.no_grad():
    out = model(**inputs)
    o0 = model(**i0)
    o1 = model(**i1)
    
    
logits = out['logits']
l0 = o0['logits']
l1 = o1['logits']

# outputs False in both cases 
print(torch.allclose(logits[0][:l0.shape[1]].unsqueeze(0), l0))
print(torch.allclose(logits[1][:l1.shape[1]].unsqueeze(0), l1))

```

### Expected behavior

Whether or not inputs are batched should not influence the outputs of the forward pass in a deterministic model.  I expect the dimensions of the logits obtained from the batched input (truncated accordingly when padding was applied) to match up with the logits obtained from passing each sample through one at a time. Otherwise model decoding behaviour becomes undefined given whatever batch size you feed the model with. 

e.g. example from above 
```python
# should output True
print(torch.allclose(logits[0][:l0.shape[1]].unsqueeze(0), l0))
print(torch.allclose(logits[1][:l1.shape[1]].unsqueeze(0), l1))
```

Comments:
Another fun example 

Decoding each question one at a time
```python
prompt = "Question: {} Answer:"

inputs = processor(raw_image, prompt.format(question1), return_tensors = 'pt')
out = model.generate(**inputs)

print(processor.tokenizer.decode(out[0], skip_special_tokens = True).strip())

>> 'none'


inputs = processor(raw_image, prompt.format(question2), return_tensors = 'pt')
out = model.generate(**inputs)

print(processor.tokenizer.decode(out[0], skip_special_tokens = True).strip())

>> 'yes, there is a dog here'

```
Now batch them together 

```python 
inputs = processor([raw_image,raw_image], [prompt.format(question1), prompt.format(question2)], padding = True, return_tensors="pt")

out = model.generate(**inputs)
[s.strip() for s in processor.tokenizer.batch_decode(out, skip_special_tokens = True)]

>> ['none', '']
```




I have the same problem. I followed to try out the example code provided by BLIP-2, yet nothing, no results showed up on the console

```
import requests
from PIL import Image
from transformers import Blip2Processor, Blip2ForConditionalGeneration
import torch

processor = Blip2Processor.from_pretrained(
    "Salesforce/blip2-opt-2.7b",
    cache_dir="models/",
    resume_download=True,
    offload_folder="offload"
)

model = Blip2ForConditionalGeneration.from_pretrained(
    "Salesforce/blip2-opt-2.7b",
    device_map="auto",
    cache_dir="models/",
    resume_download=True,
    offload_folder="offload"
    )

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

question = "how many dogs are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt").to("cuda", torch.float16)

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True).strip())

# >>>
# Expected:
# >>> There is one dog.
```

> I have the same problem. I followed to try out the example code provided by BLIP-2, yet nothing, no results showed up on the console
> 
> ```
> import requests
> from PIL import Image
> from transformers import Blip2Processor, Blip2ForConditionalGeneration
> import torch
> 
> processor = Blip2Processor.from_pretrained(
>     "Salesforce/blip2-opt-2.7b",
>     cache_dir="models/",
>     resume_download=True,
>     offload_folder="offload"
> )
> 
> model = Blip2ForConditionalGeneration.from_pretrained(
>     "Salesforce/blip2-opt-2.7b",
>     device_map="auto",
>     cache_dir="models/",
>     resume_download=True,
>     offload_folder="offload"
>     )
> 
> img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg'
> raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
> 
> question = "how many dogs are in the picture?"
> inputs = processor(raw_image, question, return_tensors="pt").to("cuda", torch.float16)
> 
> out = model.generate(**inputs)
> print(processor.decode(out[0], skip_special_tokens=True).strip())
> 
> # >>>
> # Expected:
> # >>> There is one dog.
> ```

Have you tried formatting the question before tokenization? In the paper they suggested using a prompt like "Question: {question} Answer:" so that your input would go from `how many dogs are in the picture?` to `Question: how many dogs are there in the picture? Answer:`.  

Similar to the thread in the community section for the model here -> https://huggingface.co/Salesforce/blip2-opt-2.7b/discussions/15

My issue is that the conditional outputs of the model change based on the batch size which messes with training and generating for batches of inputs :pensive:

These problems can occur due to two reasons:

1. The tokenizer padding side has not been configured. If you are using the OPT checkpoint, make sure to set `tokenizer.padding_side = left`.

2. The question needs to be formatted using the template "Question: {question} Answer: {}" for OPT models to generate a meaningful output.

By addressing these two issues, you should be able to resolve the error.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.