BLIP2 inference error: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cuda:2
Author: YongLD
Pull request: no

### System Info

**Describe the bug**
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cuda:2.

**Screenshots**
```shell
Traceback (most recent call last):
  File "/home/cike/ldy/ner/test-blip2-1.py", line 18, in <module>
    out = model.generate(**inputs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cike/anaconda3/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
     ......
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/cike/.local/lib/python3.11/site-packages/transformers/generation/utils.py", line 2494, in greedy_search
    next_tokens = next_tokens * unfinished_sequences + pad_token_id * (1 - unfinished_sequences)
                  ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:7 and cuda:2
```

**System info (please complete the following information):**
 - OS: 18.04.2 LTS
 - One mechine with 8x tesla p100-pcie-16gb

How can I fix this bug?



### Who can help?

@pacman100

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

**To Reproduce**
 I am trying to enable multi-gpus inference on the BLIP2 model.
I tried the following code snippet:
```python
model_path = "/home/cike/.cache/huggingface/hub/models--Salesforce--blip2-flan-t5-xl/snapshots/cc2bb7bce2f7d4d1c37753c7e9c05a443a226614/"
processor = Blip2Processor.from_pretrained(model_path)
model = Blip2ForConditionalGeneration.from_pretrained(model_path, device_map="auto")

img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')

question = "how many dogs are in the picture?"
inputs = processor(raw_image, question, return_tensors="pt").to("cuda")

print("model: ",model.hf_device_map)

out = model.generate(**inputs)
print(processor.decode(out[0], skip_special_tokens=True))
``` 

### Expected behavior

The BLIP2 model loads and runs successfully on multi-GPUs.

Comments:
pinging @SunMarc and @younesbelkada as well! 

Hi @YongLD, please make sure to have the latest version of transformers. We fixed a similar issue to this one in the [past](https://github.com/huggingface/transformers/pull/25735). On my side, I'm able to run on 2 GPUs. LMK how it goes. If it doesn't work, please provide you environnement config. 

Environment Config
```shell
transformers== 4.34.0
accelerate== 0.23.0
torch== 2.0.1+cu117
```
Beside, I found a warning when I run with `device_map="auto"`:
```shell 
The `language_model` is not in the `hf_device_map` dictionary and you are running your script in a multi-GPU environment. 
this may lead to unexpected behavior when using `accelerate`. Please pass a `device_map` that contains `language_model` to remove this warning.
```
Does `accelerate-large-model` support `blip2-flan-t5-xl` or `blip2-flan-t5-xxl`?

Anorther Question (Although it's a CUDA bug.)
I found DeferredCudaCallError When I use `to("cuda")` with multi-gpu, Do you know why?
```python
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
import torch
model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b").to("cuda")
```
```shell
Error:torch.cuda.DeferredCudaCallError: CUDA call failed lazily at initialization with error: device >= 0 && device < num_gpus
```


@SunMarc How to lock the usage of `device_map="auto"` to a specific GPU?
I have used `os.environ["CUDA_VISIBLE_DEVICES"] = "0,1,2"`, but it does not work;
The command `torchrun test.py --nproc_per_node=3` does not work, either.

I think it is a problem with torch and cuda. In the past, we had a similar [case](https://github.com/huggingface/accelerate/issues/1927). Can you reinstall and try again ? 

Also, the following code snippet works on my side: 
```
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
import torch
model = Blip2ForConditionalGeneration.from_pretrained("Salesforce/blip2-opt-2.7b").to("cuda")
```
As for the warning, this is something we need to fix. It shouldn't show the warning. 

@SunMarc yes, I can use `Salesforce/blip2-opt-2.7b` with `to("cuda")`, but I can not use `Salesforce--blip2-flan-t5-xxl` in 1 gpu with 16GB.
There is always a RuntimeError when I use `device_map="auto"` for Blip2 multi-gpu test, but I can use it for T5 model
```python
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto")
```
So I wonder is it a problem with `Salesforce--blip2-flan-t5-xxl` or other Blip2 model? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.