RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper_CUDA__native_layer_norm)
Author: nirdoshrawal009
Pull request: no

### System Info

I want to finetune Falcon 7b models using SFTTrainer from Transformers library. 
I have set the device_map = 'auto' while loading the model and cuda_visible_devices = '0,1'
But getting this error.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper_CUDA__native_layer_norm)

### Expected behavior

I want to finetune Falcon 7b models using SFTTrainer from Transformers library. 
I have set the device_map = 'auto' while loading the model and cuda_visible_devices = '0,1,2,3'
But getting this error.

Comments:
Hi @nirdoshrawal009, can you provide a minimal reproducer ? Thanks 

Meet the same problem, here is a minimal reproducer of mine:

```python
from transformers import pipeline

# In my case, audio is passed through HTTP requests
def process(audio):
	pipe = pipeline("automatic-speech-recognition", model='openai/whisper-large-v2', device_map='auto')
	return pipe(audio)
```

dependency version:

- transformers: 4.32.1
- accelerate: 0.24.0

Have you solved this problem? 

> ### System Info
> I want to finetune Falcon 7b models using SFTTrainer from Transformers library. I have set the device_map = 'auto' while loading the model and cuda_visible_devices = '0,1' But getting this error.
> 
> ### Who can help?
> _No response_
> 
> ### Information
> * [ ]  The official example scripts
> * [ ]  My own modified scripts
> 
> ### Tasks
> * [ ]  An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
> * [ ]  My own task or dataset (give details below)
> 
> ### Reproduction
> RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! (when checking argument for argument weight in method wrapper_CUDA__native_layer_norm)
> 
> ### Expected behavior
> I want to finetune Falcon 7b models using SFTTrainer from Transformers library. I have set the device_map = 'auto' while loading the model and cuda_visible_devices = '0,1,2,3' But getting this error.



Hi @jrt-20, i can't fix the issue since i am unable to reproduce the error. If you are willing to give me a minimal reproducer, I can help you ! 

> Hi @jrt-20, i can't fix the issue since i am unable to reproduce the error. If you are willing to give me a minimal reproducer, I can help you !
Many thanks for your kind and warm help，the code and the log are as follows:
```python
from transformers import GPT2Tokenizer, GPT2Model

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained("gpt2",device_map='auto').to("cuda")
text = "Replace me by any text you'd like."
encoded_input = tokenizer(text, return_tensors='pt')
encoded_input = encoded_input.to("cuda")
output = model(**encoded_input)
print(output)
```

```
Traceback (most recent call last):
  File "test.py", line 8, in <module>
    output = model(**encoded_input)
  File "/root/miniconda3/envs/sgp_jrt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/miniconda3/envs/sgp_jrt/lib/python3.7/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/miniconda3/envs/sgp_jrt/lib/python3.7/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 846, in forward
    inputs_embeds = self.wte(input_ids)
  File "/root/miniconda3/envs/sgp_jrt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "/root/miniconda3/envs/sgp_jrt/lib/python3.7/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/root/miniconda3/envs/sgp_jrt/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 162, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/root/miniconda3/envs/sgp_jrt/lib/python3.7/site-packages/torch/nn/functional.py", line 2210, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:7! (when checking argument for argument index in method wrapper__index_select)
```

Hi @jrt-20, you should move a model when you load it with device_map="auto" because splitting the model across all available gpus. Either do `model = GPT2Model.from_pretrained("gpt2", device_map='auto')` or `model = GPT2Model.from_pretrained("gpt2", device_map='cuda')`. I think that you should have received a warning in the log. 

lgtm,I have solved this problem

Glad that you solved the problem ! Closing the issue.