I'm loading llama2 on a single machine with dual GPUs using Accelerate, and it's giving me a CUDA error. It seems that tensors are unexpectedly being allocated to two different GPUs during runtime.
Author: sev777
Pull request: no

### System Info

I'm loading llama2 on a single machine with dual GPUs using Accelerate, and it's giving me a CUDA error. It seems that tensors are unexpectedly being allocated to two different GPUs during runtime.


My Computer use:

Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              64
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               49
Model name:          AMD EPYC 7282 16-Core Processor
Virtualization:      AMD-V

GPU: A100 40G


```python
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM

import os
os.environ['CUDA_VISIBLE_DEVICES']="0,1"


model_path = '/root/sev777/LMs/huggingface/llama2_13B'

tokenizer = LlamaTokenizer.from_pretrained(model_path)
model = LlamaForCausalLM.from_pretrained(
    model_path,device_map='auto'
)



prompt = 'Q: What is the largest animal?\nA:'
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda(0)
generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=32
)

print(generation_output)


```

My log is: 

`
Loading checkpoint shards:   0%|          | 0/3 [00:00<?, ?it/s]
Loading checkpoint shards:  33%|███▎      | 1/3 [00:10<00:20, 10.00s/it]
Loading checkpoint shards:  67%|██████▋   | 2/3 [00:19<00:09,  9.64s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:25<00:00,  7.87s/it]
Loading checkpoint shards: 100%|██████████| 3/3 [00:25<00:00,  8.39s/it]
Traceback (most recent call last):
  File "mt.py", line 20, in <module>
    generation_output = model.generate(
  File "/root/sev777/miniconda3/envs/llama_adapter/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/root/sev777/miniconda3/envs/llama_adapter/lib/python3.8/site-packages/transformers/generation/utils.py", line 1538, in generate
../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [0,0,0], thread: [0,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [0,0,0], thread: [1,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [0,0,0], thread: [2,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
...

../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [0,0,0], thread: [32,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [0,0,0    ], thread: [33return self.greedy_search(,0
,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu  File "/root/sev777/miniconda3/envs/llama_adapter/lib/python3.8/site-packages/transformers/generation/utils.py", line 2362, in greedy_search
:91: operator(): block: [0,0,0], thread: [34,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [0,0,0], thread: [35,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

...

../aten/src/ATen/native/cuda/IndexKernel.cu:91: operator(): block: [1,0,0], thread: [127,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
    return F.linear(input, self.weight, self.bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`


`

"Is this because my GPU and CPU are conflicting, or is it due to other reasons?"

Thanks!




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I'm loading llama2 on a single machine with dual GPUs using Accelerate, and it's giving me a CUDA error. It seems that tensors are unexpectedly being allocated to two different GPUs during runtime.


### Expected behavior

I'm loading llama2 on a single machine with dual GPUs using Accelerate, and it's giving me a CUDA error. It seems that tensors are unexpectedly being allocated to two different GPUs during runtime.


Comments:
If you are using `device_map='auto'`, then you are already using `Accelerate`. So you can remove the 3 lines in your code that use `Accelerator`.

Refer to: https://huggingface.co/docs/accelerate/usage_guides/big_modeling#using-transformers-diffusers-and-other-open-source-libraries

> If you are using `device_map='auto'`, then you are already using `Accelerate`. So you can remove the 3 lines in your code that use `Accelerator`.
> 
> Refer to: https://huggingface.co/docs/accelerate/usage_guides/big_modeling#using-transformers-diffusers-and-other-open-source-libraries

Thanks, after remove the code,  it will still report the same error.

cc @ArthurZucker 

Hey! The error you are getting seems to be `Index out of bound` which usually means you are feeding inputs that are greater than the size of the mbedding matrix. For this, I suspect that the `tokenizer` has extra tokens (for padding) but the embedding matrix was not properly resized. Make sure to check the inputs of the model, and if you try to run on cpu you'll get the full traceback with the positional embedding

> 
cc @ArthurZucker 

Thanks, I test on CPU, and it can run correct.
like:
```python
#Output:
Loading checkpoint shards: 100%|██████████████████| 3/3 [00:17<00:00,  5.69s/it]
['
 Q: What is the largest animal?\nA: The blue whale is the largest animal in the world. It can grow to be 100 feet long and weigh 150 tons.']
```

**But when I test on GPU, I get the same wrong.**

And follow your suspect, I find the 'position_ids' in [modeling_llama.py](https://github.com/huggingface/transformers/blob/fa6107c97edf7cf725305a34735a57875b67d85e/src/transformers/models/llama/modeling_llama.py#L711C1-L711C47) is on device: 0, and it's value is :
```python
position_ids = tensor([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11]], device='cuda:0')
```

But, when it is transferred as an argument to the [LlamaDecoderLayer.forward](https://github.com/huggingface/transformers/blob/fa6107c97edf7cf725305a34735a57875b67d85e/src/transformers/models/llama/modeling_llama.py#L427), it is moved to device 1. and the value is:

```python
position_ids = 
tensor([[-9223372034707292160, -9223372034707292160, -9223372034707292160,
         -9223372034707292160, -9223372034707292160, -9223372034707292160,
         -9223372034707292160, -9223372034707292160, -9223372034707292160,
         -9223372034707292160, -9223372034707292160, -9223372034707292160]],
       device='cuda:1')

or sometimes:
position_ids = 
tensor([[4872363901336821031, 4872363901336821031, 4872363901336821031,
         4872363901336821031, 4872363901336821031, 4872363901336821031,
                           0,                   0,                   0,
                           0,                   0,                   0]]
```

So, the [apply_rotary_pos_emb](https://github.com/huggingface/transformers/blob/fa6107c97edf7cf725305a34735a57875b67d85e/src/transformers/models/llama/modeling_llama.py#L184) will wrong.

**Furthermore, to my surprise, when I manually executed the code line by line in debug mode, it didn't seem to throw any errors.** 


Here I giev an example how I run the code without wrong ~~.

The main code is:

```python
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM, GenerationConfig

import os
os.environ['CUDA_VISIBLE_DEVICES']="0,1"


model_path = '/root/sev777/LMs/huggingface/llama2_13B'

tokenizer = LlamaTokenizer.from_pretrained(model_path)
tokenizer.pad_token_id=tokenizer.eos_token_id
tokenizer.pad_token=tokenizer.eos_token
model = LlamaForCausalLM.from_pretrained(
    model_path,device_map='auto'
)
generation_config = GenerationConfig(
    pad_token_id=tokenizer.eos_token_id,
    pad_token=tokenizer.eos_token,
)

prompt = 'I love :'
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.cuda(0)
generation_output = model.generate(
    input_ids=input_ids, max_new_tokens=3,generation_config=generation_config
)

print(tokenizer.batch_decode(generation_output))

```

And I added ‘break’ after generating a word in GenerationMixin.greedy_search().
(It's not supposed to, but I added break so I could manually end the process as quickly as possible.)
```python
  break  #
  if this_peer_finished and not synced_gpus:
      break
```

Then I add a breakpoint at [decoder_layer](https://github.com/huggingface/transformers/blob/fa6107c97edf7cf725305a34735a57875b67d85e/src/transformers/models/llama/modeling_llama.py#L711C1-L711C47), and executed the code manually step by step.

Then I get the output:
```python
['
 I love :A']
```
So, is it because my CPU and GPU are not synchronized, causing some variables to have incorrect data when switching between GPUs?

This phenomenon leads to: if I debug the code step by step, then position_ids will not experience the sudden value changes as mentioned above.


cc @SunMarc would be cool if you can have a look! 

Hi @sev777 , i'm unable to reproduce your error unfortunately. I would suggest updating/reinstalling cuda. Let us know if there is any progress on your end. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.