LLama RuntimeError: CUDA error: device-side assert triggered
Author: abdoelsayed2016
Pull request: no

### System Info



- `transformers` version: 4.28.0.dev0
- Platform: Linux-4.18.0-372.16.1.el8_6.0.1.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- Huggingface_hub version: 0.13.3
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction




`
  0%|                                                                                  | 0/1524 [00:00<?, ?it/s]Traceback (most recent call last):
  File "alpaca-lora/finetune.py", line 234, in <module>
    fire.Fire(train)
  File ".local/lib/python3.9/site-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/home/c703/c7031420/.local/lib/python3.9/site-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/home/c703/c7031420/.local/lib/python3.9/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "alpaca-lora/finetune.py", line 203, in train
    trainer.train()
  File ".conda/envs/llama/lib/python3.9/site-packages/transformers/trainer.py", line 1639, in train
    return inner_training_loop(
  File ".conda/envs/llama/lib/python3.9/site-packages/transformers/trainer.py", line 1906, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File ".conda/envs/llama/lib/python3.9/site-packages/transformers/trainer.py", line 2652, in training_step
    loss = self.compute_loss(model, inputs)
  File ".conda/envs/llama/lib/python3.9/site-packages/transformers/trainer.py", line 2684, in compute_loss
    outputs = model(**inputs)
  File ".conda/envs/llama/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File ".conda/envs/llama/lib/python3.9/site-packages/peft/peft_model.py", line 575, in forward
    return self.base_model(
  File ".conda/envs/llama/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File ".conda/envs/llama/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File ".conda/envs/llama/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 765, in forward
    outputs = self.model(
  File ".conda/envs/llama/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File ".conda/envs/llama/lib/python3.9/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File ".conda/envs/llama/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 574, in forward
    attention_mask = self._prepare_decoder_attention_mask(
  File ".conda/envs/llama/lib/python3.9/site-packages/transformers/models/llama/modeling_llama.py", line 476, in _prepare_decoder_attention_mask
    combined_attention_mask = _make_causal_mask(
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.`

### Description 

I am interested in working with the Arabic language. I have tried adding all the tokens to the LLama tokenizer, and the tokenizer seems to work fine. However, during training, I encountered an error. I am looking for a solution to resolve this error.

Comments:
@abdoelsayed2016 Thanks for raising this issue. Could you share a minimal code snippet to enable us to reproduce the error? 

Just from the traceback alone, it seems that the issue is CUDA related, rather than the transformers model. The Llama model has been under active development and was part of the official version release yesterday. I'd also suggest updating to the most version of the code to make sure you have any possible updates which might have been added. 

@abdoelsayed2016 hi, any suggestion? same error here when I want to train llama with lora.

@j-suyako did you add token to the tokenizer?

yes, I follow the [alpaca](https://github.com/tatsu-lab/stanford_alpaca/blob/main/train.py) to create the dataset, but I forget to resize the tokenizer length. Thanks for your reply!