RuntimeError: CUDA error: device-side assert triggered
Author: KaifAhmad1
Pull request: no

### System Info

``` YAML
OS: Windows 11 
Driver Version: 532.09
CUDA Version: 12.1
bitsandbytes: 0.42.0
transformers: 4.37.1
trl: 0.7.10
torch: 2.1.0+cu121
peft: 0.7.1
optimum: 1.16.2
einops: 0.7.0
```
I am using Google Colab T4 GPU for fine tuniing  `mistralai/Mistral-7B-v0.1`  in custom dataset 

### Who can help?

Hey,  @ArthurZucker @muellerzr @SunMarc  Please help me for this issue. 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

``` Python 
# Configuration for quantization
compute_dtype = getattr(torch, "bfloat16")
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=compute_dtype,
)
```


``` Python 
model = AutoModelForCausalLM.from_pretrained(model_name,
                                              quantization_config=bnb_config,
                                              device_map="auto",
                                              use_cache=False,
                                              trust_remote_code=True,
                                              low_cpu_mem_usage=True
                                              )
```

```  Python
trainer.train() 
``` 


Here is the error detail
``` 
RuntimeError                              Traceback (most recent call last)
[<ipython-input-88-6b7a202bc4f8>](https://localhost:8080/#) in <cell line: 1>()
----> 1 mode
RuntimeError                              Traceback (most recent call last)
[<ipython-input-88-6b7a202bc4f8>](https://localhost:8080/#) in <cell line: 1>()
----> 1 model = AutoModelForCausalLM.from_pretrained(model_name,
      2                                               quantization_config=bnb_config,
      3                                               device_map="auto",
      4                                               use_cache=False,
      5                                               trust_remote_code=True,

3 frames
[/usr/local/lib/python3.10/dist-packages/accelerate/utils/modeling.py](https://localhost:8080/#) in get_max_memory(max_memory)
    718             else:
    719                 for i in range(torch.cuda.device_count()):
--> 720                     _ = torch.tensor([0], device=i)
    721                 max_memory = {i: torch.cuda.mem_get_info(i)[0] for i in range(torch.cuda.device_count())}
    722         # allocate everything in the mps device as the RAM is shared

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` tol = AutoModelForCausalLM.from_pretrained(model_name,
      2                                               quantization_config=bnb_config,
      3                                               device_map="auto",
      4                                               use_cache=False,
      5                                               trust_remote_code=True,

3 frames
[/usr/local/lib/python3.10/dist-packages/accelerate/utils/modeling.py](https://localhost:8080/#) in get_max_memory(max_memory)
    718             else:
    719                 for i in range(torch.cuda.device_count()):
--> 720                     _ = torch.tensor([0], device=i)
    721                 max_memory = {i: torch.cuda.mem_get_info(i)[0] for i in range(torch.cuda.device_count())}
    722         # allocate everything in the mps device as the RAM is shared

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
```

### Expected behavior

This cell will be run without raising any error 



Use Colab for your refence for better understanding: https://colab.research.google.com/drive/1rPYEVeXVlRrR3_oM1odaIuL60R9-jm6j?usp=sharing

Comments:
I would recommend you to set `CUDA_LAUNCH_BLOCKING=1` and add a breakpoint to see where this is happening. Usually at the embedding layer, when the tokenizer has an extra token and the vocab was not resized 🤗 

BTW feel free to share the solution if you have found it! 🤗 

Hey, @ArthurZucker

Here is my solution 

``` Python 
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
    low_cpu_mem_usage=True
)

# Tokenization and Padding
tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    trust_remote_code=True,
)
tokenizer.pad_token = tokenizer.eos_token


# Train the model
trainer.train()
```

In case someone else has the same problem, I was finetuning Whisper and `model.resize_token_embeddings(len(tokenizer))` helped resolve the issue.