RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
Author: anujsahani01
Pull request: no

Quantizing  HuggingFaceH4/starchat-alpha model using transformers BitsAndBytesConfig and LoRA Config for reducing the memory usage.

```
nf4_config = BitsAndBytesConfig( load_in_4bit=True,
                                 bnb_4bit_quant_type="nf4",
                                 bnb_4bit_use_double_quant=True,
                                 bnb_4bit_compute_dtype=torch.bfloat16)
                                                      
model = AutoModelForCausalLM.from_pretrained(model_id, 
                                             config = config,
                                             quantization_config=nf4_config, 
                                             torch_dtype=torch.bfloat16,)

model.resize_token_embeddings(num_additional_token + tokenizer.vocab_size)
```
tried to reduce the model_max_length:
```
tokenizer = AutoTokenizer.from_pretrained("HuggingFaceH4/starchat-alpha",
                                          additional_special_tokens =  ["[SYSTEM]", "[ASSISTANT]", "[USER]", "[END]"],
                                          pad_token = "[PAD]",
                                          model_max_length = 1000,
                                          return_token_type_ids=False)
```

Used LoRA configuration: 
```
from peft import LoraConfig, get_peft_model
model.gradient_checkpointing_enable()
model = prepare_model_for_kbit_training(model)
lora_config = LoraConfig(
    r=8, 
    lora_alpha=32, 
    target_modules=["c_attn"], 
    lora_dropout=0.05, 
    bias="none", 
    task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
model.config.use_cache = False
```

Using the LoRA configuration i was able to reduce trainable params to 4,014,080 .
Finally encountered the error while training the model using HuggingFace TrainingArguments and Trainer to train the model.

```
training_args = TrainingArguments(
    output_dir='./NeuralCodeBot_starchat',          # output directory
    num_train_epochs= 2,             # total number of training epochs
    per_device_train_batch_size=1,   # batch size per device during training
    per_device_eval_batch_size=1,   # batch size for evaluation
    warmup_steps=50,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=100,
    learning_rate=1e-3,
    max_steps = 10000,
    fp16= True,
    push_to_hub=True,
)

trainer = Trainer(
    model= model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["test"],
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    )

trainer.train()
```
## Error Message:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-29-3435b262f1ae>](https://localhost:8080/#) in <cell line: 1>()
----> 1 trainer.train()

30 frames
[/usr/local/lib/python3.10/dist-packages/bitsandbytes/autograd/_functions.py](https://localhost:8080/#) in forward(ctx, A, B, out, bias, state)
    514         # 1. Dequantize
    515         # 2. MatmulnN
--> 516         output = torch.nn.functional.linear(A, F.dequantize_4bit(B, state).to(A.dtype).t(), bias)
    517 
    518         # 3. Save state

RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

```

When running the above code, I encountered an error during training. Unfortunately, I'm unable to provide the exact error message as I couldn't run the code on CPU due to the limitation of BitsAndBytesConfig requiring GPUs. However, I would like to request assistance from the community in resolving this issue. 
If anyone has encountered a similar error while quantizing HuggingFace language models, specifically using BitsAndBytesConfig and Trainer, I would greatly appreciate any suggestions or guidance on how to overcome this issue.

Any of your suggestions will be highly appreciated.
Thank You in advance. 





Comments:
I was able to resolve the issue by following changes

Firstly i added the special tokens that were used in my prompt card.
```
special_token_dict = tokenizer.special_tokens_map
tokenizer.add_special_tokens(special_token_dict)
```

Then changed this line of code 
``` 
model.resize_token_embeddings(num_additional_token + tokenizer.vocab_size)
```

to this
```
model.resize_token_embeddings(len(tokenizer))
```

> I was able to resolve the issue by following changes
> 
> Firstly i added the special tokens that were used in my prompt card.
> 
> ```
> special_token_dict = tokenizer.special_tokens_map
> tokenizer.add_special_tokens(special_token_dict)
> ```
> 
> Then changed this line of code
> 
> ```
> model.resize_token_embeddings(num_additional_token + tokenizer.vocab_size)
> ```
> 
> to this
> 
> ```
> model.resize_token_embeddings(len(tokenizer))
> ```

I encountered the same problem, and it took me a few hours, and couldn't solve it. Your solution successfully resolves my problem. Thanks!