AttributeError: 'BitsAndBytesConfig' object has no attribute 'get_loading_attributes'
Author: ZER01NE44
Pull request: no

### System Info

Platform: Windows 11
Python Version: 3.10.11
PyTorch Version: 2.0.1+cu118
accelerate: 0.23.0
bitsandbytes: 0.39.1
huggingface-hub: 0.17.3
tokenizers: 0.14.1
transformers: 4.35.0.dev0

### Who can help?

@SUNMARC 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. autotrain llm --train --project_name test-llama2-finetune --model TheBloke/Llama-2-13B-chat-GPTQ --data_path C:\Users\JeongHyoengyo\Desktop\data\ --text_column text --use_peft --use_int4 --learning_rate 2e-4 --train_batch_size 2 --num_train_epochs 3 --trainer sft --model_max_length 2048

While Fine-tuning **TheBloke/Llama-2-13B-chat-GPTQ**, I ran into this issue.


```shell
(Fine-tuning) C:\Users\JeongHyoengyo>autotrain llm --train --project_name test-llama2-finetune --model TheBloke/Llama-2-13B-chat-GPTQ --data_path C:\Users\JeongHyoengyo\Desktop\data\ --text_column text --use_peft --use_int4 --learning_rate 2e-4 --train_batch_size 2 --num_train_epochs 3 --trainer sft --model_max_length 2048
bin C:\Users\JeongHyoengyo\anaconda3\envs\Fine-tuning\lib\site-packages\bitsandbytes\libbitsandbytes_cuda118.dll
> INFO    Running LLM
> INFO    Params: Namespace(version=False, train=True, deploy=False, inference=False, data_path='C:\\Users\\JeongHyoengyo\\Desktop\\data\\', train_split='train', valid_split=None, text_column='text', model='TheBloke/Llama-2-13B-chat-GPTQ', learning_rate=0.0002, num_train_epochs=3, train_batch_size=2, warmup_ratio=0.1, gradient_accumulation_steps=1, optimizer='adamw_torch', scheduler='linear', weight_decay=0.0, max_grad_norm=1.0, seed=42, add_eos_token=False, block_size=-1, use_peft=True, lora_r=16, lora_alpha=32, lora_dropout=0.05, logging_steps=-1, project_name='test-llama2-finetune', evaluation_strategy='epoch', save_total_limit=1, save_strategy='epoch', auto_find_batch_size=False, fp16=False, push_to_hub=False, use_int8=False, model_max_length=2048, repo_id=None, use_int4=True, trainer='sft', target_modules=None, merge_adapter=False, token=None, backend='default', username=None, use_flash_attention_2=False, func=<function run_llm_command_factory at 0x0000025FC810FC70>)
Using pad_token, but it is not set yet.
> ERROR   train has failed due to an exception:
> ERROR   Traceback (most recent call last):
  File "C:\Users\JeongHyoengyo\anaconda3\envs\Fine-tuning\lib\site-packages\autotrain\utils.py", line 280, in wrapper
    return func(*args, **kwargs)
  File "C:\Users\JeongHyoengyo\anaconda3\envs\Fine-tuning\lib\site-packages\autotrain\trainers\clm\__main__.py", line 124, in train
    model = AutoModelForCausalLM.from_pretrained(
  File "C:\Users\JeongHyoengyo\anaconda3\envs\Fine-tuning\lib\site-packages\transformers\models\auto\auto_factory.py", line 565, in from_pretrained
    return model_class.from_pretrained(
  File "C:\Users\JeongHyoengyo\anaconda3\envs\Fine-tuning\lib\site-packages\transformers\modeling_utils.py", line 2690, in from_pretrained
    loading_attr_dict = quantization_config.get_loading_attributes()
AttributeError: 'BitsAndBytesConfig' object has no attribute 'get_loading_attributes'
``` 
I've already changed the Transformers model from 4.30.0 to 4.35.0, but the same problem appears.
Also, it was the same when i downloaded bitsandbytes from https://github.com/TimDettmers/bitsandbytes.
Current bitsandbytes version is 0.39.1 because 0.41.1 fail below code.
```python
python -m bitsandbytes
``` 
I don't know how to fix it. Please help me.

### Expected behavior
Fine-tuning sucessfully.

Comments:
+1

cc @younesbelkada 

Hi @ZER01NE44 
Thanks for the issue, you seem to be using auto-train - and I see many issues here
1- you are passing the flag `--use_int4` on a GPTQ model. `--use-int4` will try to convert the model in 4bit using bitsandbytes. You cannot convert a GPTQ model with bitsandbytes hence the error
2- It seems that you are using a windows machine, I think bitsandbytes and windows are not compatible
To solve your issue you can try out two things
1- remove `--use_int4` flag
2- use another model than `TheBloke/Llama-2-13B-chat-GPTQ` e.g. `meta-llama/Llama-2-13b-chat-hf`

Thank you so much, @younesbelkada 

I removed --use_int4 and proceeded as you said, but i got an below error.

```shell
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
``` 

If bitsandbytes are not compatible with Windows, can't I use GPTQ even if I fix this error?
Also, do you know how to fix this error?

Hi @ZER01NE44 
Thank you, can you share the full traceback of the error? Also the error might be specific to auto-train, can you also open a ticket there and tag me together with @abhishekkrthakur ? 🙏 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I am also facing the same issue.
log:
(training_llama) ubuntu@ip-172-31-10-111:~/llma2/training/qlora$ sh scripts/finetune.sh 

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please run

python -m bitsandbytes

 and submit this information together with your error trace to: https://github.com/TimDettmers/bitsandbytes/issues
================================================================================
bin /home/ubuntu/miniconda3/envs/training_llama/lib/python3.11/site-packages/bitsandbytes-0.39.0-py3.11.egg/bitsandbytes/libbitsandbytes_cuda117.so
/home/ubuntu/miniconda3/envs/training_llama/lib/python3.11/site-packages/bitsandbytes-0.39.0-py3.11.egg/bitsandbytes/cuda_setup/main.py:149: UserWarning: /home/ubuntu/miniconda3/envs/training_llama did not contain ['libcudart.so', 'libcudart.so.11.0', 'libcudart.so.12.0'] as expected! Searching further paths...
  warn(msg)
CUDA SETUP: CUDA runtime path found: /usr/local/cuda-11.7/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 8.6
CUDA SETUP: Detected CUDA version 117
CUDA SETUP: Loading binary /home/ubuntu/miniconda3/envs/training_llama/lib/python3.11/site-packages/bitsandbytes-0.39.0-py3.11.egg/bitsandbytes/libbitsandbytes_cuda117.so...
loading base model /home/ubuntu/llma2/text-generation-webui/models/TheBloke_Llama-2-70B-Chat-GPTQ/...
Traceback (most recent call last):
  File "/home/ubuntu/llma2/training/qlora/qlora.py", line 769, in <module>
    train()
  File "/home/ubuntu/llma2/training/qlora/qlora.py", line 600, in train
    model = get_accelerate_model(args, checkpoint_dir)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/llma2/training/qlora/qlora.py", line 264, in get_accelerate_model
    model = AutoModelForCausalLM.from_pretrained(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/miniconda3/envs/training_llama/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/miniconda3/envs/training_llama/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2786, in from_pretrained
    loading_attr_dict = quantization_config.get_loading_attributes()
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'BitsAndBytesConfig' object has no attribute 'get_loading_attributes'
(training_llama) ubuntu@ip-172-31-10-111:~/llma2/training/qlora$ 


Hey @qburst-fidha, it is not possible to quantize an already quantized model (GPTQ model).  I suggest you to use this model instead  [Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf). 

> Hey @qburst-fidha, it is not possible to quantize an already quantized model (GPTQ model). I suggest you to use this model instead [Llama-2-70b-chat-hf](https://huggingface.co/meta-llama/Llama-2-70b-chat-hf).

Is there any other way to finetune the GPTQ model

Here is a [gist](https://gist.github.com/SunMarc/dcdb499ac16d355a8f265aa497645996) on how to finetune the GPTQ model !

This error still exists.
When I use the awq model at llamafactory to select hqq quantization to 4bit.I get this error.And I uninstall bitsandbytes then get this.
```output
[INFO|trainer.py:741] 2025-01-23 16:42:49,333 >> Using auto half precision backend
Traceback (most recent call last):
  File "/root/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 1365, in get_optimizer_cls_and_kwargs
    from bitsandbytes.optim import AdamW, Lion, RMSprop
ModuleNotFoundError: No module named 'bitsandbytes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/anaconda3/bin/llamafactory-cli", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/mnt/d/my/work/LLM/ai_train/LLaMA-Factory/src/llamafactory/cli.py", line 112, in main
    run_exp()
  File "/mnt/d/my/work/LLM/ai_train/LLaMA-Factory/src/llamafactory/train/tuner.py", line 92, in run_exp
    _training_function(config={"args": args, "callbacks": callbacks})
  File "/mnt/d/my/work/LLM/ai_train/LLaMA-Factory/src/llamafactory/train/tuner.py", line 72, in _training_function
    run_dpo(model_args, data_args, training_args, finetuning_args, callbacks)
  File "/mnt/d/my/work/LLM/ai_train/LLaMA-Factory/src/llamafactory/train/dpo/workflow.py", line 83, in run_dpo
    train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 2171, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 2272, in _inner_training_loop
    self.create_optimizer_and_scheduler(num_training_steps=max_steps)
  File "/root/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 1166, in create_optimizer_and_scheduler
    self.create_optimizer()
  File "/mnt/d/my/work/LLM/ai_train/LLaMA-Factory/src/llamafactory/train/dpo/trainer.py", line 114, in create_optimizer
    return super().create_optimizer()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 1214, in create_optimizer
    optimizer_cls, optimizer_kwargs = self.get_optimizer_cls_and_kwargs(self.args, opt_model)
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/anaconda3/lib/python3.12/site-packages/transformers/trainer.py", line 1420, in get_optimizer_cls_and_kwargs
    raise ValueError("Trainer tried to instantiate bnb optimizer but `bitsandbytes` is not installed!")
ValueError: Trainer tried to instantiate bnb optimizer but `bitsandbytes` is not installed!
```

what kind of optimizer are you using ? The error is triggered because you are using an optimizer that comes from bitsandbytes