Issue with Fine-tuning LLM for Classification
Author: NickL77
Pull request: no

### System Info

```
- `transformers` version: 4.34.0
- Platform: Linux-6.5.4-76060504-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no
```

### Who can help?

@younesbelkada @muellerzr  @pacman100 @ArthurZucker 


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am trying to fine-tune Mistral 7B on a classification task. I've deduced I need to use `AutoModelForSequenceClassification` and load it via:

```
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
)

mistral_model = AutoModelForSequenceClassification.from_pretrained("mistralai/Mistral-7B-v0.1", quantization_config=bnb_config, load_in_4bit=True, device_map="auto")

for name, param in mistral_model.named_parameters():
    # Freeze all parameters
    if param.dtype in [torch.float16, torch.float32, torch.float64]:
        param.requires_grad = False

        # Unfreeze the last two layers in 'layers' and 'score'
        if name.startswith('model.layers') and (int(name.split('.')[2]) >= 30):
            param.requires_grad = True
        elif name.startswith('score'):
            param.requires_grad = True
```

When quantizing using bnb, I get the following error when running the `trainer`: 

```
File "site-packages/transformers/trainer.py", line 412, in Trainer.__init__
    if _is_quantized_and_base_model and not _is_peft_model:
        raise ValueError(
            "You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of"
            " the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft"
            " for more details"
        )

ValueError: You cannot perform fine-tuning on purely quantized models. Please attach trainable adapters on top of the quantized model to correctly perform fine-tuning. Please see: https://huggingface.co/docs/transformers/peft for more details

```
I understand that it's best to fine-tune using a peft method, but here I'm freezing most layers and keeping a only a few layers trainable. Is there a reason it is not recommended/supported to do this? Or is there an alternative method?

-----------------------------

If I load the model without quantizing via:

```
mistral_model = AutoModelForSequenceClassification.from_pretrained("mistralai/Mistral-7B-v0.1", device_map="auto")
```

I then get the following error:

```
File "site-packages/transformers/trainer.py", line 515, in Trainer.__init__
    self._move_model_to_device(model, args.device)

File "site-packages/transformers/trainer.py", line 739, in Trainer._move_model_to_device
    model = model.to(device)

File "site-packages/accelerate/big_modeling.py", line 415, in <method-name>
    raise RuntimeError("You can't move a model that has some modules offloaded to cpu or disk.")

RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.
```

### Expected behavior

Ideally I can train the classifier with quantization, but I'm ok with using smaller batch sizes to get it working without quantization.

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead?

Regarding quantization, you cannot train a fully quantized model it's just not supported (not supported by any library AFAIK) because it's way to unstable (gradient computation, overflows etc). @younesbelkada will probably explain it better than me when he comes back from holidays! 

I don't know which training script you are using but would recommend you to check [this](https://colab.research.google.com/drive/1jCkpikz0J2o20FBQmYmAGdiKmJGOMo-o?usp=sharing) from the `peft` library. 

Thanks!

Hi @NickL77 
Thanks a lot for the issue, as @ArthurZucker mentioned, you cannot perform pure fine-tuning on quantized models. To fully leverage quantization, you need to train adapters on top of the quantized models, using for example PEFT library. 
Check out for example this documentation section: https://huggingface.co/docs/transformers/peft on how to train adapters using PEFT and transformers

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.