Perplexity Issue Re-loading models with bnb quantisation
Author: RonanKMcGovern
Pull request: no

### System Info

transformers latest

### Who can help?

@SunMarc and @younesbelkada

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Load a base model with quantization, for example:
```
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=bnb_config,
    device_map={"":0},
    # torch_dtype=torch.float16, # comment back in if loading without using quantization.
    cache_dir=cache_dir)
```
2. Train that model using QLoRA to get an adapter. Measure perplexity of the fine-tuned model, let's call this "fine-tuned perplexity".
3. Dequantize the base model to get a dequantized base model.
4. Apply the adapter to the dequantized base model. Then merge and unload.

Now, running inference:
A. Load the merged model without quantization.
- This model will perform with the "fine-tuned perplexity".
B. Load the merged model with bnb quantization (same config as used for training).
- This model has significantly increased perplexity for fine-tuning tasks. It's as though the fine-tuning adapter was not applied.

### Expected behavior

A and B should display the same perplexity.

Quantizing the model should not degrade performance because the merged model is a quantized model...

Comments:
Hi @RonanKMcGovern, thanks for reporting. Can you give us a minimal reproducer and more information on the perplexity degradation ? Also, what did you use for dequantization ? Did you merge the adapters on top of the base model or did you use `merge_and_unload` from this [PR](https://github.com/huggingface/peft/pull/851/files). 

I face the same problem. I use the gist by https://gist.github.com/ChrisHayduk/1a53463331f52dca205e55982baf9930 
After I merge and save the model, and load in 4 bit (I even tried using quantisation by llama.cpp) - the finetuning is seemingly gone. 

For the PR - I tried - but it seems like merged_model.save_pretrained don't work on 4 bit quantised model that just got merged with the lora adapter (or did I misunderstand how it's supposed to work)? Will Peft eventually include a way to seamlessly merge the 4 bit model with adapter, and then allow us to save the model (whether quantised or otherwise) without losing perplexity?

Quick update - I'll write up more details here on Monday but this may be related to the config.json not being set corrected for my merged file.

BTW, I'm following this approach for dequantizing and merging: https://gist.github.com/ChrisHayduk/1a53463331f52dca205e55982baf9930

Updates @SunMarc :

I'm dequantizing and merging with with: https://gist.github.com/ChrisHayduk/1a53463331f52dca205e55982baf9930

With this script, the config.json has a quantization config specified, so when I load the merged model it automatically loads with quantization. However, the perplexity is increased.

I'm not running perplexity tests but rather evaluating performance on ten questions. Here is the question and answer obtained after fine-tuning:
```

 
 In touch rugby, how many metres must the defending team retreat after a touch? 


The defending team must retreat 7 metres after a touch.

Correct Answer: 7 metres.
```
and here is the answer when I load the merged model with quantization:
```
In touch rugby, the defending team must retreat a distance of 10 meters after a touch. everybody must retreat the same distance, including the player who made the touch.
```

@SunMarc is this information sufficient to assess the issue? Thanks

@RonanKMcGovern Can you provide details on the model and adapter used here so I can help reproduce the issue?

Hi @RonanKMcGovern @ChrisHayduk 
Thanks for the issue, the merge and unload in 4bit/8bit is natively supported on PEFT main. Do you face the same issue on the main build of PEFT ?


Sure @ChrisHayduk ,

Base model: "meta-llama/Llama-2-7b-chat-hf"
QLoRA Adapter: "Trelis/Llama-2-13b-chat-hf-touch-rugby-rules-adapters"
Merged model (dequantized + merged): "Trelis/Llama-2-13b-chat-hf-touch-rugby-rules"

Also, bnb config:
```
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
```


To reproduce:
1. Load and run the merged model.
2. Load the base model, quantize with that bnb config and then apply the adapter.

Lastly, here are some manual questions you may wish to run for comparison:
```
    questions = [
        "In the context of Touch Rugby International Playing Rules 2020, what is the purpose of the Dead Ball Line?", #copied from the test data set to ensure training is working
        "How many players are on the field on each team in touch rugby?",
        "In touch rugby, does a forward pass result in a roll ball, a scrum, or something else?",
        "In touch rugby, how many metres must the defending team retreat after a touch?",
        "In touch rugby, how many substitutions are allowed during a game?",
        "In touch rugby, how long is half time?",
        "In touch rugby, how does the game commence?",
        "In touch rugby, how many metres must defenders retreat when there is a penalty? Is the same as after a touch is made?",
        "In touch rugby, how many touches is a team entitled to prior to a change in possession?",
        "In touch rugby, what happens if a player is touched by an opponent prior to making a pass?",
        "In touch rugby, how many points is a try worth?"
    ]

    answers = [
        "The Dead Ball Line marks the end boundaries of the field of play and indicates when the ball is out of play.",
        "6 players.",
        "Penalty.",
        "7 metres.",
        "There is no limit.",
        "5 minutes.",
        "The game begins with a tap on the halfway line.",
        "10 metres.",
        "Possession changes on the sixth (6th) touch.",
        "The defending team is awarded a penalty.",
        "1 point."
    ]
```

Note that this isn't a particularly good fine-tuning, i.e. there is only a small improvement in questions correct. Still, the merged model get's all the questions wrong, while with the adapter applied, a few are correct.


@younesbelkada :

1. Can you clarify what you mean by "Do you face the same issue on the main build of PEFT ?"?

2. Yes, I use the merge and unload feature. Specifically I quantize a base model, then dequantize it and then merge in the adapter to that dequantized model. It's messy but that's technically the most accurate way to do it, I believe. Unfortunately this then gives a specific issue whereby the perplexity of that merged model is increased.

This is all quite messy. Really the core issue and best solution here is to allow for serialisation of quantized models, because then they could be pushed to hub and reloaded.

@RonanKMcGovern Seems like there's currently a PR for saving and serialisation of 4bit/8bit models after merging. https://github.com/huggingface/transformers/pull/26037  
I tried the 'official' merge implemented in https://github.com/huggingface/peft/pull/851 but was unable to save (not frankly sure why you would even want to merge if we don't want to save the model?).

Thanks @jaredquekjz , agreed and I'm excited about that PR for saving.

Thanks for that link on the merge, I wasn't aware of that. Agreed saving seems important. I dunno, maybe it's possible to convert the merged model to a dict and just save that as a pytorch file along with the dict format. I suppose the quesition then is how you reload...

I'm going to close this because my question is on merging to a dequantized model and https://github.com/huggingface/transformers/pull/26037 addresses that.

Having this exact issue, when I load in 4bit it's like the fine tuning was never applied. @RonanKMcGovern did you figure out a fix?

@sampbarrow I didn't. I reload the base model in bf16 and apply the adapter. It's not ideal but it's the best for now.

> @sampbarrow I didn't. I reload the base model in bf16 and apply the adapter. It's not ideal but it's the best for now.

i wound up quantizing the full model with exllama just to try it out, i dont have the vram to run it at 16 bit. it works and it was very easy to do but i havent compared the quality.