Inference speed becomes slower after quantization
Author: xinyual
Pull request: no

### System Info

- `transformers` version: 4.37.0.dev0
- Platform: Linux-5.15.0-1038-aws-x86_64-with-glibc2.10
- Python version: 3.8.17
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0.dev0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I use the script from here to finetune my quantized mistral-7b-instruct model with lora: https://colab.research.google.com/drive/1VoYNfYDKcKRQRor98Zbf2-9VQTtGJ24k?usp=sharing#scrollTo=Ybeyl20n3dYH
After training, I inference as:
```
bnb_config = BitsAndBytesConfig(
            load_in_4bit=True,
            bnb_4bit_use_double_quant=True,
            bnb_4bit_quant_type="nf4",
            bnb_4bit_compute_dtype=torch.bfloat16
        )
model = AutoModelForCausalLM.from_pretrained(base_model,
                                    quantization_config=bnb_config,
                                    device_map="auto",
                                     torch_dtype=torch.bfloat16,
                                     trust_remote_code=True)
model = PeftModel.from_pretrained(
    model,
    lora_weights,
    device_map ="auto"
)
model.half()
model.eval()
```
The bnb_config is same as training process. 
After that, I inference like:
```
with torch.no_grad():
      generation_output = model.generate(
          input_ids=input_ids,
          do_sample=True,
          max_length=input_l + max_new_tokens + 100,
          temperature=0.001
          )
```
Compared with the model without quantization, the speed is slow. From my knowledge, we use low-bit computation so it would be fast. My hardware is AWS g5.12xlarge. Is that normal?


### Expected behavior

Please tell me whether it is normal to be slower or I made some mistakes in my scripts. Thanks!

Comments:
cc @younesbelkada @SunMarc 

hi @xinyual 
Thanks a lot for the issue, in fact, in your script you are using `bnb_4bit_use_double_quant` which slows down inference at the price of being more memory efficient since the linear layers will be quantized twice. 
if you disable double quant it should be faster, but not sure it will be faster than fp16, this will depend on your problem setup (batch size, seq lenght, etc.) If you want a fast 4-bit model for deployment I would advise to 

- Quantize your fine-tuned model with AWQ algorithm using auto-awq: https://github.com/casper-hansen/AutoAWQ
- Use AWQ fused modules for generation which is ~3x faster than native fp16: https://huggingface.co/docs/transformers/quantization#benchmarks 

You can also consider quantizing your model with GPTQ algo but GPTQ do not support fused modules yet

Read more about the benefits of each quantization scheme and when to use them in this blogpost: https://huggingface.co/blog/overview-quantization-transformers 

Let me know if you face into any issue. cc @casper-hansen just FYI 

It would probably be good to disable double quantization when using AutoAWQ, maybe raise an error? If you quantize with AWQ and then BNB, you will just have a worse model in the end.

> It would probably be good to disable double quantization when using AutoAWQ, maybe raise an error? If you quantize with AWQ and then BNB, you will just have a worse model in the end.

Yes this is already handled in transformers ! What I meant is the following workflow:

- Fine-tune your model using QLoRA
- Merge the LoRA weights into the base model and push / save the final checkpoint in float16
- Quantize the merged model using AWQ

Sorry for late response! Could you please give me instruction about `Merge the LoRA weights into the base model and push / save the final checkpoint in float16`?
After
```
model = PeftModel.from_pretrained(
    model,
    lora_weights,
    device_map ="auto"
)
```
The model.save_pretrained will only save lora.

Hi @xinyual 
Sure yes, please refer to this comment: https://huggingface.co/Salesforce/codegen2-7B/discussions/1#6543f4eb2996405c23882b03 to understand what merging means and how to perform it using PEFT. Once you have called `model.merge_and_unload()` you can call save_pretrained directly

Thanks for your effort! I use AWQ and then it speeds up a lot. Very appreciate about that. I will close this issue
.

Thanks @xinyual , this is great !