Abnormally slow inference speed of quantized model?
Author: JiancongWang
Pull request: no

### System Info



To reproduce, I am running on CUDA 12.1/Driver 530 on an A100 with Ubuntu 20.04. I am running with the following packages
accelerate                    0.21.0.dev0
triton                        2.0.0
transformers                  4.31.0.dev0
torch                         2.0.1
bitsandbytes                  0.40.0.post3

Output from the transformers-cli env is 
- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.15.0-1015-aws-x86_64-with-glibc2.17
- Python version: 3.8.12
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Hi guys,
I am trying out the load_in_4bit/load_in_8bit options to see if it speed up model inference. My understanding is that by quantizing the model, the inference speed will improve. However that's not the case. I use the following simple script to test out speed on a T5 XXL model for 4bits/8bits/fp32, and actually fp32 model runs the fastest (0.07sec), and the 4bits/8bits run almost in the same speed (0.1sec). So I want to double check. 

The script I am using to test the speed is here

```
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch
import pdb
import gc
import time
import os
device_id = 3
tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
input_text = "translate English to German: How old are you?" * 20
input_ids = tokenizer(input_text, return_tensors="pt").input_ids[:, :128].to(f"cuda:{device_id}")

# Actual model loading
max_memory = f'{int(torch.cuda.mem_get_info()[0]/1024**3)-2}GB'
model_4bits = T5ForConditionalGeneration.from_pretrained(
    "google/flan-t5-xxl", 
    load_in_4bit=True,
    max_memory=max_memory)

# model_8bits = T5ForConditionalGeneration.from_pretrained(
#     "google/flan-t5-xxl", 
#     load_in_8bit=True,
#     max_memory=max_memory)

# model_fp32 = T5ForConditionalGeneration.from_pretrained(
#     "google/flan-t5-xxl").to(f"cuda:{device_id}")

def benchmark(model_name, model, input_ids):
    # warmup
    for _ in range(200):
        model.encoder(input_ids)

    torch.cuda.synchronize()
    with torch.no_grad():
        start = time.time()
        for i in range(200):
            model.encoder(input_ids)
            
        torch.cuda.synchronize()
        end = time.time()

        print(f"{model_name} inference time is {(end-start)/200} sec")

benchmark("model_4bits", model_4bits, input_ids) # model_4bits inference time is 0.1036052393913269 sec
# benchmark("model_8bits", model_8bits, input_ids) # model_8bits inference time is 0.1006016504764557 sec
# benchmark("model_fp32", model_fp32, input_ids) # model_fp32 inference time is 0.0731453263759613 sec
```

### Expected behavior

I expect the 4 bits model runs faster than the 8bits model, which in turn runs faster than the fp32 model. That's not the case I observe. 

Comments:
Having the same issue here. Also want to ask if it is true that 8bit model is slower than fp16 during inference

cc @younesbelkada for `4-bit` 🙏 

Hi everyone,
Sadly 8bit models are expected to be slower than fp16 models, see this: https://huggingface.co/blog/hf-bitsandbytes-integration#faster-inference-speed-for-smaller-models for reference
bitsandbytes juste released a new version for faster inference (batch_size=1) https://github.com/TimDettmers/bitsandbytes/releases/tag/0.40.0
Can you try to upgrade bitsandbytes and run the benchmark again?

> Hi everyone, Sadly 8bit models are expected to be slower than fp16 models, see this: https://huggingface.co/blog/hf-bitsandbytes-integration#faster-inference-speed-for-smaller-models for reference bitsandbytes juste released a new version for faster inference (batch_size=1) https://github.com/TimDettmers/bitsandbytes/releases/tag/0.40.0 Can you try to upgrade bitsandbytes and run the benchmark again?

Hi younesbelkada, I get my bitsandbytes from directly pulling the github repository and compile from source yesterday. So it is already the latest 0.40 version that includes the 4 bits bs1 inference. 

This is strange .. Can you report that to bitsandbytes library? 🙏 

> This is strange .. Can you report that to bitsandbytes library? 🙏

Sure. I will post on the issues of bitsandbytes library and link this issue. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I don't think 4/8 bits models should be faster than fp16 models.
They are designed to reduce the memory requirements, not the computation times.

I experienced same issue trying to quantize CLIP using quanto. 

https://github.com/huggingface/optimum-quanto/issues/202

the issue is also closed citing this is normal. I have to say I am oblivious and naive to how quantization is done and what goes on underneath the hood. I was really assuming and hoping inference is faster. Maybe this is true for certain memory constrained env.