GPTQ Quantization reduces number of parameters by a lot (Factor of more than 30) to the point that model is unusable 
Author: manitadayon
Pull request: no
Labels: name: Quantization

name: bug

### System Info

DataBricks with the following packages:
```
transformers: 4.45.2
huggingface-hub: 0.26.1
accelerate: 1.0.1
optimum: 1.23.1
auto-gptq: 0.7.1
bitsandbytes: 0.44.1
```
### Reproduction

I was trying to quantize Llama 3.1 70B instruct using GPTQ using the following code:

```
gptq_config = GPTQConfig(bits = 4, dataset= 'wikitext2', group_size = 128, desc_act = False, batch_size = 1)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map = 'auto', quantization_config =  gptq_config )
```

The process is successful, however the number of parameters is around 2 Billion. For the inference the model is very slow and gives very poor result. 

### Expected behavior

Using bitandbytes or theoretically you would expect the number of parameters to be 35 Billion. May I know if there is any problem with the package?

Comments:
cc @SunMarc 

Hey @manitadayon, we have pre-quantized models [here](https://huggingface.co/hugging-quants/Meta-Llama-3.1-70B-Instruct-GPTQ-INT4). If you look at the files size, it's about 40GB which is what we expect. We shared a lot of ways to deploy this specific model in the model card. How slow is the model compared to the bf16 model ? Thanks ! 

Hi @SunMarc, thanks for the response. I am talking about inference of 3 min is now 10 min or inference of 1.5 min is now 5 min. This is based on max_token length.

My quantized model still takes roughly 40GB of GPU memory but now has 2 billion parameters.

1- Can you please elaborate if that much reduction in number parameters is normal? (since for bitsandbytes the number of parameters is 36 Billion)

2- Can you please let me know what is wrong with snippet I shared if this is not normal? Is there any parameter I should tweak? I try to follow the documentation. This is important since I eventually need it for my own model. 


Ok I have done a numerous testing with GPTQ quantization both with the hugging face one, the AutoGPTQ package and also loaded the pre-quantized model (that mentioned above). They all give roughly 2 Billion parameters. Hence the original quantization was correct.

However they make the original model almost useless (unusable). Can someone please explain, this is the behavior expected from GPTQ quantization? 

Like why do we use this quantization? Please share if you know any techniques to make the model inference faster.

 

@manitadayon I think there are two issues here with your problem.

1. GPTQ 4bit compresses 8 previous float16 (8x2 bytes) tensors into 1 tensor (1x int32 4bytes)  and does runtime dequantization/decompression for inference. So the total number of `parameters` does not change. I think your math for `2B` parameters is wrong since even if 2Bx8 for 4bit does not match the 70B param. You need calculate the parameters size of all modules in each layer and multiply by the number of layers + embedding module which is not part of the repeating layers.  

2. As for bad quality of output post-quantization, I highly recommend you to use [GPTQModel](https://github.com/modelcloud/gptqmodel) to gptq quantize your model and avoid using transformers or even autogptq directly as gptqmodel has eliminated most of the bad defaults that may plague bad qunatization for new users to qunatization.  Lastly, calibration based quantization is not magic. You need to learn and use the control knobs to create the best quant and it takes a lot of trail and error. Start with a small 8B model, if that quant is good with your params, then quant the 70B model since the process is slow. 

EDIT: fixed math

Hey @manitadayon, I tried the same code as you and I have a model with 11B parameters [here](https://huggingface.co/medmekk/quantized-llama-70B) which is the logical size to have, since the tensor types after quantization are I32 (you can check that on the hub, and it's different from bitsandbytes where they use U8), so each parameter holds 8 quantized weights (in 4 bit quantization of course), which will roughly divide the number of parameters by 8 => 70/8 = ~9 and sure 9 != 11 because we did consider that even layer norms and the lm_head are quantized, which is not the case, but it's a good approximation. So having only 2B parameters after quantizing a 70B model seems weird ! are you sure it's the 70B model and not the 8B ?

Hi @MekkCyber and @Qubitium 
Yes, it is Llama 70B parameter. It maybe the fact that my way of calculating the number of parameters is not correct for this case:
This is how I am doing:

`sum(p.numel() for p in model.parameters())`

I have tried this using pre-existing huggingface model for llama 70B GPTQ quantized, my quantized version using GPTQConfig and AutoGPTQ and they all returned ~ 2 Billion parameters. The same code returns 36 Billion parameters for bitandbytes quantization. (It is possible that I am missing something, please let me know if this is the case). The quality of response is still pretty good though.
However the model (in all three cases tested) is 3-4 times slower than quantized using bitsandbytes version.  

> sum(p.numel() for p in model.parameters())

You shouldn't rely on that to calculate the number of parameters since each quantization methods packs the weights differently ! You should instead check directly the size of your files as this will give you a better metrics. 

> As for bad quality of output post-quantization, I highly recommend you to use [GPTQModel](https://github.com/modelcloud/gptqmodel) to gptq quantize your model and avoid using transformers or even autogptq directly as gptqmodel has eliminated most of the bad defaults that may plague bad qunatization for new users to qunatization

Could you shared a bit about the defaults you've seen with transformers/autogptq quants ? If possible, it would be nice to allow users to run the model you quantized on transformers directly also ! 

> You shouldn't rely on that to calculate the number of parameters since each quantization methods packs the weights differently ! You should instead check directly the size of your files as this will give you a better metrics.

Yes I second what @SunMarc said, you can't rely on `model.parameters()` because it doesn't take into account the quantized weights, so the 2B parameters are just the layer norms, lm_head, and embeddings parameters (which are pytorch native), and to verify that you can use the variant `named_parameters()` to verify the name of the parameters, like the following code : 
```
for name, param in model.named_parameters():
    print(f"Parameter Name: {name}")
    print(f"Parameter Tensor: {param.numel()}")
```
output 
```
Parameter Name: model.embed_tokens.weight
Parameter Tensor: 1050673152
Parameter Name: model.layers.0.input_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.0.post_attention_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.1.input_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.1.post_attention_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.2.input_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.2.post_attention_layernorm.weight
..................
Parameter Name: model.layers.78.input_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.78.post_attention_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.79.input_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.layers.79.post_attention_layernorm.weight
Parameter Tensor: 8192
Parameter Name: model.norm.weight
Parameter Tensor: 8192
Parameter Name: lm_head.weight
Parameter Tensor: 1050673152
```
Hope this clarifies the part about the number of parameters


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.