Can't load a 4bit GPTQ Model from the hub
Author: colinpannikkat
Pull request: no

### System Info

- `transformers` version: 4.32.1
- Platform: Linux-3.10.0-1160.83.1.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.6
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.21.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@SunMarc @yous @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to load a 4bit GPTQ model using

> from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("TheBloke/vicuna-7B-v1.5-GPTQ", device_map="auto", load_in_4bit=True)
print("model loaded")

In doing so I get the output:
>Traceback (most recent call last):
  File "/nfs/hpc/share/pannikkc/llm-attacks-quantized/experiments/launch_scripts/test.py", line 64, in <module>
    model = AutoModelForCausalLM.from_pretrained("TheBloke/vicuna-7B-v1.5-GPTQ", device_map="auto", load_in_4bit=True)
  File "/nfs/hpc/share/pannikkc/llm-attacks-quantized/env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 516, in from_pretrained
    return model_class.from_pretrained(
  File "/nfs/hpc/share/pannikkc/llm-attacks-quantized/env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2475, in from_pretrained
    loading_attr_dict = quantization_config.get_loading_attributes()
AttributeError: 'BitsAndBytesConfig' object has no attribute 'get_loading_attributes'

If I add a GPTQConfig(bits=4), then the output is
>You passed `quantization_config` to `from_pretrained` but the model you're loading already has a `quantization_config` attribute and has already quantized weights. However, loading attributes (e.g. disable_exllama, use_cuda_fp16) will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored.
Traceback (most recent call last):
  File "/nfs/hpc/share/pannikkc/llm-attacks-quantized/experiments/launch_scripts/test.py", line 66, in <module>
    model = AutoModelForCausalLM.from_pretrained("TheBloke/vicuna-7B-v1.5-GPTQ", device_map="auto", load_in_4bit=True, quantization_config=config)
  File "/nfs/hpc/share/pannikkc/llm-attacks-quantized/env/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 516, in from_pretrained
    return model_class.from_pretrained(
  File "/nfs/hpc/share/pannikkc/llm-attacks-quantized/env/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2888, in from_pretrained
    llm_int8_skip_modules = quantization_config.llm_int8_skip_modules
AttributeError: 'GPTQConfig' object has no attribute 'llm_int8_skip_modules'

### Expected behavior

I would expect the model to load.

Comments:
cc @younesbelkada as well 😉 

To load quantized GPTQ model with transformers, you just need to call `model = AutoModelForCausalLM.from_pretrained("TheBloke/vicuna-7B-v1.5-GPTQ", device_map="auto")`. The `load_in_4bit` arg is for bitsandbytes quantization method. We will deprecate it in later version as it is quite confusing since we added a new quantization method. 

That works! Thank you very much. Another question I have is regarding the data type of the model after loading. Even if it is a 4-bit quantized model, when I check the model datatype with `model.dtype`, the data type of the model is still torch.float16. Shouldn't it be fp4, or am I misunderstanding the quantization process? I also noticed that the inference is still slower than the non-quantized version of the model, shouldn't it be faster?

Happy that it works ! `model.dtype` show the dtype of the model that was used to perform the quantization. There is no benefit in modifying this attribute and it will only confuse the user. You can print the model and see that the linear layers were modified. You will see that the weights are not in `float16` but in `int32`. We will soon publish a new blogpost about performance of GPTQ , stay tuned. There is already a benchamark in the announcement blogpost i think. It is comparable to fp16 but in some cases it is indeed slower. 

Thank you!