ValueError: Unrecognized configuration class with GPTQConfig
Author: vishrathi
Pull request: no
Labels: name: Quantization

Hi, 

I tried the following code in colab to quantize distilbert using GPTQ as per instruction given my HuggingFace (https://huggingface.co/docs/transformers/main/en/quantization/gptq):

from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig
import torch
model_id = "distilbert/distilbert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(model_id)
gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)
quantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_config=gptq_config)

I get the following error. What am I missing? Do I need to import something?

Thanks.

ValueError                                Traceback (most recent call last)
[<ipython-input-10-62077e2f4cf2>](https://localhost:8080/#) in <cell line: 6>()
      4 tokenizer = AutoTokenizer.from_pretrained(model_id)
      5 gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)
----> 6 quantized_model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_config=gptq_config)

[/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py](https://localhost:8080/#) in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    565                 pretrained_model_name_or_path, *model_args, config=config, **hub_kwargs, **kwargs
    566             )
--> 567         raise ValueError(
    568             f"Unrecognized configuration class {config.__class__} for this kind of AutoModel: {cls.__name__}.\n"
    569             f"Model type should be one of {', '.join(c.__name__ for c in cls._model_mapping.keys())}."

ValueError: Unrecognized configuration class <class 'transformers.models.distilbert.configuration_distilbert.DistilBertConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of BartConfig, BertConfig, BertGenerationConfig, BigBirdConfig, BigBirdPegasusConfig, BioGptConfig, BlenderbotConfig, BlenderbotSmallConfig, BloomConfig, CamembertConfig, LlamaConfig, CodeGenConfig, CohereConfig, CpmAntConfig, CTRLConfig, Data2VecTextConfig, DbrxConfig, ElectraConfig, ErnieConfig, FalconConfig, FuyuConfig, GemmaConfig, Gemma2Config, GitConfig, GPT2Config, GPT2Config, GPTBigCodeConfig, GPTNeoConfig, GPTNeoXConfig, GPTNeoXJapaneseConfig, GPTJConfig, JambaConfig, JetMoeConfig, LlamaConfig, MambaConfig, Mamba2Config, MarianConfig, MBartConfig, MegaConfig, MegatronBertConfig, MistralConfig, MixtralConfig, MptConfig, MusicgenConfig, MusicgenMelodyConfig, MvpConfig, NemotronConfig, OlmoConfig, OpenLlamaConfig, OpenAIGPTConfig, OPTConfig, PegasusConfig, PersimmonConfig, PhiConfig, Phi3Config, PLBartConfig, ProphetNetConfig, QDQBertConfig, Qwen2Config, Qwen2MoeConfig, RecurrentGemmaConfig, ReformerConfig, RemBertConfig, RobertaConfig, RobertaPreLayerNormConfig, RoCBertConfig, RoFormerConfig, RwkvConfig, Speech2Text2Config, StableLmConfig, Starcoder2Config, TransfoXLConfig, TrOCRConfig, WhisperConfig, XGLMConfig, XLMConfig, XLMProphetNetConfig, XLMRobertaConfig, XLMRobertaXLConfig, XLNetConfig, XmodConfig.

Comments:
cc @SunMarc 

Hey @vishrathi, this happens because you are trying to load the distillbert model with `AutoModelForCausalLM`. You need to switch to `AutoModelForMaskedLM` instead. 

Thanks @SunMarc. I changed the code to:

from transformers import AutoModelForMaskedLM, AutoTokenizer, GPTQConfig

model_id = "distilbert/distilbert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model_id)
gptq_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)
quantized_model = AutoModelForMaskedLM.from_pretrained(model_id, device_map="auto", quantization_config=gptq_config)

But now I got the following error. Your help is much appreciated. What value should I set for device_map?

ValueError                                Traceback (most recent call last)
[<ipython-input-3-61e3e56f0d9d>](https://q1z52omk54-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240912-060302_RC00_673778578#) in <cell line: 1>()
----> 1 quantized_model = AutoModelForMaskedLM.from_pretrained(model_id, device_map="auto", quantization_config=gptq_config)

2 frames
[/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py](https://q1z52omk54-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240912-060302_RC00_673778578#) in _get_no_split_modules(self, device_map)
   1977                 if isinstance(module, PreTrainedModel):
   1978                     if module._no_split_modules is None:
-> 1979                         raise ValueError(
   1980                             f"{module.__class__.__name__} does not support `device_map='{device_map}'`. To implement support, the model "
   1981                             "class needs to implement the `_no_split_modules` attribute."

ValueError: DistilBertForMaskedLM does not support `device_map='auto'`. To implement support, the model class needs to implement the `_no_split_modules` attribute.

Can you try without the `device_map` arg ! This should be optional 

Thanks @SunMarc. I tried without device_map arg. But got the following error:

`low_cpu_mem_usage` was None, now set to True since model is quantized.
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-5-ca5b27dc471b>](https://q1z52omk54-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240912-060302_RC00_673778578#) in <cell line: 1>()
----> 1 quantized_model = AutoModelForMaskedLM.from_pretrained(model_id, quantization_config=gptq_config)

6 frames
[/usr/local/lib/python3.10/dist-packages/transformers/configuration_utils.py](https://q1z52omk54-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240912-060302_RC00_673778578#) in __getattribute__(self, key)
    262         if key != "attribute_map" and key in super().__getattribute__("attribute_map"):
    263             key = super().__getattribute__("attribute_map")[key]
--> 264         return super().__getattribute__(key)
    265 
    266     def __init__(self, **kwargs):

AttributeError: 'DistilBertConfig' object has no attribute 'use_cache'

Hmmm just tested a bit with this architecture. GPTQ quantization process is better suited for llms, so it requires a bit more work to make it work with encoder models. I suggest testing other quantization methods that doesn't require this calibration step. For example, you can have a look at bnb or torchao quantization. 

Thanks @SunMarc. I tried bnb with following code in Google colab (from https://huggingface.co/docs/accelerate/en/usage_guides/quantization) and I got the following error which follows the code. Can you help me out please what is going wrong? Thanks.

!pip install bitsandbytes
from accelerate import init_empty_weights
from transformers import DistilBertTokenizer, DistilBertModel
model = DistilBertModel.from_pretrained("distilbert-base-cased")
model_config = model.config
model_config.model_type = 'distilbert/distilbert-base-cased'
model_config.vocab_size = 50257
model_config.block_size = 1024
with init_empty_weights():
    empty_model = DistilBertModel(model_config)
import torch
from accelerate.utils import BnbQuantizationConfig
bnb_quantization_config = BnbQuantizationConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.bfloat16,        bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4")
from huggingface_hub import snapshot_download
weights_location = snapshot_download(repo_id="distilbert/distilbert-base-cased")
from accelerate.utils import load_and_quantize_model
quantized_model = load_and_quantize_model(empty_model, weights_location=weights_location, bnb_quantization_config=bnb_quantization_config, device_map = "auto")

NotImplementedError                       Traceback (most recent call last)
[<ipython-input-8-65d51fc0720f>](https://mbfeqm3xsna-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240918-060111_RC00_675939473#) in <cell line: 2>()
      1 from accelerate.utils import load_and_quantize_model
----> 2 quantized_model = load_and_quantize_model(empty_model, weights_location=weights_location, bnb_quantization_config=bnb_quantization_config, device_map = "auto")

7 frames
[/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py](https://mbfeqm3xsna-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240918-060111_RC00_675939473#) in convert(t)
   1165             except NotImplementedError as e:
   1166                 if str(e) == "Cannot copy out of meta tensor; no data!":
-> 1167                     raise NotImplementedError(
   1168                         f"{e} Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() "
   1169                         f"when moving module from meta to a different device."

NotImplementedError: Cannot copy out of meta tensor; no data! Please use torch.nn.Module.to_empty() instead of torch.nn.Module.to() when moving module from meta to a different device.

Hi @SunMarc, 

I also tried torchao by following https://huggingface.co/docs/transformers/main/en/quantization/torchao but got into the following error. If you can not help with bitsandbytes, please help with torchao:

!pip install --upgrade torch torchao
import torch
import transformers
import torchao
from transformers import TorchAoConfig, AutoModelForMaskedLM, AutoTokenizer

model_name = "distilbert/distilbert-base-cased"
# We support int4_weight_only, int8_weight_only and int8_dynamic_activation_int8_weight
# More examples and documentations for arguments can be found in https://github.com/pytorch/ao/tree/main/torchao/quantization#other-available-quantization-techniques
quantization_config = TorchAoConfig("int4_weight_only", group_size=128)
quantized_model = AutoModelForMaskedLM.from_pretrained(model_name, device_map="auto", quantization_config=quantization_config)

tokenizer = AutoTokenizer.from_pretrained(model_name)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")

# compile the quantized model to get speedup
import torchao
torchao.quantization.utils.recommended_inductor_config_setter()
quantized_model = torch.compile(quantized_model, mode="max-autotune")

output = quantized_model.generate(**input_ids, max_new_tokens=10)
print(tokenizer.decode(output[0], skip_special_tokens=True))

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
[<ipython-input-6-9d4fd5e9fa66>](https://k5vqerabp3c-496ff2e9c6d22116-0-colab.googleusercontent.com/outputframe.html?vrz=colab_20240920-060127_RC00_676789073#) in <cell line: 1>()
----> 1 import TorchAoConfig, AutoModelForMaskedLM, AutoTokenizer
      2 
      3 model_name = "distilbert/distilbert-base-cased"
      4 # We support int4_weight_only, int8_weight_only and int8_dynamic_activation_int8_weight
      5 # More examples and documentations for arguments can be found in https://github.com/pytorch/ao/tree/main/torchao/quantization#other-available-quantization-techniques

ModuleNotFoundError: No module named 'TorchAoConfig'


Hi @SunMarc,

I want to quantize "distilbert/distilbert-base-cased". Please suggest which method can do it.

Thanks.

Hey @vishrathi, make sure to install the latest transformers ! Try the following two snippet ! 

### Torchao
```python
quantization_config = TorchAoConfig("int4_weight_only", group_size=128)
quantized_model = AutoModelForMaskedLM.from_pretrained(model_name, device_map="auto", quantization_config=quantization_config)
```

### BNB
```python
quantization_config = BitsAndBytesConfig(load_in_4bit=True)
quantized_model = AutoModelForMaskedLM.from_pretrained(model_name, device_map="auto", quantization_config=quantization_config)
```

Thanks a lot @SunMarc!! I could manage to get the quantization done using BNB.

Nice ! Closing this issue then