AttributeError: 'HQQLinear' object has no attribute 'weight'
Author: mxjmtxrm
Pull request: no

### System Info

- `transformers` version: 4.41.0.dev0
- Platform: Linux-5.15.0-92-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.0a0+81ea7a4 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@SunMarc and @younesbelkada

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I load model with HQQ quantization config,  I met the following error:
```
File "/workspace/code/utils.py", line 88, in create_and_prepare_model
    model = AutoModelForCausalLM.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 3693, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4127, in _load_pretrained_model
    new_error_msgs, offload_index, state_dict_index = _load_state_dict_into_meta_model(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 848, in _load_state_dict_into_meta_model
    old_param = getattr(old_param, split)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1687, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'HQQLinear' object has no attribute 'weight'
```
It seems that there is a bug in `quantizer_hqq.py`:
```
def check_quantized_param(
        self,
        model: "PreTrainedModel",
        param_value: "torch.Tensor",
        param_name: str,
        state_dict: Dict[str, Any],
        **kwargs,
    ) -> bool:
        module, tensor_name = get_module_from_name(model, param_name)

        return isinstance(module, torch.nn.Linear)

```
In the above code, it should check if the tensor_name is 'bias'. Otherwise the layer will be replace by HQQlinear if the tensor_name is bias.
How to solve this problem?

### Expected behavior

--

Comments:
Hi @mxjmtxrm 
Can you share which model are you trying to quantize?

It is my own model based on hf llama2 7B. I just modify the bias of qkv proj is True.  So the pretrained ckpt contains `.bias`, and then the above error arised.

I use the distil-whisper model. I'm getting the same error. BitsAndBytesConfig optimization method works. HQQ method gives error.

@mobicham,

Can you check?

Can you share a code snippet to reproduce this please?

Thanks everyone ! Indeed I was able to repro with: 

```python
from transformers import AutoModelForSpeechSeq2Seq, HqqConfig

model_id = "distil-whisper/distil-large-v2"

quant_config  = HqqConfig(nbits=1, group_size=64, quant_zero=False, quant_scale=False, axis=0)

model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, quantization_config=quant_config, device_map="cuda")
print(model)
```

Fixed it, will do a PR right now.

Awesome, thanks !


Hey,

I'm running into the same issue again when trying to quantize `Qwen2-VL` using `hqq`. Specifically, the `get_dtype` and `get_device` methods seem to be the culprits in this very case when trying to perform inference ([source](https://github.com/huggingface/transformers/blob/v4.52.4/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L974)).

I'm still not entirely convinced if this is a `transformers` or `hqq` issue at this point. Accordingly, I'm also not really certain what the best approach would be to prevent this from happening once and for all, and not having to handcraft more or less hacky fixes [like this](https://github.com/huggingface/transformers/pull/38466) for every model that is affected by this.

cc @mobicham @younesbelkada 

@kurzdev that seems to be a `transformers` issue indeed. It's not a good idea to use `.weight` to get the dtype/device since there's no guarantee the quant layers will have the `weight` attribute.
 
Meanwhile, you can use this trick to patch the quant linear layer (`HQQLinear` in your case) to manually add a dummy weight after quantization:
```Python
def patch_dummy_weight(model, dtype, device, quant_instances=HQQLinear):
    def set_layer(layer):
        if(not hasattr(layer, 'weight')):
            dummy_layer = torch.nn.Parameter((1, 1), dtype=dtype, device=device)
            layer.weight = torch.nn.Parameter(dummy_layer, requires_grad=False)
        return layer

    for name, layer in model.named_children():
        if isinstance(layer, quant_instances):
            setattr(model, name, set_layer(layer))
        else:
            patch_dummy_weight(model, dtype, device, quant_instances)
```

@mobicham hey, thanks - this didn't pop up on my feed for some reason. can confirm patching works! I'll look into maybe creating a PR removing such weight dependencies, allowing to circumvent doing this in the future :)