Is T5 model supported with HQQ quantization ? (AttributeError: 'HQQLinear' object has no attribute 'weight')
Author: szwagros
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.2
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	- compute_environment: LOCAL_MACHINE
	- distributed_type: NO
	- mixed_precision: no
	- use_cpu: False
	- debug: False
	- num_processes: 1
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: all
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- enable_cpu_affinity: False
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4080 SUPER




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I've created and save quantized version like this:

    quant_config = HqqConfig(nbits=4, group_size=64)

    model = T5EncoderModel.from_pretrained(
        '/storage/Models/FLUX.1-dev/',
        torch_dtype=torch.bfloat16,
        subfolder = "text_encoder_2",
        device_map="cuda", 
        quantization_config=quant_config
    )

    model.save_pretrained(
        "./quantized_pipeline/",
        safe_serialization=True  # Use safetensors format
    )
    
    
During inference I create flux pipeline:

        text_encoder_2 = T5EncoderModel.from_pretrained(
            "./quantized_pipeline/",
            subfolder="text_encoder_2",
            torch_dtype=torch.bfloat16,
            device_map="cuda"
        )


        self.pipeline: FluxPipeline = FluxPipeline.from_pretrained(
            self.model_config.path,
            torch_dtype=torch.bfloat16,
            local_files_only=True,
            text_encoder_2=text_encoder_2
        )    
    
    
    
But when I actually start inference I always get this error:


  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 339, in forward
    forwarded_states = self.DenseReluDense(forwarded_states)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 316, in forward
    isinstance(self.wo.weight, torch.Tensor)
               ^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1931, in __getattr__
    raise AttributeError(
AttributeError: 'HQQLinear' object has no attribute 'weight'


Is it because T5 is not supported or am I doing something wrong ?


### Expected behavior

Flux pipeline should work without errors. 

Comments:
cc @SunMarc @MekkCyber for quantization

Just mentioning that this conversion was done before (not sure how) https://huggingface.co/HighCWu/FLUX.1-dev-4bit/tree/main/text_encoder_2

Linking this issue here: https://github.com/mobiusml/hqq/issues/150

The suggested [method](https://github.com/mobiusml/hqq/issues/150#issuecomment-2665731834) crashed with
```
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1939, in state_dict
    module.state_dict(destination=destination, prefix=prefix + name + '.', keep_vars=keep_vars)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1939, in state_dict
    module.state_dict(destination=destination, prefix=prefix + name + '.', keep_vars=keep_vars)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1939, in state_dict
    module.state_dict(destination=destination, prefix=prefix + name + '.', keep_vars=keep_vars)
  [Previous line repeated 4 more times]
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 577, in state_dict
    raise Exception(
Exception: Unsupported serialization for quantized scale/zero and self.encoded_state_dict=True
```
Deepseek R1 suggested me to fix it with 
```python
 layer.encoded_state_dict = False
```
which worked, but there was another issue

```

File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2934, in save_pretrained
    shared_names, disjoint_names = _find_disjoint(shared_ptrs.values(), state_dict)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 614, in _find_disjoint
    areas.append((tensor.data_ptr(), _end_ptr(tensor), name))
                  ^^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'data_ptr'
```

Then I updated `hqq` an it worked (without layer.encoded_state_dict patches)

So to fix the issue, we just need to update hqq @remixer-dec ?

It actually works if i quantize model with this config:


```
  quant_config = HqqConfig(
      nbits=4,
      group_size=64,
      skip_modules=['encoder.block.layer.DenseReluDense.wo'],
      axis=0
  )

```

or just add dummy weight to every HQQLInear layer:

```
def add_weight_to_hqq_modules(model):
    for name, module in model.named_modules():
        if isinstance(module, HQQLinear):
            in_features = getattr(module, 'in_features', 1)
            out_features = getattr(module, 'out_features', 1)

            dummy_weight = torch.empty(
                (1, 1),
                dtype=torch.bfloat16,
                device='cuda'
            )

            module.register_buffer('weight', dummy_weight, persistent=False)

    return model
```
Without this workarounds (quatnt config or dummy weight)  it errors out here:


https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#L316





@SunMarc 
using the code from the issue / official hqq readme, 
saving works but the latest version of hqq (hqq-0.2.3.post1) breaks [flux-4bit](https://github.com/HighCWu/flux-4bit) repo because the repo uses old [quant_config format](https://huggingface.co/HighCWu/FLUX.1-dev-4bit/blob/main/text_encoder_2/config.json#L25-L39) and it saves with new nested one:
```
    "quantization_config": {
        "quant_config": {
            "offload_meta": false,
            "scale_quant_params": null,
            "weight_quant_params": {
                "axis": 0,
                "channel_wise": true,
                "group_size": 64,
                "nbits": 4,
                "optimize": true,
                "round_zero": true,
                "view_as_float": false
            },
            "zero_quant_params": null
        },
        "quant_method": "hqq",
        "skip_modules": [
            "encoder.block.layer.DenseReluDense.wo"
        ]
    },

```
so it crashes with
```
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 398, in __init__
    self.quant_config.pop("offload_meta")
KeyError: 'offload_meta'
```
however if you replace quant_config in config.json file manually (or patch flux-4bit with `quant_config=quant_config.get("quant_config", quant_config)`), flux-4bit works with old version of hqq
and new version of hqq causes another issue (both with original code and with skip and with dummy weight):
```
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 729, in load_state_dict
    self.W_q = state_dict.pop("W_q")
    ^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1749, in __setattr__
    raise TypeError(f"cannot assign '{torch.typename(value)}' as parameter '{name}' "
TypeError: cannot assign 'torch.cuda.ByteTensor' as parameter 'W_q' (torch.nn.Parameter or None expected)
```
with the first variant suggested by @szwagros conversion works, but it breaks even old version of hqq, after replacing the config file  this issue appears:
```
    module.load_state_dict(state_dict)
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 720, in load_state_dict
    self.W_q = state_dict.pop("W_q")
               ^^^^^^^^^^^^^^^^^^^^^
KeyError: 'W_q'
```
using patch code that I linked, it does not save quant config into config.json and you have to add it manually, after it, it works with old hqq (0.2.2)

@szwagros do you mind sharing if you use it natively with diffusers' FluxPipeline?

TLDR: use latest version to quantize, add new quant_config handling, use old hqq in inference with flux-4bit until (if ever) the issue is fixed in hqq

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.