Repository: huggingface/transformers
Pull request: Fix HQQ model param device transfer issue
# What does this PR do?

Fixes #36254

And this PR makes it possible to infer bnb-4bit Flux with hqq-4bit quantized T5 model in the diffusers pipeline.

I uploaded models to [HighCWu/FLUX.1-dev-bnb-hqq-4bit](https://huggingface.co/HighCWu/FLUX.1-dev-bnb-hqq-4bit).

I use hqq-4bit to quantize T5. It seems to quantize T5 better than bnb-4bit.

The issue fixed by this PR mentioned my old model repo _Originally posted by @Rocketknight1 in [#36254](https://github.com/huggingface/transformers/issues/36254#issuecomment-2665917926)_ that I used some hacked code to quantize a T5 model, and used some hacked code to load the quantized model. Later, the hqq code base was upgraded and the old code was no longer available. So I submitted this PR to make the new format of hqq-4bit T5 model work properly in the diffusers pipeline.

@SunMarc


Problem statement:
Is T5 model supported with HQQ quantization ? (AttributeError: 'HQQLinear' object has no attribute 'weight')
### System Info

- `transformers` version: 4.48.2
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.11.11
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	- compute_environment: LOCAL_MACHINE
	- distributed_type: NO
	- mixed_precision: no
	- use_cpu: False
	- debug: False
	- num_processes: 1
	- machine_rank: 0
	- num_machines: 1
	- gpu_ids: all
	- rdzv_backend: static
	- same_network: True
	- main_training_function: main
	- enable_cpu_affinity: False
	- downcast_bf16: no
	- tpu_use_cluster: False
	- tpu_use_sudo: False
	- tpu_env: []
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 4080 SUPER




### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I've created and save quantized version like this:

    quant_config = HqqConfig(nbits=4, group_size=64)

    model = T5EncoderModel.from_pretrained(
        '/storage/Models/FLUX.1-dev/',
        torch_dtype=torch.bfloat16,
        subfolder = "text_encoder_2",
        device_map="cuda", 
        quantization_config=quant_config
    )

    model.save_pretrained(
        "./quantized_pipeline/",
        safe_serialization=True  # Use safetensors format
    )
    
    
During inference I create flux pipeline:

        text_encoder_2 = T5EncoderModel.from_pretrained(
            "./quantized_pipeline/",
            subfolder="text_encoder_2",
            torch_dtype=torch.bfloat16,
            device_map="cuda"
        )


        self.pipeline: FluxPipeline = FluxPipeline.from_pretrained(
            self.model_config.path,
            torch_dtype=torch.bfloat16,
            local_files_only=True,
            text_encoder_2=text_encoder_2
        )    
    
    
    
But when I actually start inference I always get this error:


  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 339, in forward
    forwarded_states = self.DenseReluDense(forwarded_states)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/accelerate/hooks.py", line 170, in new_forward
    output = module._old_forward(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/transformers/models/t5/modeling_t5.py", line 316, in forward
    isinstance(self.wo.weight, torch.Tensor)
               ^^^^^^^^^^^^^^
  File "/home/szwagros/anaconda3/envs/image/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1931, in __getattr__
    raise AttributeError(
AttributeError: 'HQQLinear' object has no attribute 'weight'


Is it because T5 is not supported or am I doing something wrong ?


### Expected behavior

Flux pipeline should work without errors.

Hints from issue discussion:
cc @SunMarc @MekkCyber for quantization
Just mentioning that this conversion was done before (not sure how) https://huggingface.co/HighCWu/FLUX.1-dev-4bit/tree/main/text_encoder_2
Linking this issue here: https://github.com/mobiusml/hqq/issues/150

The suggested [method](https://github.com/mobiusml/hqq/issues/150#issuecomment-2665731834) crashed with
```
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1939, in state_dict
    module.state_dict(destination=destination, prefix=prefix + name + '.', keep_vars=keep_vars)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1939, in state_dict
    module.state_dict(destination=destination, prefix=prefix + name + '.', keep_vars=keep_vars)
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1939, in state_dict
    module.state_dict(destination=destination, prefix=prefix + name + '.', keep_vars=keep_vars)
  [Previous line repeated 4 more times]
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 577, in state_dict
    raise Exception(
Exception: Unsupported serialization for quantized scale/zero and self.encoded_state_dict=True
```
Deepseek R1 suggested me to fix it with 
```python
 layer.encoded_state_dict = False
```
which worked, but there was another issue

```

File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 2934, in save_pretrained
    shared_names, disjoint_names = _find_disjoint(shared_ptrs.values(), state_dict)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/transformers/modeling_utils.py", line 614, in _find_disjoint
    areas.append((tensor.data_ptr(), _end_ptr(tensor), name))
                  ^^^^^^^^^^^^^^^
AttributeError: 'int' object has no attribute 'data_ptr'
```

Then I updated `hqq` an it worked (without layer.encoded_state_dict patches)
So to fix the issue, we just need to update hqq @remixer-dec ?
It actually works if i quantize model with this config:


```
  quant_config = HqqConfig(
      nbits=4,
      group_size=64,
      skip_modules=['encoder.block.layer.DenseReluDense.wo'],
      axis=0
  )

```

or just add dummy weight to every HQQLInear layer:

```
def add_weight_to_hqq_modules(model):
    for name, module in model.named_modules():
        if isinstance(module, HQQLinear):
            in_features = getattr(module, 'in_features', 1)
            out_features = getattr(module, 'out_features', 1)

            dummy_weight = torch.empty(
                (1, 1),
                dtype=torch.bfloat16,
                device='cuda'
            )

            module.register_buffer('weight', dummy_weight, persistent=False)

    return model
```
Without this workarounds (quatnt config or dummy weight)  it errors out here:


https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_t5.py#L316




@SunMarc 
using the code from the issue / official hqq readme, 
saving works but the latest version of hqq (hqq-0.2.3.post1) breaks [flux-4bit](https://github.com/HighCWu/flux-4bit) repo because the repo uses old [quant_config format](https://huggingface.co/HighCWu/FLUX.1-dev-4bit/blob/main/text_encoder_2/config.json#L25-L39) and it saves with new nested one:
```
    "quantization_config": {
        "quant_config": {
            "offload_meta": false,
            "scale_quant_params": null,
            "weight_quant_params": {
                "axis": 0,
                "channel_wise": true,
                "group_size": 64,
                "nbits": 4,
                "optimize": true,
                "round_zero": true,
                "view_as_float": false
            },
            "zero_quant_params": null
        },
        "quant_method": "hqq",
        "skip_modules": [
            "encoder.block.layer.DenseReluDense.wo"
        ]
    },

```
so it crashes with
```
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 398, in __init__
    self.quant_config.pop("offload_meta")
KeyError: 'offload_meta'
```
however if you replace quant_config in config.json file manually (or patch flux-4bit with `quant_config=quant_config.get("quant_config", quant_config)`), flux-4bit works with old version of hqq
and new version of hqq causes another issue (both with original code and with skip and with dummy weight):
```
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 729, in load_state_dict
    self.W_q = state_dict.pop("W_q")
    ^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1749, in __setattr__
    raise TypeError(f"cannot assign '{torch.typename(value)}' as parameter '{name}' "
TypeError: cannot assign 'torch.cuda.ByteTensor' as parameter 'W_q' (torch.nn.Parameter or None expected)
```
with the first variant suggested by @szwagros conversion works, but it breaks even old version of hqq, after replacing the config file  this issue appears:
```
    module.load_state_dict(state_dict)
  File "/opt/conda/lib/python3.11/site-packages/hqq/core/quantize.py", line 720, in load_state_dict
    self.W_q = state_dict.pop("W_q")
               ^^^^^^^^^^^^^^^^^^^^^
KeyError: 'W_q'
```
using patch code that I linked, it does not save quant config into config.json and you have to add it manually, after it, it works with old hqq (0.2.2)

@szwagros do you mind sharing if you use it natively with diffusers' FluxPipeline?

TLDR: use latest version to quantize, add new quant_config handling, use old hqq in inference with flux-4bit until (if ever) the issue is fixed in hqq
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Commit message: Merge branch 'main' into hqq-compat-with-diffusers

Patch to review:
diff --git a/src/transformers/modeling_utils.py b/src/transformers/modeling_utils.py
index 480e6f3f3f32..83a09a864baf 100644
--- a/src/transformers/modeling_utils.py
+++ b/src/transformers/modeling_utils.py
@@ -3892,7 +3892,20 @@ def get_memory_footprint(self, return_buffers=True):
     @wraps(torch.nn.Module.cuda)
     def cuda(self, *args, **kwargs):
         if getattr(self, "quantization_method", None) == QuantizationMethod.HQQ:
-            raise ValueError("`.cuda` is not supported for HQQ-quantized models.")
+            from hqq.core.quantize import HQQLinear
+
+            # Since HQQLinear stores some tensors in the 'meta' attribute,
+            # it's necessary to manually call the `cuda` method on HQQLinear layers.
+            self = super().cuda(*args, **kwargs)
+            for module in self.modules():
+                if isinstance(module, HQQLinear):
+                    if len(args) > 0:
+                        device = args[0]
+                    else:
+                        device = kwargs.get("device", "cuda")
+                    module.cuda(device)
+            return self
+
         # Checks if the model has been loaded in 4-bit or 8-bit with BNB
         if getattr(self, "quantization_method", None) == QuantizationMethod.BITS_AND_BYTES:
             if getattr(self, "is_loaded_in_8bit", False):
@@ -3921,7 +3934,25 @@ def to(self, *args, **kwargs):
                     break
 
         if getattr(self, "quantization_method", None) == QuantizationMethod.HQQ:
-            raise ValueError("`.to` is not supported for HQQ-quantized models.")
+            from hqq.core.quantize import HQQLinear
+
+            # Since HQQLinear stores some tensors in the 'meta' attribute, we must
+            # explicitly move the parameters to the target device for each HQQLinear layer after `to`.
+            self = super().to(*args, **kwargs)
+            for module in self.modules():
+                if isinstance(module, HQQLinear):
+                    if "dtype" in kwargs:
+                        dtype = kwargs.get("dtype", None)
+                    elif dtype_present_in_args:
+                        dtype = arg
+                    else:
+                        dtype = None
+                    # Due to the current messy implementation of HQQLinear, updating `compute_dtype`
+                    # followed by calling the `cuda` method achieves the intended behavior of `to`,
+                    # even when the target device is CPU.
+                    module.compute_dtype = dtype or module.compute_dtype
+                    module.cuda(kwargs.get("device", None) or args[0])
+            return self
 
         if dtype_present_in_args and getattr(self, "quantization_method", None) == QuantizationMethod.QUARK:
             raise ValueError("Casting a Quark quantized model to a new `dtype` is not supported.")
diff --git a/src/transformers/quantizers/quantizer_hqq.py b/src/transformers/quantizers/quantizer_hqq.py
index 38d8a15cbfcf..aa83e493f225 100755
--- a/src/transformers/quantizers/quantizer_hqq.py
+++ b/src/transformers/quantizers/quantizer_hqq.py
@@ -198,6 +198,15 @@ def create_quantized_param(
         if is_hqq_available():
             from hqq.core.quantize import HQQLinear
 
+            # TODO: This is a compatibility hack. HQQ-quantized linear layers do not have a `weight` attribute,
+            # but some models attempt to access `weight.dtype` during the forward pass. To prevent runtime errors,
+            # we patch HQQLinear with a dummy `weight` property that returns an empty tensor with the correct dtype and device.
+            @property
+            def weight(_self: HQQLinear):
+                return torch.empty(0, dtype=_self.compute_dtype, device=_self.device)
+
+            HQQLinear.weight = weight
+
         module, tensor_name = get_module_from_name(model, param_name)
         layer_name = ".".join(param_name.split(".")[:-1])
         parent_module = find_parent(model, layer_name)


Code review comments:
diff_hunk: @@ -3921,7 +3934,25 @@ def to(self, *args, **kwargs):
                     break
 
         if getattr(self, "quantization_method", None) == QuantizationMethod.HQQ:
-            raise ValueError("`.to` is not supported for HQQ-quantized models.")
+            from hqq.core.quantize import HQQLinear
+
+            # Since HQQLinear stores some tensors in the 'meta' attribute, we must
+            # explicitly move the parameters to the target device for each HQQLinear layer after `to`.
+            self = super().to(*args, **kwargs)
path: src/transformers/modeling_utils.py
text: @user1:
same here 

@author:
Done by removing `self =`

diff_hunk: @@ -3921,7 +3934,25 @@ def to(self, *args, **kwargs):
                     break
 
         if getattr(self, "quantization_method", None) == QuantizationMethod.HQQ:
-            raise ValueError("`.to` is not supported for HQQ-quantized models.")
+            from hqq.core.quantize import HQQLinear
+
+            # Since HQQLinear stores some tensors in the 'meta' attribute, we must
+            # explicitly move the parameters to the target device for each HQQLinear layer after `to`.
+            self = super().to(*args, **kwargs)
+            for module in self.modules():
+                if isinstance(module, HQQLinear):
+                    if "dtype" in kwargs:
+                        dtype = kwargs.get("dtype", None)
+                    elif dtype_present_in_args:
+                        dtype = arg
+                    else:
+                        dtype = None
+                    # Due to the current messy implementation of HQQLinear, updating `compute_dtype`
+                    # followed by calling the `cuda` method achieves the intended behavior of `to`,
+                    # even when the target device is CPU.
+                    module.compute_dtype = dtype or module.compute_dtype
path: src/transformers/modeling_utils.py
text: @user1:
assign only if dtype is not None, it will make things clearer 

@author:
Done

diff_hunk: @@ -3921,7 +3934,25 @@ def to(self, *args, **kwargs):
                     break
 
         if getattr(self, "quantization_method", None) == QuantizationMethod.HQQ:
-            raise ValueError("`.to` is not supported for HQQ-quantized models.")
+            from hqq.core.quantize import HQQLinear
+
+            # Since HQQLinear stores some tensors in the 'meta' attribute, we must
+            # explicitly move the parameters to the target device for each HQQLinear layer after `to`.
+            self = super().to(*args, **kwargs)
+            for module in self.modules():
+                if isinstance(module, HQQLinear):
+                    if "dtype" in kwargs:
+                        dtype = kwargs.get("dtype", None)
+                    elif dtype_present_in_args:
+                        dtype = arg
+                    else:
+                        dtype = None
+                    # Due to the current messy implementation of HQQLinear, updating `compute_dtype`
+                    # followed by calling the `cuda` method achieves the intended behavior of `to`,
+                    # even when the target device is CPU.
+                    module.compute_dtype = dtype or module.compute_dtype
+                    module.cuda(kwargs.get("device", None) or args[0])
path: src/transformers/modeling_utils.py
text: @user1:
have the same code as with cuda, makes things clearer

@author:
Done

diff_hunk: @@ -3892,7 +3892,20 @@ def get_memory_footprint(self, return_buffers=True):
     @wraps(torch.nn.Module.cuda)
     def cuda(self, *args, **kwargs):
         if getattr(self, "quantization_method", None) == QuantizationMethod.HQQ:
-            raise ValueError("`.cuda` is not supported for HQQ-quantized models.")
+            from hqq.core.quantize import HQQLinear
+
+            # Since HQQLinear stores some tensors in the 'meta' attribute,
+            # it's necessary to manually call the `cuda` method on HQQLinear layers.
+            self = super().cuda(*args, **kwargs)
path: src/transformers/modeling_utils.py
text: @user1:
don't need to assign self here

@author:
Done by removing `self =`

diff_hunk: @@ -3921,7 +3934,25 @@ def to(self, *args, **kwargs):
                     break
 
         if getattr(self, "quantization_method", None) == QuantizationMethod.HQQ:
-            raise ValueError("`.to` is not supported for HQQ-quantized models.")
+            from hqq.core.quantize import HQQLinear
+
+            # Since HQQLinear stores some tensors in the 'meta' attribute, we must
+            # explicitly move the parameters to the target device for each HQQLinear layer after `to`.
+            self = super().to(*args, **kwargs)
+            for module in self.modules():
+                if isinstance(module, HQQLinear):
+                    if "dtype" in kwargs:
+                        dtype = kwargs.get("dtype", None)
+                    elif dtype_present_in_args:
+                        dtype = arg
+                    else:
+                        dtype = None
path: src/transformers/modeling_utils.py
text: @user1:
using `dtype_present_in_args` should be enough 

@author:
Thanks for the suggestion!

You're right that using `dtype_present_in_args` as an indicator is sufficient for checking the existence of `dtype`. However, in the current version, the value of `dtype` itself isn't always accessible later in the code, especially when it's passed via `args` rather than `kwargs`. That's why I included the explicit extraction logic again in the HQQ handling block.

That said, I agree the current structure can be made cleaner. I have a new iead. How about update the implementation to extract `dtype` once during the initial check:

```python
dtype = kwargs.get("dtype", None)
dtype_present_in_args = dtype is not None

if not dtype_present_in_args:
    for arg in args:
        if isinstance(arg, torch.dtype):
            dtype = arg
            dtype_present_in_args = True
            break
```

With this change, we no longer need to re-check `kwargs` or `args` later when handling `HQQLinear`, which improves readability and avoids redundancy.

Let me know what you think!


@user1:
Oh okay I see, it's fine, let's keep your current version !