RuntimeError with dtype=torch.float16 and dtype=torch.bfloat16 in HQQ 4-bit, 8-bit quantized Granite4 models on RTX3050 TI
Author: prathamesh-chavan-22
Pull request: no

### Environment
- Transformers version: 4.57.0
- PyTorch version: 2.8.0+cu128
- Accelerate version: 1.10.1
- GPU: Nvidia RTX 3050Ti 4GB VRAM

### Steps to reproduce
1. Load HQQ 4-bit Granite model using dtype=torch.float16
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig

# 1️⃣ Quantization config
quant_config = HqqConfig(nbits=8, group_size=64)

# 2️⃣ Load model
model_name = "ibm-granite/granite-4.0-h-micro"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.bfloat16, 
    device_map="cuda",           # "auto or "cuda" if single GPU
    quantization_config=quant_config
)

# 3️⃣ Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 4️⃣ Prepare prompt
prompt = "Translate the following English sentence to French: Hello, how are you?"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

# 5️⃣ Generate output
with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,       # optional: enable sampling for more diverse output
        temperature=0.7,      # optional: adjust creativity
        top_p=0.9             # optional: nucleus sampling
    )

# 6️⃣ Decode output
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)
```

Observed behavior

RuntimeError: expected mat1 and mat2 to have the same dtype (struct c10::BFloat16 != float)

Expected behavior

Model should generate outputs without dtype errors

Workaround

1. Cast inputs to float32 before generation:
```python
inputs = {k: v.float() for k, v in inputs.items()}
```
2. Use dtype=torch.float32 or don't mention it at all

**Notes**

1. Docs show dtype=torch.float16

2. Error occurs in some PyTorch/Accelerate setups even on GPUs that support float16/bfloat16

@MekkCyber 


Comments:
cc @mekkcyber for quantization maybe?

Okkk 

Hi @prathamesh-chavan-22 the compiler throwing `RuntimeError: expected mat1 and mat2 to have the same dtype (struct c10::BFloat16 != float)` because of the you have loaded the model using bfloat16 and you are passing the input in float16. 
bfloat16 and float16 are completely different because of the precision difference.

use 
```
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=torch.float16, 
    device_map="cuda",           # "auto or "cuda" if single GPU
    quantization_config=quant_config
)
```

I would suggest you to use float16 or float32 because you are using _GPU: Nvidia RTX 3050Ti 4GB VRAM_ 

Its working for float32, and even by changing the inputs to same as it will raise the error, for both float16 and bfloat16, while my gpu does support bfloat16 i have tested it

Getting the same error here, it's not an hqq issue since quantizing with hqq lib works fine. I have no idea why the activations suddenly become fp32: 

### float32 error:
```Python
from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig
import torch

model_name = "Qwen/Qwen3-32B"
device, dtype = "cuda:0", torch.float16

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=dtype,
    device_map=device,
    quantization_config=HqqConfig(nbits=4, group_size=64),
)

with torch.no_grad():
    model(torch.randint(0, 100, (1,1), device='cuda:0', dtype=torch.int32))
```


### works:
```Python
from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig
import torch

model_name = "Qwen/Qwen3-32B"
device, dtype = "cuda:0", torch.float16

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    dtype=dtype,
    device_map=device,
)

from hqq.models.hf.base import AutoHQQHFModel
from hqq.core.quantize import BaseQuantizeConfig
quant_config = BaseQuantizeConfig(nbits=4, group_size=64) 
AutoHQQHFModel.quantize_model(model, quant_config=quant_config, compute_dtype=dtype, device=device)

with torch.no_grad():
    model(torch.randint(0, 100, (1,1), device='cuda:0', dtype=torch.int32))
```

Any update on this please?

Hi @mobicham ! sorry for the delay. We are refactoring our quantization API for the next release, hope this fixes the issue. If not we will fix any remaining problem afterwards.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

It seems that  the `HqqHfQuantizer` does not override its parent `HfQuantizer`'s `validate_environment` method since v4.57.0.  And the `dtype` is determined only in its constructor. Thus, the attribute `dtype` cannot be updated and fallbacks to `torch.float32`.

https://github.com/huggingface/transformers/blob/8ac2b916b042b1f78b75c9eb941c0f5d2cdd8e10/src/transformers/quantizers/quantizer_hqq.py#L57-L80

Downgrade to v4.56.X or upgrade to v5 solves the problem.