RuntimeError when loading llmcompressor W8A8 quantized model: int8 dtype in weight initialization
Author: AdelineXinyi
Pull request: no
Labels: name: Good First Issue

I'm trying to load the quantized model `RedHatAI/Qwen2.5-VL-7B-Instruct-quantized.w8a8` but encountering a dtype compatibility issue during model initialization. The model appears to be quantized using `llmcompressor` with W8A8 quantization scheme.

**Note**: I need to load this model without vLLM because I may need to add custom hooks for my research, so I'm looking for a direct loading method using transformers/llmcompressor.

## Error Message

```python
RuntimeError: expected a floating-point or complex dtype, but got dtype=torch.int8
```

**Full Stack Trace:**
```python
File "/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py", line 366, in _init_weights
    module.weight.data.normal_(mean=0.0, std=std)
File "/torch/_refs/__init__.py", line 6214, in normal_
    return normal(mean, std, self.shape, out=self, generator=generator)
...
RuntimeError: expected a floating-point or complex dtype, but got dtype=torch.int8
```

## Traceback

The error occurs during model weight initialization where transformers tries to call `normal_()` on int8 tensors. The `normal_()` function in PyTorch only works with floating-point tensors, but the quantized model contains int8 weights.

**Specific failure point:**
- File: `modeling_qwen2_5_vl.py`, line 366
- Function: `_init_weights()` 
- Operation: `module.weight.data.normal_(mean=0.0, std=std)`
- Issue: Trying to apply normal distribution to int8 tensors

## Model Information

Based on the model's `config.json`:
- **Quantization method**: `compressed-tensors`
- **Format**: `int-quantized` 
- **Scheme**: W8A8 (8-bit weights and activations)
- **Base model**: `Qwen/Qwen2.5-VL-7B-Instruct`
- **Compression ratio**: ~1.2x
- **Ignored layers**: All visual layers (`visual.blocks.*`, `visual.merger.*`, `lm_head`)

## What I've Tried

### 1. llmcompressor methods:
```python
# Method 1: TraceableQwen2_5_VLForConditionalGeneration
from llmcompressor.transformers.tracing import TraceableQwen2_5_VLForConditionalGeneration
model = TraceableQwen2_5_VLForConditionalGeneration.from_pretrained(
    model_path, device_map="auto", torch_dtype="auto", trust_remote_code=True
)

# Method 2: SparseAutoModelForCausalLM  
from llmcompressor.transformers import SparseAutoModelForCausalLM
model = SparseAutoModelForCausalLM.from_pretrained(
    model_path, device_map="auto", torch_dtype="auto", trust_remote_code=True
)
```

### 2. Standard transformers methods:
```python
# Method 3: Various dtype configurations
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_path,
    torch_dtype=torch.bfloat16,  # Also tried: torch.float16, "auto", None
    trust_remote_code=True,
    device_map="auto"
)

# Method 4: AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    model_path, trust_remote_code=True, torch_dtype="auto"
)
```

**All methods fail at the same weight initialization step, so I wonder should the model be loaded with `_fast_init=False` or other special parameters?**

## Additional Observations

1. **Warning about ignored layers**: The loader warns about missing visual layers, but this seems expected since they were ignored during quantization
2. **Model files exist**: The quantized model directory contains the expected `.safetensors` files and configuration
3. **Original model works**: The base `Qwen/Qwen2.5-VL-7B-Instruct` loads and works perfectly

## Environment

- **Python**: 3.10
- **PyTorch**: 2.7.0+cu126
- **Transformers**: 4.52.4
- **LLMCompressor**: 0.6.0
- **Compressed-tensors**: 0.10.2


This model was likely created using llmcompressor's oneshot quantization:
```python
from llmcompressor.modifiers.quantization import GPTQModifier
from llmcompressor.transformers import oneshot

recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W8A8", 
        sequential_targets=["Qwen2_5_VLDecoderLayer"],
        ignore=["lm_head", "re:visual.*"],
    ),
]
```
If this is more of an llmcompressor-specific model loading issue rather than a transformers compatibility issue, please let me know and I'll file this issue in the llmcompressor repository instead.



Comments:
Hello.

The problem arises from the fact that Pytorch doesn't implement a `normal` (or in-place `normal_` as in your case) for integers.

I tested locally a workaround that generates the weights data in float32 and then clips them to be int8 (or generally any int type). I'm not entirely sure about the correctness of such initialization and how it may impact things numerically. 

@SunMarc I'm actually wondering why we need to initialize weights when trying to load a model. Isn't that slow and pointless? Those would be immediately replaced by the actual weights of the pretrained model.

In any case, as said before, I can make a PR for the "int" normal init, let me know if that could help. 

> @SunMarc I'm actually wondering why we need to initialize weights when trying to load a model. Isn't that slow and pointless? Those would be immediately replaced by the actual weights of the pretrained model.

Actually, we try to not initialize the weights if the weights will be replaced by actual weights. Since this is related to llmcompressor, maybe open a PR on their side. I'll leave that as a good first issue so that someone can check why we are still initializing the weights in int8. 

In `src/transformers/modeling_utils.py`, the conditional at https://github.com/huggingface/transformers/blob/53c9dcd6fd31cb9e8a10248693a905d0223b8316/src/transformers/modeling_utils.py#L5687 checks if zero3 is enabled and the model is quantized.

In the if branch, we use DeepSpeed to initialize the parameters of `not_initialized_submodules`, while in the else branch we call the `initialize_weights` method.

However, I think the `not is_quantized` condition should also be added to the else branch, because we should avoid calling the initialize_weights method when dealing with dtypes other than float or double, since we don't have initialization functions for other dtypes.

@SunMarc does that make sense to you? This would impact models in general, since this flow is in the `_load_pretrained_model`method of `PreTrainedModel`

I want to Contribute


I want to look at it 

Can I look into this ?


Hi @AdelineXinyi , 
I’d like to work on this issue as part of Hacktoberfest.

I’ve gone through the discussion and the error trace — it seems that during model initialization, normal_() is being called on quantized tensors (int8 dtype), which naturally fails since it expects a floating type.

Proposed approach:

Detect if a module’s weights are of an integer or quantized dtype during _init_weights.

Skip or safely bypass the floating-point initialization in that case (for example, only call normal_() if tensor.dtype.is_floating_point is True).

Optionally log a small info message indicating that quantized weights were detected and skipped for initialization.

Add a lightweight test (mock or quantized example) to ensure no crash occurs when loading int8 models.

Before I start implementing this, can you please confirm if this direction aligns with expectations?
If yes, I’ll go ahead and open a PR implementing this change.

Thanks!

Hi, I would like to give it a try, can you assign it to me?

Hi! I'd like to take this up.  
I've reviewed the code and I'm confident I can implement the fix.  
Please assign it to me if it's still unclaimed.

A lot of refactor happened since the issue was posted. I don't think this is relevant anymore, so i'm closing this ! Feel free to reopen with a reproducer if you are still facing the issue with transformers v5