Qwen3Moe failed with FineGrainedFP8Config
Author: jessiewiswjc
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.57.3
- Platform: Linux-5.15.0-136-generic-x86_64-with-glibc2.35
- Python version: 3.12.10
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.5.3
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1+cu126 (CUDA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H20

### Who can help?

@ArthurZucker @Cyrilvallez

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I tried quantizing the https://huggingface.co/Qwen/Qwen3-30B-A3B to FP8 with the following code:
```
from transformers import FineGrainedFP8Config, AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-30B-A3B/"
quantization_config = FineGrainedFP8Config(modules_to_not_convert=["model.layers.*.mlp.gate"])
quantized_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto", quantization_config=quantization_config)

tokenizer = AutoTokenizer.from_pretrained(model_name)
input_text = "What are we having for dinner?"
input_ids = tokenizer(input_text, return_tensors="pt").to(quantized_model.device.type)

output = quantized_model.generate(**input_ids, max_new_tokens=10)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```
but it threw the following error:
```
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards:   0%|                                                   | 0/16 [00:01<?, ?it/s]
Traceback (most recent call last):
  File "/home/wangjiancheng/test.py", line 5, in <module>
    quantized_model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", device_map="auto", quantization_config=quantization_config)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 604, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/modeling_utils.py", line 277, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5048, in from_pretrained
    ) = cls._load_pretrained_model(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/modeling_utils.py", line 5468, in _load_pretrained_model
    _error_msgs, disk_offload_index = load_shard_file(args)
                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/modeling_utils.py", line 843, in load_shard_file
    disk_offload_index = _load_state_dict_into_meta_model(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/modeling_utils.py", line 774, in _load_state_dict_into_meta_model
    hf_quantizer.create_quantized_param(model, param, param_name, param_device)
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/quantizers/quantizer_finegrained_fp8.py", line 143, in create_quantized_param
    _load_parameter_into_model(model, param_name.rsplit(".", 1)[0] + ".weight_scale_inv", scale)
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/transformers/modeling_utils.py", line 667, in _load_parameter_into_model
    module.load_state_dict({param_type: tensor}, strict=False, assign=True)
  File "/home/wangjiancheng/miniforge3/envs/vllm/lib/python3.12/site-packages/torch/nn/modules/module.py", line 2593, in load_state_dict
    raise RuntimeError(
RuntimeError: Error(s) in loading state_dict for FP8Linear:
        size mismatch for weight_scale_inv: copying a param with shape torch.Size([16]) from checkpoint, the shape in current model is torch.Size([1, 16]).
```

However, the [official FineGrainedFP8Config code](https://huggingface.co/docs/transformers/v4.57.3/en/quantization/finegrained_fp8) can run successfully in my environment. The only difference between the official code and my code is that the model was changed from LLaMA3 to Qwen3Moe.


### Expected behavior

My code can run successfully.

Comments:
cc @mekkcyber

Hi @jessiewiswjc, could you try providing a `weight_block_size` in the quantization config ? something like `weight_block_size = (128, 128)`

Hi @jessiewiswjc ,
Maybe the error is due to, I guess : 
1. Architecture incompatibility because Qwen uses MoE architecture and softmax routing which creates variable [N] tensors causing shape mismatch.
2. Or due to this part - (modules_to_not_convert=["model.layers.*.mlp.gate"]) in your code which causes the shape mismatch.
3. You need a GPU with Compute Capability>=9 (H100), and install a PyTorch version compatible with the CUDA version of your GPU. It is from official page of [official FineGrainedFP8Config code](https://huggingface.co/docs/transformers/v4.57.3/en/quantization/finegrained_fp8)

Also, in your code , torch_dtype="auto" has been deprecated from torch_dtype to dtype.

Maybe these points, will help you to resolve the error.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.