Fix Qwen3 MoE GGUF architecture mismatch
Author: ctcanbol
Pull request: yes

# What does this PR do?

Currently, GGUF versions of Qwen3 MoE models raises "_ValueError: The checkpoint you are trying to load has model type qwen3moe but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date_" error. This PR resolves this issue.

Fixes https://github.com/huggingface/transformers/pull/39638

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @SunMarc @MekkCyber


Comments:
Report below when max_model_len is set. It seems that max_position_embeddings is wrong.
```
User-specified max_model_len (131072) is greater than the derived max_model_len (max_position_embeddings=32768 or model_max_length=None in model's config.json).
```

After temporarily change the max_model_len to 32768, it reports *KeyError: 'embed_tokens.qweight_type'* while load_weights.
The details is here:
```
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239] Exception in worker VllmWorkerProcess while processing method load_model.                                                                                                                
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239] Traceback (most recent call last):                                                                                                                                                       
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/executor/multiproc_worker_utils.py", line 233, in _run_worker_process                              
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     output = run_method(worker, method, args, kwargs)                                                                                                                                    
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                    
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/utils/__init__.py", line 2985, in run_method                                                       
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     return func(*args, **kwargs)                                                                                                                                                         
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]            ^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                         
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/worker/worker.py", line 211, in load_model                                                         
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     self.model_runner.load_model()                                                                                                                                                       
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/worker/model_runner.py", line 1086, in load_model                                                  
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     self.model = get_model(vllm_config=self.vllm_config)                                                                                                                                 
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                 
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/model_loader/__init__.py", line 59, in get_model                                    
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     return loader.load_model(vllm_config=vllm_config,                                                                                                                                    
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                    
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/model_loader/gguf_loader.py", line 130, in load_model                               
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     self.load_weights(model, model_config)                                                                                                                                               
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/model_loader/gguf_loader.py", line 113, in load_weights                             
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     model.load_weights(                                                                                                                                                                  
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_moe.py", line 543, in load_weights                                     
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     return loader.load_weights(weights)                                                                                                                                                  
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                  
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 291, in load_weights                                         
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     autoloaded_weights = set(self._load_module("", self.module, weights))                                                                                                                
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 249, in _load_module
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     yield from self._load_module(prefix,
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/models/utils.py", line 222, in _load_module
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     loaded_params = module_load_weights(weights)
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]   File "/lib/python3.11/site-packages/vllm/model_executor/models/qwen3_moe.py", line 477, in load_weights
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]     param = params_dict[name]
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239]             ~~~~~~~~~~~^^^^^^
(VllmWorkerProcess pid=3671127) ERROR 08-07 16:49:54 [multiproc_worker_utils.py:239] KeyError: 'embed_tokens.qweight_type'
```

I do not know if it is a problem of transformers or vllm.


Maybe a vllm issue... Need to test with llamacpp to be sure.

> Maybe a vllm issue... Need to test with llamacpp to be sure.

emmm, why it should be tested with llamacpp? I think it does not use transformers though.

@Readon this doesn't seem related to transformers, it looks you are using vllm for inference 

from trying your PR, it sould also be added to:
`src/transformers/models/auto/configuration_auto.py`
because I get:
```sh
File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 1273, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `qwen3moe` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
```

> from trying your PR, it sould also be added to: `src/transformers/models/auto/configuration_auto.py` because I get:
> 
> ```shell
> File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py", line 1273, in from_pretrained
>     raise ValueError(
> ValueError: The checkpoint you are trying to load has model type `qwen3moe` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
> ```

I think it will not end with this. Maybe we need to change all qwen3_moe definitions in transformers with qwen3moe and align the two worlds (or add also qwen3moe)...

I am testing the bug fix with the latest commit in this branch and did not encounter the error reported by @space192. I recommend @space192 to double-check whether they are using the library version from this PR.

---

However, in my case, I encountered the following error after trying out the latest commit in this branch to load a bf16 GGUF file of the Qwen3-30B-A3B model:

```
  File "/data_fast/home/jusjinuk/codes/transformers/src/transformers/modeling_gguf_pytorch_utils.py", line 121, in _split_moe_expert_tensor
    name = tensor_key_mapping[name]
KeyError: 'blk.24.ffn_down_exps.weight’
```

Upon investigation, I found that the issue arises because the code is not reading the config in [this code](https://github.com/ctcanbol/transformers/blob/344eaa3b90850bee89c2fb13bb0b16248bd0ce33/src/transformers/modeling_gguf_pytorch_utils.py#L436-L446). Thus, it uses the default `Qwen3MoeConfig` to instantiate the Qwen3Moe model in Transformers, which sets num_hidden_layers to 24 [by default](https://github.com/ctcanbol/transformers/blob/344eaa3b90850bee89c2fb13bb0b16248bd0ce33/src/transformers/models/qwen3_moe/configuration_qwen3_moe.py#L175). This does not match the actual number of layers in the GGUF files.

The reason for not loading the correct config is because the code reads `GGUF_CONFIG_MAPPING` with the architecture name updated [here](https://github.com/ctcanbol/transformers/blob/f43d297e569998121f5e3fad9b0ce271298574c1/src/transformers/modeling_gguf_pytorch_utils.py#L423), but `GGUF_CONFIG_MAPPING` has its keys with the [old architecture name](https://github.com/ctcanbol/transformers/blob/main/src/transformers/integrations/ggml.py#L105) (`qwen3moe`), thus is not correctly reading the necessary config.

To address this, I updated L105 to reflect the updated architecture name in the `GGUF_CONFIG_MAPPING` as shown in [this commit](https://github.com/jusjinuk/transformers/commit/f43d297e569998121f5e3fad9b0ce271298574c1). I believe qwen2moe should be updated similarly, but I left it unchanged since I have not personally tested the Qwen2Moe architecture. I also noticed a missing parameter (`head_dim`) and added it at L112 in the [same commit](https://github.com/jusjinuk/transformers/commit/f43d297e569998121f5e3fad9b0ce271298574c1).

---

After these changes, I confirmed that the output is generated without any issues, tested with the following code:
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
import torch

gguf_model_path = "gguf_files/Qwen_Qwen3-30B-A3B-Instruct-2507-bf16"
gguf_file_name = "qwen3-30b-a3b-instruct-2507-bf16.gguf"
tokenizer_name = "Qwen/Qwen3-30B-A3B-Instruct-2507"

model = AutoModelForCausalLM.from_pretrained(
    gguf_model_path, 
    gguf_file=gguf_file_name, 
    torch_dtype=torch.bfloat16, 
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)
streamer = TextStreamer(tokenizer)

prompt = "Explain me what Large Language Models are."
chat = [
    {"role": "user", "content": prompt}
]
tokenized_chat = tokenizer.apply_chat_template(
    chat, tokenize=True, add_generation_prompt=True, return_tensors="pt")
tokenized_chat = tokenized_chat.to("cuda")
outputs = model.generate(tokenized_chat, max_new_tokens=100, streamer=streamer, eos_token_id=tokenizer.eos_token_id)
```
I have submitted a PR to this branch [here](https://github.com/ctcanbol/transformers/pull/1), as I don't have the direct write access to this branch. I hope @ctcanbol can merge these changes into the main branch of the repository, and then finally merge the whole changes into the mainline transformers library.

cc. @ArthurZucker @SunMarc @MekkCyber

Thank you.

After @jusjinuk 's addition, I think this PR is ready to merge.

@SunMarc @MekkCyber Could you help merge these changes by approving the workflow?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39976). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.