Repository: huggingface/transformers
Pull request: Add GGUF support for MiniMax-M2.1 model

# What does this PR do?

Add GGUF loading support for MiniMax-M2.1 (456B MoE) model.

MiniMax-M2.1 is a large Mixture-of-Experts model with 456B total parameters (45.9B active), 256 experts and 8 experts per token. This PR enables loading its GGUF-quantized checkpoints (e.g. [unsloth/MiniMax-M2.1-GGUF](https://huggingface.co/unsloth/MiniMax-M2.1-GGUF)) via `from_pretrained(..., gguf_file=...)`.

### Changes

**`src/transformers/integrations/ggml.py`**
- Add `"minimax_m2"` entry to `GGUF_CONFIG_MAPPING` with model-specific config fields (including MoE fields: `expert_count`, `expert_used_count`, `expert_feed_forward_length`, `expert_gating_func`).
- Convert `expert_gating_func` integer (from GGUF metadata) to `scoring_func` string (`{0: "none", 1: "softmax", 2: "sigmoid"}`).
- Add `"minimax_m2"` entry to `GGUF_CONFIG_DEFAULTS_MAPPING` with `"use_routing_bias": True`. This is needed because GGUF metadata does not include a field for routing bias, but the model weights contain `e_score_correction_bias` tensors that require `use_routing_bias=True` to be loaded correctly.
- Register `GGUFQwen2Converter` for `minimax_m2` in `GGUF_TO_FAST_CONVERTERS` (tokenizer is compatible with Qwen2).

**`src/transformers/modeling_gguf_pytorch_utils.py`**
- Add `MiniMaxM2TensorProcessor` class following the `TensorProcessor` API introduced in #42854 (same pattern as `Qwen2MoeTensorProcessor`):
  - `preprocess_name()`: strips per-expert indices from HF weight names so that multiple experts can map to one fused GGUF tensor.
  - `perform_fallback_tensor_mapping()`: maps merged `gate_up_proj` to both `ffn_gate_exps` and `ffn_up_exps` GGUF tensors; maps `e_score_correction_bias` to `exp_probs_b.bias`.
  - `process()`: matches GGUF MoE expert tensors and merges gate+up into `gate_up_proj [num_experts, 2*intermediate, hidden]`.
  - `_set_moe_expert_tensor()`: merges gate and up weights into the fused `gate_up_proj` tensor, passes down weights directly.
- Register processor in `TENSOR_PROCESSORS`, add model type and architecture mappings.

### Testing

Due to the model size (456B parameters, 227GB for Q8_0 GGUF), no CI-compatible unit tests are included. This is consistent with other large MoE models (e.g., Qwen3-30B-A3B in #42854).

Verified end-to-end via vLLM serving the Q8_0 GGUF checkpoint on two GPU platforms:

**8×AMD W7900D (48GB each)**

| Benchmark | GGUF Q8_0 | Official BF16 |
|-----------|-----------|---------------|
| GSM8K 8-shot | **91.5%** | 92.0% |
| MMLU 5-shot | **85.66%** | 86.2% |

**AMD MI350X (288GB each)**

| Benchmark | TP | GGUF Q8_0 | Official BF16 |
|-----------|----|-----------|---------------|
| GSM8K 8-shot | 8 | **91.7%** | 92.0% |
| GSM8K 8-shot | 4 | **92.2%** | 92.0% |
| MMLU 5-shot | 4 | **86.01%** | 86.2% |

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

@SunMarc @MekkCyber @ArthurZucker


Changed files:
filename: src/transformers/integrations/ggml.py
patch: @@ -287,6 +287,24 @@
         "attention.layer_norm_rms_epsilon": "rms_norm_eps",
         "vocab_size": "vocab_size",
     },
+    "minimax_m2": {
+        "context_length": "max_position_embeddings",
+        "block_count": "num_hidden_layers",
+        "feed_forward_length": "intermediate_size",
+        "embedding_length": "hidden_size",
+        "rope.dimension_count": "rotary_dim",
+        "rope.freq_base": "rope_theta",
+        "attention.head_count": "num_attention_heads",
+        "attention.head_count_kv": "num_key_value_heads",
+        "attention.key_length": "head_dim",
+        "attention.value_length": None,
+        "attention.layer_norm_rms_epsilon": "rms_norm_eps",
+        "expert_count": "num_local_experts",
+        "expert_used_count": "num_experts_per_tok",
+        "expert_feed_forward_length": None,
+        "vocab_size": "vocab_size",
+        "expert_gating_func": "scoring_func",
+    },
 }
 
 GGUF_TOKENIZER_MAPPING = {
@@ -320,6 +338,12 @@
         #      (the parameter right after LLM_FFN_SILU corresponds to norm_topk_prob)
         "norm_topk_prob": True,
     },
+    "minimax_m2": {
+        # MiniMax-M2 uses routing bias (e_score_correction_bias) for MoE expert selection,
+        # but this is not stored in GGUF metadata. Set it as default so the model weights
+        # (which include e_score_correction_bias tensors) are loaded correctly.
+        "use_routing_bias": True,
+    },
 }
 
 
@@ -766,6 +790,7 @@ def converted(self) -> Tokenizer:
     "umt5": GGUFT5Converter,
     "deci": GGUFLlamaConverter,
     "decilm": GGUFLlamaConverter,
+    "minimax_m2": GGUFQwen2Converter,
 }
 
 

filename: src/transformers/modeling_gguf_pytorch_utils.py
patch: @@ -299,6 +299,59 @@ def process(self, weights, name, **kwargs):
         return GGUFTensor(weights, name, {})
 
 
+class MiniMaxM2TensorProcessor(TensorProcessor):
+    HF_EXPERT_RENAME_PATTERN = re.compile(r"mlp\.experts\.\d+\.")
+    HF_MOE_W13_PATTERN = re.compile(r"(?:model\.)?layers\.(?P<bid>\d+)\.mlp\.experts\.gate_up_proj")
+    GGUF_MOE_WEIGHTS_PATTERN = re.compile(r"(?P<name>.*\.ffn_(?P<w>gate|down|up)_exps)\.weight$")
+    HF_BIAS_PATTERN = re.compile(r"(?:model\.)?layers\.(?P<bid>\d+)\.mlp\.e_score_correction_bias")
+
+    def __init__(self, config=None):
+        super().__init__(config=config)
+
+    def preprocess_name(self, hf_name: str) -> str:
+        return re.sub(self.HF_EXPERT_RENAME_PATTERN, "mlp.experts.", hf_name)
+
+    def perform_fallback_tensor_mapping(
+        self, gguf_to_hf_name_map: dict[str, str], suffix: str, qual_name: str, hf_name: str
+    ):
+        # Map merged gate_up_proj to both ffn_gate_exps and ffn_up_exps GGUF tensors.
+        if m := re.fullmatch(self.HF_MOE_W13_PATTERN, hf_name):
+            full_hf_name = qual_name + hf_name
+            gguf_to_hf_name_map[f"blk.{m['bid']}.ffn_gate_exps{suffix}"] = full_hf_name
+            gguf_to_hf_name_map[f"blk.{m['bid']}.ffn_up_exps{suffix}"] = full_hf_name
+        # Map e_score_correction_bias to GGUF exp_probs_b.bias.
+        elif m := re.fullmatch(self.HF_BIAS_PATTERN, hf_name):
+            gguf_to_hf_name_map[f"blk.{m['bid']}.exp_probs_b.bias"] = qual_name + hf_name
+
+    def process(self, weights, name: str, **kwargs):
+        if m := re.fullmatch(self.GGUF_MOE_WEIGHTS_PATTERN, name):
+            tensor_key_mapping = kwargs.get("tensor_key_mapping")
+            parsed_parameters = kwargs.get("parsed_parameters")
+            if tensor_key_mapping:
+                self._set_moe_expert_tensor(weights, parsed_parameters, tensor_key_mapping[m["name"]], m["w"])
+            return GGUFTensor(weights, None, {})
+        return GGUFTensor(weights, name, {})
+
+    def _set_moe_expert_tensor(self, weights: np.ndarray, parsed_parameters: dict[str, dict], hf_name: str, w: str):
+        torch_weights = torch.from_numpy(np.copy(weights))
+        if w == "down":
+            parsed_parameters["tensors"][hf_name] = torch_weights
+        else:
+            # Merge gate and up into gate_up_proj [num_experts, 2*intermediate, hidden]
+            shape = list(weights.shape)
+            shard_dim = 1
+            shard_size = shape[shard_dim]
+            shape[shard_dim] = shard_size * 2
+            if hf_name not in parsed_parameters["tensors"]:
+                parsed_parameters["tensors"][hf_name] = torch.zeros(shape, dtype=torch_weights.dtype)
+            out: torch.Tensor = parsed_parameters["tensors"][hf_name]
+            if w == "gate":
+                out = out.narrow(shard_dim, 0, shard_size)
+            else:  # w == "up"
+                out = out.narrow(shard_dim, shard_size, shard_size)
+            out.copy_(torch_weights)
+
+
 TENSOR_PROCESSORS = {
     "llama": LlamaTensorProcessor,
     "qwen2moe": Qwen2MoeTensorProcessor,
@@ -312,6 +365,7 @@ def process(self, weights, name, **kwargs):
     "gemma2": Gemma2TensorProcessor,
     "gemma3": Gemma2TensorProcessor,
     "lfm2": Lfm2TensorProcessor,
+    "minimax-m2": MiniMaxM2TensorProcessor,
 }
 
 
@@ -360,6 +414,8 @@ def get_gguf_hf_weights_map(
         model_type = "gemma3"
     elif model_type == "umt5":
         model_type = "t5"
+    elif model_type == "minimax_m2":
+        model_type = "minimax-m2"
     arch = None
     for key, value in MODEL_ARCH_NAMES.items():
         if value == model_type:
@@ -462,6 +518,8 @@ def load_gguf_checkpoint(gguf_checkpoint_path, return_tensors=False, model_to_lo
         updated_architecture = "qwen2_moe"
     elif "qwen3moe" in architecture:
         updated_architecture = "qwen3_moe"
+    elif "minimax-m2" in architecture:
+        updated_architecture = "minimax_m2"
 
     # For stablelm architecture, we need to set qkv_bias and use_parallel_residual from tensors
     # If `qkv_bias=True`, qkv_proj with bias will be present in the tensors
@@ -525,6 +583,13 @@ def load_gguf_checkpoint(gguf_checkpoint_path, return_tensors=False, model_to_lo
     if parsed_parameters["config"]["model_type"] == "gemma3":
         parsed_parameters["config"]["model_type"] = "gemma3_text"
 
+    # MiniMax-M2: convert expert_gating_func integer to scoring_func string
+    if parsed_parameters["config"].get("model_type") == "minimax_m2":
+        _gating_func_map = {0: "none", 1: "softmax", 2: "sigmoid"}
+        _scoring = parsed_parameters["config"].get("scoring_func")
+        if isinstance(_scoring, int):
+            parsed_parameters["config"]["scoring_func"] = _gating_func_map.get(_scoring, "softmax")
+
     if parsed_parameters["config"]["model_type"] == "lfm2":
         gguf_num_key_value_heads = parsed_parameters["config"]["num_key_value_heads"]
         # LFM2 GGUF checkpoint defines num_key_value_heads as a list of integers .e.g [0, 0, 8, 0, 0, 8, 0, 0, 8, 0, 8, 0, 8, 0, 8, 0] but we need to set it to the max value for HF