[deepseek_v4] conversion_mapping doesn't cover mtp.* paths — MTP keys silently random-init even after _keys_to_ignore is empty
Author: pasta-paul
Pull request: no

## Summary

`transformers.conversion_mapping.get_checkpoint_conversion_mapping("deepseek_v4")` returns 41 `WeightRenaming` entries that rename upstream-internal naming to HF naming (`attn.` → `self_attn.`, `ffn.` → `mlp.`, `attn_norm.` → `input_layernorm.`, `attn.wq_a.` → `self_attn.q_a_proj.`, `attn.attn_sink` → `self_attn.sinks`, etc.).

**Entries 6–38 are anchored at `^layers\.(\d+)\.`** — they only fire on main-layer keys. None cover `mtp.\d+.*` paths.

Combined with the existing `_keys_to_ignore_on_load_unexpected = [r"(^|\.)mtp\..*"]` regex on `DeepseekV4PreTrainedModel` (filed separately as huggingface/transformers#46127), `mtp.*` keys never reach the model at all. **Even after that regex is dropped** (as #46127 does), the MTP keys arrive in upstream form (`mtp.0.attn.wq_a.weight`) — but the MTP submodules expect HF naming (`mtp.0.self_attn.q_a_proj.weight`). The keys are then flagged "unexpected", the submodules remain "uninitialized", and `_initialize_weights` falls through to `_init_weights` → `init.normal_` random-initializes the MTP block.

## Symptom

The model loads "successfully" (no errors, no warnings about missing keys after the regex is dropped), `model.mtp[0]` exists with the right structure, `from_pretrained` returns. But `model.mtp[0].self_attn.q_a_proj.weight` is random Gaussian, not the value in the safetensors file. **Silent corruption of the MTP draft head.** Any downstream calibration / quantization / inference using `model.mtp` produces garbage.

## Repro

```python
# (assumes huggingface/transformers#46127 is applied — DeepseekV4NextNPredictor
#  exists, _keys_to_ignore_on_load_unexpected = [])
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("<DSv4-Flash BF16 with mtp.* keys>")

# Compare loaded vs source
import safetensors.torch as st
from pathlib import Path
loaded_w = model.model.mtp[0].self_attn.q_a_proj.weight
for shard in sorted(Path("<path>").glob("model-*.safetensors")):
    with st.safe_open(shard, framework="pt") as f:
        if "mtp.0.attn.wq_a.weight" in f.keys():
            source_w = f.get_tensor("mtp.0.attn.wq_a.weight")
            break

diff = (loaded_w.cpu().float() - source_w.cpu().float()).abs().max().item()
print(f"max_diff = {diff}")
# Without conversion mapping for mtp.*: diff ≈ random Gaussian range (e.g. 0.1+)
# With the mtp.* mapping extension: diff ≈ 0
```

## Proposed fix

Add 33 `mtp.\d+.*` equivalents mirroring the existing `^layers\.(\d+)\.` entries to `_checkpoint_conversion_mapping` for the `deepseek_v4` architecture. The 6 model-level entries (`embed.`, `head.`, `norm.`, `hc_head_*`) do NOT need to be mirrored — MTP doesn't have its own copy of those (it shares `embed_tokens` and `lm_head` with the main model).

Specifically, for each of these patterns, add a parallel entry anchored at `^mtp\.(\d+)\.`:

```
^layers\.(\d+)\.attn_norm\.                      → layers.\1.input_layernorm.
^layers\.(\d+)\.ffn_norm\.                       → layers.\1.post_attention_layernorm.
^layers\.(\d+)\.hc_attn_fn$                       → layers.\1.attn_hc.fn
^layers\.(\d+)\.hc_attn_base$                     → layers.\1.attn_hc.base
^layers\.(\d+)\.hc_attn_scale$                    → layers.\1.attn_hc.scale
^layers\.(\d+)\.hc_ffn_fn$                        → layers.\1.ffn_hc.fn
^layers\.(\d+)\.hc_ffn_base$                      → layers.\1.ffn_hc.base
^layers\.(\d+)\.hc_ffn_scale$                     → layers.\1.ffn_hc.scale
^layers\.(\d+)\.attn\.                            → layers.\1.self_attn.
^layers\.(\d+)\.ffn\.                             → layers.\1.mlp.
^layers\.(\d+)\.self_attn\.attn_sink$             → layers.\1.self_attn.sinks
^layers\.(\d+)\.self_attn\.(.*?)\.wq_a\.          → layers.\1.self_attn.\2.q_a_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wq_b\.          → layers.\1.self_attn.\2.q_b_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wkv\.           → layers.\1.self_attn.\2.kv_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wgate\.         → layers.\1.self_attn.\2.gate_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wo_a\.          → layers.\1.self_attn.\2.o_a_proj.
^layers\.(\d+)\.self_attn\.(.*?)\.wo_b\.          → layers.\1.self_attn.\2.o_b_proj.
^layers\.(\d+)\.self_attn\.wq_a\.                 → layers.\1.self_attn.q_a_proj.
^layers\.(\d+)\.self_attn\.wq_b\.                 → layers.\1.self_attn.q_b_proj.
^layers\.(\d+)\.self_attn\.wkv\.                  → layers.\1.self_attn.kv_proj.
^layers\.(\d+)\.self_attn\.wo_a\.                 → layers.\1.self_attn.o_a_proj.
^layers\.(\d+)\.self_attn\.wo_b\.                 → layers.\1.self_attn.o_b_proj.
^layers\.(\d+)\.self_attn\.q_norm\.               → layers.\1.self_attn.q_a_norm.
^layers\.(\d+)\.mlp\.gate\.bias$                  → layers.\1.mlp.gate.e_score_correction_bias
^layers\.(\d+)\.mlp\.shared_experts\.w1\.         → layers.\1.mlp.shared_experts.gate_proj.
^layers\.(\d+)\.mlp\.shared_experts\.w2\.         → layers.\1.mlp.shared_experts.down_proj.
^layers\.(\d+)\.mlp\.shared_experts\.w3\.         → layers.\1.mlp.shared_experts.up_proj.
```

The entries at indexes 17–22 (compressor/indexer renames) only need to mirror if MTP can be configured with `compressed_sparse_attention` or `heavily_compressed_attention` layer_type. For DSv4-Flash, MTP uses `sliding_attention` (compressor = None — see #46127 discussion), so those 6 entries don't *need* to mirror, but mirroring them is harmless (the regex just won't match anything).

## Runtime workaround for downstream users

Until upstream lands, here's the runtime mirror:

```python
from transformers.conversion_mapping import (
    get_checkpoint_conversion_mapping,
    register_checkpoint_conversion_mapping,
)
existing = get_checkpoint_conversion_mapping("deepseek_v4")
added = []
for entry in existing:
    sp = getattr(entry, "source_patterns", None)
    tp = getattr(entry, "target_patterns", None)
    if sp is None or tp is None:
        continue
    sp_list = sp if isinstance(sp, (list, tuple)) else [sp]
    tp_list = tp if isinstance(tp, (list, tuple)) else [tp]
    new_sp, new_tp = [], []
    for s, t in zip(sp_list, tp_list):
        if isinstance(s, str) and s.startswith(r"^layers\.(\d+)\."):
            new_sp.append(s.replace(r"^layers\.(\d+)\.", r"^mtp\.(\d+)\.", 1))
            new_tp.append(t.replace("layers.\\1.", "mtp.\\1.", 1))
    if new_sp:
        added.append(type(entry)(
            source_patterns=new_sp if len(new_sp) > 1 else new_sp[0],
            target_patterns=new_tp if len(new_tp) > 1 else new_tp[0],
        ))
register_checkpoint_conversion_mapping(
    "deepseek_v4", list(existing) + added, overwrite=True)
```

## Detection — value-verification assertion

A 50-line fixture that catches this regression class (and the related layer_type bug at #46127) by comparing a loaded MTP tensor to its source:

```python
import safetensors.torch as st
from pathlib import Path

loaded_w = model.model.mtp[0].self_attn.q_a_proj.weight
source_w = None
for shard in sorted(Path(model_path).glob("model-*.safetensors")):
    with st.safe_open(shard, framework="pt") as f:
        if "mtp.0.attn.wq_a.weight" in f.keys():
            source_w = f.get_tensor("mtp.0.attn.wq_a.weight")
            break
assert source_w is not None
diff = (loaded_w.cpu().float() - source_w.cpu().float()).abs().max().item()
assert diff < 1e-4, f"MTP weight mismatch: {diff} (silent random-init?)"
```

This belongs as a test under `tests/models/deepseek_v4/` paired with #46127.

## Related

- #46127 — adds `DeepseekV4NextNPredictor` class + `Model.mtp` ModuleList + `sliding_attention` layer_type for MTP. The class shim PR. This issue is the *companion* — even with the class shim, the conversion mapping needs to be extended for MTP keys to actually load into the new submodules.
- vllm-project/llm-compressor#2735 — calibration-side rollup of both issues.
- vllm-project/llm-compressor#2739 — companion mapping extension PR (for the `ARCH_TO_2D_MAPPINGS` that lives on llm-compressor's side).


Comments:
cc @arthurzucker for DeepSeek V4

## Production validation — workaround validated end-to-end

Update with new evidence: `install_mtp_conversion_mapping_extension()` (a runtime patch that clones the 41 main-layer entries in `ARCH_TO_2D_MAPPINGS["deepseek_v4"]` and registers them under `mtp.\d+.*` equivalents) is now validated in a shipped artifact.

### Setup
- Repo: https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp
- Artifact: https://huggingface.co/canada-quant/DeepSeek-V4-Flash-W4A16-FP8-MTP (public)
- Workaround: [`scripts/transformers_mtp_shim.py::install_mtp_conversion_mapping_extension`](https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp/blob/main/scripts/transformers_mtp_shim.py)

### Why it matters
Without the extension, even when `_keys_to_ignore_on_load_unexpected` is patched empty AND `DeepseekV4NextNPredictor` class exists (PR #46127), the MTP `mtp.0.*` keys arrive in upstream form (`mtp.0.attn.wq_a` etc.) and don't match the HF-named submodules → submodules get random-init from `_init_weights` → silent. We catch this empirically:
- Smoke calibration WITHOUT extension: artifact has 0 working MTP keys at vLLM serve time → 0% MTP acceptance (silent)
- Smoke calibration WITH extension: 797 MTP keys load → 67.9%+ acceptance (smoke) / 69.94% (Phase 2 over 200 prompts)

The conversion mapping for DSv4 has 33 main-layer entries; the equivalent count for mtp.* is the same minus a few (no compressor/indexer in MTP) — should be a small static-list addition. Happy to draft a PR if helpful.

Raw eval data: https://github.com/canada-quant/dsv4-flash-w4a16-fp8-mtp/tree/main/benchmarks/phase2

 cc @Cyrilvallez MTP coming soon? 👁️ 👁️ 

> **Disclosure:** this comment was generated with AI assistance.

**Tracking note — MTP-class cluster.** This filing is one of four artifacts for what is fundamentally a single bug — "DSv4 MTP weights have no home in the canonical loader path":

| Filing | Layer | Role |
|---|---|---|
| huggingface/transformers#46127 (PR) | transformers | Add `DeepseekV4NextNPredictor` class so the MTP block has a module to load into |
| huggingface/transformers#46129 (issue) | transformers | `conversion_mapping` doesn't cover `mtp.*` paths — companion to #46127 |
| vllm-project/llm-compressor#2735 (issue) | llm-compressor | Canonical DSv4 calibration example drops MTP; regex `^layers\.` anchored excludes `mtp.*` |
| vllm-project/llm-compressor#2739 (PR) | llm-compressor | Extend `ARCH_TO_2D_MAPPINGS` for the MTP block — fixes #2735 |

Posting this header on all four so maintainers see one effort rather than four independent threads. Happy to consolidate further if reviewers prefer a single tracking issue.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.