0xSero

/

GLM-5.2-504B-Nvidia

like 19

Summary

config.json contains both n_routed_experts: 168 (correct — matches the checkpoint's tensors) and a stale
num_experts: 256 carried over from the unpruned base. This PR removes the stale key.

Why it breaks every vLLM user

transformers' GlmMoeDsaConfig.__post_init__ does an unconditional backward-compat re-route
(configuration_glm_moe_dsa.py:161-163):

# BC: re-route `num_experts` to `n_routed_experts`
if (num_experts := kwargs.get("num_experts")) is not None:
    self.n_routed_experts = num_experts

So the deprecated key silently overwrites the explicit one:

AutoConfig.from_pretrained(M)                       -> n_routed_experts=256   # config.json says 168!
AutoConfig.from_pretrained(M, num_experts=168)      -> n_routed_experts=256   # kwarg lands after __post_init__
AutoConfig.from_pretrained(M, n_routed_experts=168) -> n_routed_experts=168
# with num_experts removed:                         -> n_routed_experts=168   # correct, no override needed

vLLM then builds FusedMoE for 256 experts that don't exist -> 64 experts/rank instead of 42 -> ~111 GB/GPU
instead of ~80 -> torch.OutOfMemoryError in modelopt.py:1458 create_weights. The log shows
Local/global number of experts: 64/256 (should be 42/168).

Verified against this repo's own model.safetensors.index.json: it contains exactly 168 contiguous routed
experts (indices 0..167). So n_routed_experts: 168 is right and num_experts: 256 contradicts both the sibling
key and the tensors.

Why --hf-overrides isn't a sufficient workaround

--hf-overrides '{"n_routed_experts":168}' fixes the main model, but vLLM deliberately discards dict-form
overrides for the draft model (vllm/config/speculative.py:602):

"Dict overrides are target-specific key patches and are not applied to the draft."

Only callable overrides propagate, and the CLI can only produce a dict. So with the workaround, MTP fails:

self.drafter.load_model(self.model)
AssertionError: Attempted to load weight (torch.Size([168])) into parameter (torch.Size([256]))

Deleting the key here fixes both paths at the source.

Verified

Removing the key locally, the model loads at 79.1 GB/GPU and serves correctly (Local/global number of experts: 42/168). Tested on 4x RTX PRO 6000 Blackwell (sm_120), vLLM 0.25.1. Needle-in-haystack passes to 233,809 tokens
at 250K context.

Thanks for the checkpoint — it's the only GLM-5.2 NVFP4 cut that fits 4x96 GB, and it works well once this is out
of the way. Full write-up: https://github.com/hermia-ai/glm-5.2-sm120-stock-vllm

(Note: I did not touch num_experts_per_tok — only the top-level num_experts.)