aidendle94/GLM-5.2-MXFP4-Experts-GPTQ

GLM-5.2 Hybrid v3 — GPTQ-calibrated MXFP4 experts + FP8 attention

A complete, standalone mixed-precision GLM-5.2 checkpoint — the successor to GLM-5.2-Hybrid-FP8-MXFP4 with every routed expert re-quantized by GPTQ against its own routed activations instead of round-to-nearest. Same size, same speed, same kernels as v2 — better numbers everywhere:

Metric (vs BF16 reference)	v2 (RTN experts)	v3 (this repo)
Mean teacher-forced KLD	0.0976	0.0803 (−17.7%)
Median per-token KLD	0.00123	0.00083 (−33%)
Top-1 agreement	93.3%	93.9%
Reasoning-length inflation	+63%	+56%
Decode (4× DGX Spark, TP4+DCP4, MTP)	~24 tok/s	~24 tok/s
Checkpoint size	382 GiB	382 GiB

Component	Precision	Source
Attention (MLA projections, indexer)	block-FP8 (128×128)	RedHatAI
Dense MLPs (layers 0–2)	NVFP4 (group-16)	RedHatAI
Shared experts (layers 3–77)	block-FP8	quantized from zai-org BF16
Routed experts, layers 3–77	MXFP4, GPTQ-calibrated	this work, from zai-org BF16
MTP draft (layer 78)	MXFP4	festr2
Norms, gates, lm_head	BF16	RedHatAI

Expert files are L{3..77}.safetensors (256 experts each, AMD-Quark-compatible MXFP4 layout); model.safetensors.index.json stitches them with the FP8/NVFP4 non-expert tensors in hybrid-ct-*.safetensors. Per-expert held-out validation in L*_metrics.json (19,200/19,200 experts ≥ RTN; mean output error −19.6%).

Method (the part that's new)

Calibration data = 240K tokens captured from a serving deployment with an MoE hook recording each layer's true inputs and top-8 routing, so every expert is calibrated on the tokens it actually serves. GPTQ with act-order + error feedback onto the MXFP4 grid; down_proj calibrated against the quantized gate/up intermediate. Key finding for MoE GPTQ: per-expert Hessians are rank-deficient (tokens < dims) — textbook 1% damping makes GPTQ worse than RTN; damping at 100% of the mean diagonal is required (wide sweet spot 0.1–10×). Full method, per-layer table, and run logs in this repo's history and run-logs/. Quantization compute: 3.5 h on 4× H200 ($100).

Requirements

The checkpoint is 382 GiB and does not fit one GB10 (121 GiB unified memory). It requires four GB10 nodes (TP4) on a fast fabric — 200G RoCE/InfiniBand recommended; Ethernet boots but collectives are slow. Each node needs the driver, Docker, the NVIDIA Container Toolkit, and ~110 GiB free disk.

Quickstart

1. Download the checkpoint to the same path on every node:

hf download aidendle94/GLM-5.2-MXFP4-Experts-GPTQ --local-dir /models/glm52-hybrid

2. Pull the turnkey image on every node:

docker pull aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.3

vLLM + the b12x GB10 kernels with all required overlays baked in (the hybrid_mxfp4_ct quant config, index-authoritative loader, NVFP4 KV cache with per-token scaling, sparse-MLA CKV gather, boot-time kernel prewarm). No host-side patching.

3. Launch one container per node — identical except RANK (0 on head, 1–3 on workers, --headless on workers). Set HEAD_IP, NIC, and HCA for your fabric:

docker run -d --name glm52-hybrid \
  --network host --ipc host --shm-size 10gb --gpus all \
  --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
  --device /dev/infiniband:/dev/infiniband \
  -v /models/glm52-hybrid:/hybridmodel:ro \
  -v $HOME/glm-jit-cache:/cache/jit \
  -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_B12X_MOE=1 \
  -e VLLM_USE_B12X_SPARSE_INDEXER=1 \
  -e VLLM_DCP_SHARD_DRAFT=1 -e VLLM_DCP_GLOBAL_TOPK=1 -e VLLM_DSV4_INDEXER_SP=1 \
  -e VLLM_B12X_MLA_CKV_GATHER=1 -e VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=262144 \
  -e VLLM_B12X_MLA_DECODE_SPARSE_GATHER=1 -e VLLM_B12X_MLA_DECODE_GATHER_V2=1 \
  -e VLLM_ADAPTIVE_SPEC_DEPTHS=2,4 -e VLLM_NVFP4_MLA_PER_TOKEN_SCALE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a -e XDG_CACHE_HOME=/cache/jit \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 -e NCCL_IB_HCA=$HCA -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_SOCKET_IFNAME=$NIC -e GLOO_SOCKET_IFNAME=$NIC \
  -e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 -e NCCL_CUMEM_ENABLE=0 \
  -e RANK=$RANK -e HEAD_IP=$HEAD_IP \
  aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.3 \
  bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec vllm serve /hybridmodel \
    --served-model-name glm-5.2 --host 0.0.0.0 --port 8210 \
    --trust-remote-code --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --enable-prefix-caching \
    --tensor-parallel-size 4 --decode-context-parallel-size 4 --dcp-comm-backend ag_rs \
    --attention-backend B12X_MLA_SPARSE \
    --hf-overrides "{\"index_topk_pattern\":\"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS\"}" \
    --kv-cache-dtype nvfp4_ds_mla --kv-cache-memory-bytes 5500000000 \
    --max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 3 \
    --gpu-memory-utilization 0.89 \
    --speculative-config "{\"model\":\"/hybridmodel/mtp-draft\",\"method\":\"mtp\",\"num_speculative_tokens\":4,\"draft_attention_backend\":\"B12X_MLA_SPARSE\",\"adaptive_speculative_tokens_window\":32}" \
    --compilation-config "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"],\"cudagraph_capture_sizes\":[1,2,3,4,5,6,8]}" \
    --distributed-timeout-seconds 3600 --cpu-distributed-timeout-seconds 3600 \
    --distributed-executor-backend mp \
    --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP --master-port 29501'

4. Wait for readiness. First boot loads ~15–20 min per node, then compiles kernels (the image prewarms all decode shapes so none compile during serving). The /cache/jit mount makes later boots ~6 min. Ready when curl http://<head>:8210/health returns 200.

curl http://<head>:8210/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"glm-5.2","messages":[{"role":"user","content":"Capital of Estonia?"}],"max_tokens":64}'

The API is OpenAI-compatible. Reasoning defaults to maximum effort; pass "chat_template_kwargs":{"enable_thinking":false} to disable thinking or "reasoning_effort":"high" to lower it.

Configuration notes

• bash -lc 'unset …; exec vllm serve …' — the image bakes VLLM_PREFIX_CACHE_RETENTION_INTERVAL (valid for the DeepSeek lineage it also serves), but GLM-5.2 has no sliding-window KV group, so its KV coordinator rejects any value — and fails only after the full weight load. Docker -e VAR= sets it to an empty string, which the parser also rejects (int("")); there is no docker flag to remove a baked ENV. The only reliable fix is a shell unset before launch, hence the bash -lc wrapper. exec replaces the shell so SIGTERM reaches vLLM for clean shutdown.
• Per-node variables: RANK/HEAD_IP are referenced inside the single-quoted bash -lc string, so they must reach the container's environment — they are passed with -e RANK=$RANK -e HEAD_IP=$HEAD_IP (your host shell expands the right-hand side). NIC/HCA sit outside the quotes and are expanded by the host shell directly.
• index_topk_pattern (the 78-char FFFSSS… string) is derived from this config's indexer_types and must be passed verbatim.
• mtp-draft/ in this repo is the speculative-decoding draft. To disable speculation, omit --speculative-config (decode drops to ~12 tok/s).
• The 3600 s timeouts are needed on first boot: ranks finish their loads minutes apart.

Tuning (validated)

Setting	Value	Notes
--kv-cache-memory-bytes	5.5e9	688K-token pool; 7.0e9 → 875K, less host-memory headroom
--max-num-batched-tokens	2048	812 tok/s prefill; 1024 → 691 tok/s, ~0.5 GiB more headroom
VLLM_NVFP4_MLA_PER_TOKEN_SCALE	1	Per-token KV outer scale (KLD 0.40 vs 4.45 static-calibrated). Keep on; supersedes VLLM_NVFP4_MLA_SCALES_FILE
VLLM_FP8_LM_HEAD	0 (optional)	opt-in fp8 (W8A16) lm_head: +4-5% decode, tiny logit perturbation (quality-tested: greedy outputs equivalent, GSM8K 96%). Off by default for exact-bf16 logits
VLLM_ADAPTIVE_SPEC_DEPTHS	2,4	Pinning depth 4 measured worse (decode −25%)
--gpu-memory-utilization	0.89	Weights are ~99 GiB/rank; higher risks OOM

On 121 GiB unified-memory nodes, weights + KV + overhead run near capacity. If you use an OOM killer, allow for transient prefill spikes or whitelist the workers.

Running without the turnkey image

config.json declares quant_method: "hybrid_mxfp4_ct", a composed quant config (compressed-tensors for attention/dense, Mxfp4 for the routed experts) unknown to stock vLLM. Apply the two pieces in vllm_overlay/ — the quant-config class and an index-authoritative loader filter (see vllm_overlay/INSTALL.md). MoE then runs through any vLLM MXFP4 backend; only B12X on GB10 (SM121) is tested. The NVFP4 KV cache (nvfp4_ds_mla) requires a b12x build — with stock vLLM use --kv-cache-dtype fp8 (656 B/token vs 368, roughly half the KV capacity).

How it was built

RedHat's attention and dense tensors were extracted byte-exact into hybrid-ct-*.safetensors; festr2's unchanged MXFP4 expert tensors are referenced by a stitched model.safetensors.index.json, which is authoritative for tensor-to-file mapping. In v2 the shared experts (layers 3–77) were re-quantized to block-FP8 (128×128, symmetric, scale = amax/448) directly from the zai-org BF16 reference and swapped into the hybrid-ct files; the MTP layer-78 shared expert stays BF16. This localized change — the shared expert is ~5% of the non-expert weights — is the only difference from v1.

The MXFP4 draft measures spec acceptance ~3.0–3.2, versus ~4.4 for a distribution-matched draft on the unmodified base. Testing showed neither a BF16-expert draft nor deeper speculation changes this — the gap comes from the main model's MXFP4 output shift, inherent to the memory trade; the adaptive depth ladder extracts the best decode from it.

Validation

• Full-model teacher-forced KLD vs bf16: 0.102 (v2 FP8 shared; v1 NVFP4 shared was ~0.124), wikitext window, gather-off, 10-repeat, top-1 agreement 93%, median per-token KLD ~0.001
• 9/9 correctness (code, math, multilingual, factual), greedy
• GSM8K 96% (first 100 test problems, greedy, thinking off, fp8 lm_head on)
• Needle retrieval exact at 91,835 tokens
• Prefill 812 tok/s @32K; decode ~24 tok/s single-stream code with MTP
• NVFP4 KV per-token scaling: KLD 0.40 vs bf16 reference (4.45 static-calibrated, 5.22 uncalibrated)

Credits

zai-org (GLM-5.2, MIT) · RedHat AI (FP8/NVFP4 quantization) · festr2 (MXFP4 expert assembly) · AMD (Quark-calibrated MXFP4 weights). Assembly, loader glue, and GB10 validation by this repo. MIT, as inherited from all sources.

Credits to Koush, David Young, Dooner, Festr, Luke and many others who contribute to the b12x community.