aidendle94/GLM-5.2-Hybrid-FP8-MXFP4

GLM-5.2 Hybrid — FP8/NVFP4 attention + MXFP4 experts

⚠️ A newer version is available: v3

aidendle94/GLM-5.2-MXFP4-Experts-GPTQ supersedes this model. It re-quantizes every routed expert with GPTQ calibrated on real routed activations instead of round-to-nearest — same size, same speed, same kernels, but full-model teacher-forced KLD vs BF16 drops 0.102 → 0.080 (−17.7%), top-1 agreement rises to 93.9%, and reasoning-length inflation shrinks from +63% to +56%. It is a complete, standalone checkpoint (drop-in replacement). New deployments should use v3. This repo is kept for provenance and as a fallback.

A mixed-precision GLM-5.2 checkpoint for memory-constrained multi-node inference, built and validated on 4× NVIDIA DGX Spark (GB10) with vLLM TP4+DCP4.

v2 (this revision) — the shared experts moved from NVFP4 to block-FP8, quantized directly from the BF16 reference rather than transplanted from a 4-bit source. This cuts the full-model teacher-forced KL divergence vs bf16 from ~0.124 to 0.102 (wikitext window, gather-off, 10-repeat, top-1 agreement 93%) at no change to speed or the memory footprint. Everything else — attention, dense, routed experts, MTP draft, KV — is identical to v1.

Component	Precision	Source
Attention (MLA projections, indexer)	block-FP8 (128×128, compressed-tensors)	RedHatAI/GLM-5.2-NVFP4-FP8
Dense MLPs (layers 0–2)	NVFP4 (group-16, compressed-tensors)	RedHatAI
Shared experts (layers 3–77)	block-FP8 (128×128, compressed-tensors)	quantized from zai-org/GLM-5.2 BF16 (v2)
Routed experts, layers 3–77	MXFP4 (e2m1 + ue8m0 per-32)	festr2/GLM-5.2-BF16-AMDMXFP4experts
MTP draft experts (layer 78)	MXFP4	festr2
Norms, gates, lm_head, MTP dense + shared	BF16	RedHatAI

Why: MXFP4 uses one e8m0 scale per 32 elements vs NVFP4's e4m3 per 16 — same 4-bit weights, half the scale bytes. Moving the routed experts (95%+ of the model) to MXFP4 shrinks the checkpoint from 416 to 382 GiB; the freed ~5 GiB/GPU becomes KV cache.

Measured (4× DGX Spark, TP4+DCP4, MTP): prefill 812 tok/s @32K (depth-flat past 48K), decode ~24 tok/s single-stream code (spec acceptance ~3.0–3.4), GSM8K 96% (greedy, thinking off, fp8 lm_head arm), needle retrieval exact at 91,835 tokens, KV pool up to 875K tokens.

Requirements

The checkpoint is 382 GiB and does not fit one GB10 (121 GiB unified memory). It requires four GB10 nodes (TP4) on a fast fabric — 200G RoCE/InfiniBand recommended; Ethernet boots but collectives are slow. Each node needs the driver, Docker, the NVIDIA Container Toolkit, and ~110 GiB free disk.

Quickstart

1. Download the checkpoint to the same path on every node:

hf download aidendle94/GLM-5.2-Hybrid-FP8-MXFP4 --local-dir /models/glm52-hybrid

2. Pull the turnkey image on every node:

docker pull aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.3

vLLM + the b12x GB10 kernels with all required overlays baked in (the hybrid_mxfp4_ct quant config, index-authoritative loader, NVFP4 KV cache with per-token scaling, sparse-MLA CKV gather, boot-time kernel prewarm). No host-side patching.

3. Launch one container per node — identical except RANK (0 on head, 1–3 on workers, --headless on workers). Set HEAD_IP, NIC, and HCA for your fabric:

docker run -d --name glm52-hybrid \
  --network host --ipc host --shm-size 10gb --gpus all \
  --cap-add IPC_LOCK --ulimit memlock=-1:-1 \
  --device /dev/infiniband:/dev/infiniband \
  -v /models/glm52-hybrid:/hybridmodel:ro \
  -v $HOME/glm-jit-cache:/cache/jit \
  -e VLLM_USE_V2_MODEL_RUNNER=1 -e VLLM_USE_B12X_MOE=1 \
  -e VLLM_USE_B12X_SPARSE_INDEXER=1 \
  -e VLLM_DCP_SHARD_DRAFT=1 -e VLLM_DCP_GLOBAL_TOPK=1 -e VLLM_DSV4_INDEXER_SP=1 \
  -e VLLM_B12X_MLA_CKV_GATHER=1 -e VLLM_B12X_MLA_CKV_GATHER_MAX_TOKENS=262144 \
  -e VLLM_B12X_MLA_DECODE_SPARSE_GATHER=1 -e VLLM_B12X_MLA_DECODE_GATHER_V2=1 \
  -e VLLM_ADAPTIVE_SPEC_DEPTHS=2,4 -e VLLM_NVFP4_MLA_PER_TOKEN_SCALE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e CUTE_DSL_ARCH=sm_121a -e TORCH_CUDA_ARCH_LIST=12.1a -e XDG_CACHE_HOME=/cache/jit \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 -e NCCL_IB_HCA=$HCA -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_SOCKET_IFNAME=$NIC -e GLOO_SOCKET_IFNAME=$NIC \
  -e NCCL_MAX_NCHANNELS=4 -e NCCL_MIN_NCHANNELS=4 -e NCCL_CUMEM_ENABLE=0 \
  -e RANK=$RANK -e HEAD_IP=$HEAD_IP \
  aidendle94/sparkrun-vllm-ds4-gb10:production-hybrid-1.3 \
  bash -lc 'unset VLLM_PREFIX_CACHE_RETENTION_INTERVAL; exec vllm serve /hybridmodel \
    --served-model-name glm-5.2 --host 0.0.0.0 --port 8210 \
    --trust-remote-code --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --enable-prefix-caching \
    --tensor-parallel-size 4 --decode-context-parallel-size 4 --dcp-comm-backend ag_rs \
    --attention-backend B12X_MLA_SPARSE \
    --hf-overrides "{\"index_topk_pattern\":\"FFFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSSFSSS\"}" \
    --kv-cache-dtype nvfp4_ds_mla --kv-cache-memory-bytes 5500000000 \
    --max-model-len 262144 --max-num-batched-tokens 2048 --max-num-seqs 3 \
    --gpu-memory-utilization 0.89 \
    --speculative-config "{\"model\":\"/hybridmodel/mtp-draft\",\"method\":\"mtp\",\"num_speculative_tokens\":4,\"draft_attention_backend\":\"B12X_MLA_SPARSE\",\"adaptive_speculative_tokens_window\":32}" \
    --compilation-config "{\"cudagraph_mode\":\"FULL_AND_PIECEWISE\",\"custom_ops\":[\"all\"],\"cudagraph_capture_sizes\":[1,2,3,4,5,6,8]}" \
    --distributed-timeout-seconds 3600 --cpu-distributed-timeout-seconds 3600 \
    --distributed-executor-backend mp \
    --nnodes 4 --node-rank $RANK --master-addr $HEAD_IP --master-port 29501'

4. Wait for readiness. First boot loads ~15–20 min per node, then compiles kernels (the image prewarms all decode shapes so none compile during serving). The /cache/jit mount makes later boots ~6 min. Ready when curl http://<head>:8210/health returns 200.

curl http://<head>:8210/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"model":"glm-5.2","messages":[{"role":"user","content":"Capital of Estonia?"}],"max_tokens":64}'

The API is OpenAI-compatible. Reasoning defaults to maximum effort; pass "chat_template_kwargs":{"enable_thinking":false} to disable thinking or "reasoning_effort":"high" to lower it.

Configuration notes

• bash -lc 'unset …; exec vllm serve …' — the image bakes VLLM_PREFIX_CACHE_RETENTION_INTERVAL (valid for the DeepSeek lineage it also serves), but GLM-5.2 has no sliding-window KV group, so its KV coordinator rejects any value — and fails only after the full weight load. Docker -e VAR= sets it to an empty string, which the parser also rejects (int("")); there is no docker flag to remove a baked ENV. The only reliable fix is a shell unset before launch, hence the bash -lc wrapper. exec replaces the shell so SIGTERM reaches vLLM for clean shutdown.
• Per-node variables: RANK/HEAD_IP are referenced inside the single-quoted bash -lc string, so they must reach the container's environment — they are passed with -e RANK=$RANK -e HEAD_IP=$HEAD_IP (your host shell expands the right-hand side). NIC/HCA sit outside the quotes and are expanded by the host shell directly.
• index_topk_pattern (the 78-char FFFSSS… string) is derived from this config's indexer_types and must be passed verbatim.
• mtp-draft/ in this repo is the speculative-decoding draft. To disable speculation, omit --speculative-config (decode drops to ~12 tok/s).
• The 3600 s timeouts are needed on first boot: ranks finish their loads minutes apart.

Tuning (validated)

Setting	Value	Notes
--kv-cache-memory-bytes	5.5e9	688K-token pool; 7.0e9 → 875K, less host-memory headroom
--max-num-batched-tokens	2048	812 tok/s prefill; 1024 → 691 tok/s, ~0.5 GiB more headroom
VLLM_NVFP4_MLA_PER_TOKEN_SCALE	1	Per-token KV outer scale (KLD 0.40 vs 4.45 static-calibrated). Keep on; supersedes VLLM_NVFP4_MLA_SCALES_FILE
VLLM_FP8_LM_HEAD	0 (optional)	opt-in fp8 (W8A16) lm_head: +4-5% decode, tiny logit perturbation (quality-tested: greedy outputs equivalent, GSM8K 96%). Off by default for exact-bf16 logits
VLLM_ADAPTIVE_SPEC_DEPTHS	2,4	Pinning depth 4 measured worse (decode −25%)
--gpu-memory-utilization	0.89	Weights are ~99 GiB/rank; higher risks OOM

On 121 GiB unified-memory nodes, weights + KV + overhead run near capacity. If you use an OOM killer, allow for transient prefill spikes or whitelist the workers.

Running without the turnkey image

config.json declares quant_method: "hybrid_mxfp4_ct", a composed quant config (compressed-tensors for attention/dense, Mxfp4 for the routed experts) unknown to stock vLLM. Apply the two pieces in vllm_overlay/ — the quant-config class and an index-authoritative loader filter (see vllm_overlay/INSTALL.md). MoE then runs through any vLLM MXFP4 backend; only B12X on GB10 (SM121) is tested. The NVFP4 KV cache (nvfp4_ds_mla) requires a b12x build — with stock vLLM use --kv-cache-dtype fp8 (656 B/token vs 368, roughly half the KV capacity).

How it was built

RedHat's attention and dense tensors were extracted byte-exact into hybrid-ct-*.safetensors; festr2's unchanged MXFP4 expert tensors are referenced by a stitched model.safetensors.index.json, which is authoritative for tensor-to-file mapping. In v2 the shared experts (layers 3–77) were re-quantized to block-FP8 (128×128, symmetric, scale = amax/448) directly from the zai-org BF16 reference and swapped into the hybrid-ct files; the MTP layer-78 shared expert stays BF16. This localized change — the shared expert is ~5% of the non-expert weights — is the only difference from v1.

The MXFP4 draft measures spec acceptance ~3.0–3.2, versus ~4.4 for a distribution-matched draft on the unmodified base. Testing showed neither a BF16-expert draft nor deeper speculation changes this — the gap comes from the main model's MXFP4 output shift, inherent to the memory trade; the adaptive depth ladder extracts the best decode from it.

Validation

• Full-model teacher-forced KLD vs bf16: 0.102 (v2 FP8 shared; v1 NVFP4 shared was ~0.124), wikitext window, gather-off, 10-repeat, top-1 agreement 93%, median per-token KLD ~0.001
• 9/9 correctness (code, math, multilingual, factual), greedy
• GSM8K 96% (first 100 test problems, greedy, thinking off, fp8 lm_head on)
• Needle retrieval exact at 91,835 tokens
• Prefill 812 tok/s @32K; decode ~24 tok/s single-stream code with MTP
• NVFP4 KV per-token scaling: KLD 0.40 vs bf16 reference (4.45 static-calibrated, 5.22 uncalibrated)

Credits

zai-org (GLM-5.2, MIT) · RedHat AI (FP8/NVFP4 quantization) · festr2 (MXFP4 expert assembly) · AMD (Quark-calibrated MXFP4 weights). Assembly, loader glue, and GB10 validation by this repo. MIT, as inherited from all sources.

Credits to Koush, David Young, Dooner, Festr, Luke and many others who contribute to the b12x community.