GLM-5.2-Alis-MLX-Dynamic-4.5bpw

Apple Silicon (MLX) mixed-precision quantization of zai-org/GLM-5.2 — a 744B-parameter (~40B active) Mixture-of-Experts model with DeepSeek-V3.2-style MLA + DeepSeek Sparse Attention (DSA, glm_moe_dsa).

This is the quality-flagship sibling of GLM-5.2-Alis-MLX-Dynamic-3.5bpw: the routed experts (≈96% of all parameters) are promoted from 3-bit affine to NVFP4 (4-bit float, group-size 16) — a ~2.2× reduction in expert weight-space relative error (rel-err) — while everything else (recipe, MLA/DSA fixes, int8 MLA-KV, native MTP layer) is identical. Choose by context budget on a 512 GB M3 Ultra:

	this build (4.5 bpw, nvfp4 experts)	3.5 bpw sibling
quality	higher (expert rel-err 0.104 vs 0.225)	good
max context in 512 GB (int8 KV)	~500–600K tokens	~1M tokens
decode, tok/s (single request, short ctx)	16.3 · 18.7–19.0 with --mtp	20.4 · 20.7

Native MTP included: the checkpoint ships GLM-5.2's MTP (nextn) layer for lossless self-speculative decoding (--mtp). On this cleaner 4-bit target, speculation pays measurably more than on the 3-bit sibling — accept-len 2.61–2.87 and +15–16% (vs ~2.3 and ≈+2% on the sibling) — see Native MTP.

⚠️ Requires a patched mlx-lm with the glm_moe_dsa correctness fixes and int8 MLA-KV support — and, new for this build, an mlx core with NVFP4 kernels: mlx ≥ 0.30.3 (tested on 0.31.2). Older runtimes fail to load mode="nvfp4" tensors.

This build uses IndexShare — the DSA indexer is stored only on the 21 "full" layers ([0,1,2,6,10,…,74]); the other 57 layers reuse it. Stock mlx-lm / omlx build an indexer on every layer, so loading fails with ValueError: Missing 285 parameters: …self_attn.indexer… (57 shared layers × 5 params). Use avlp12/mlx-lm, branch glm-5.2-dsa-inference (upstream PR ml-explore/mlx-lm#1419, open as of writing), which adds IndexShare + non-interleaved indexer RoPE + int8 MLA-KV. Serve with mlx_lm.server from that fork. On stock mlx-lm, --kv-bits is also a silent no-op for this architecture (see Correctness and Long context).

pip install "git+https://github.com/avlp12/mlx-lm@glm-5.2-dsa-inference"

---

Quality

The routed experts are ~96% of all parameters; their precision dominates build quality. At the same storage cost as 4-bit affine, NVFP4 (E2M1 values + FP8-E4M3 per-16 group scales + per-tensor FP32 global scale) cuts expert reconstruction error by ~7% on heavy-tailed expert weights vs affine-4, and by ~2.2× vs the 3-bit sibling:

Expert weight-space reconstruction (mean rel-err, cosine)
3-bit affine g64 (3.5 bpw sibling)	0.225 / 0.978
NVFP4 g16 (this build)	0.104 / 0.994

Per-layer conversion telemetry across all 75 MoE layers (gate/up/down): rel-err 0.1006–0.1035, no outlier layers (threshold 0.13).

End-to-end PPL (strided, ctx 2048 / stride 1024)	this build	3.5 bpw sibling	Δ
wikitext-2 (prose)	2.713	2.946	−7.9%
code (mlx_lm src)	1.821	1.893	−3.8%

Strided perplexity from a fixed local harness — relative numbers for comparing builds, not directly comparable to perplexities other quantizers report on different corpora.

Base model	zai-org/GLM-5.2 (744B total / ~40B active)
Bits/weight	~4.5 (per-tensor mixed; experts nvfp4)
On-disk size	~424 GB (395 GiB) — incl. the 4.5 GB native MTP layer
Peak memory (load + ~2K-ctx gen, MTP attached)	424 GB (measured)
Peak memory (600K ctx, int8 KV)	~455–460 GB estimated — brushes the ~460 GB wired limit
Format	MLX (Apple Silicon), requires mlx ≥ 0.30.3
Context	~500K comfortable / ~600K max in 512 GB (int8 KV; DSA sparse attention) — for 1M use the 3.5 bpw sibling
Speculative decoding	native MTP (nextn layer 78 included)

---

Recipe

Identical to the 3.5 bpw sibling except the expert tier — bits are allocated by sensitivity: cheap-but-accurate float4 on the robust expert bulk, full precision on the discrete routing/sparse-attention control paths.

Component	Bits	Why
Routed experts (gate/up/down)	NVFP4 g16 (4-bit float)	~96% of params — the bulk; float format wins on heavy-tailed experts
Shared experts · MLA attn (incl. kv_a/q_a) · dense MLP	4-bit affine g64	on every token's critical path
Token embedding · LM head	6-bit affine g64	distribution-sensitive
Router (mlp.gate)	bf16	drives discrete top-8 routing — never quantized
DSA lightning indexer	fp16	drives discrete top-k selection

The NVFP4 expert tensors are stored as per-projection stacked 3-D weights (switch_mlp.{gate,up,down}_proj) with {"group_size": 16, "bits": 4, "mode": "nvfp4"} per-path overrides in config.json (225 entries, 75 MoE layers × 3 projections). gather_qmm with mode="nvfp4" was gate-checked for exactness (measured max-abs difference vs the dequantized-weight reference: 0.000000 on the expert forward) and speed (parity with same-bpw affine at decode batch sizes, ±1%).

---

Long context (~500–600K) — int8 KV

GLM-5.2's MLA stores the compressed latent (kv_lora 512 + rope 64 per layer), so the KV cache is small: ~95 GB per 1M tokens at fp16, ~48 GB per 1M at int8. At this build's weight footprint the practical ceiling on 512 GB is ~500K tokens comfortable, ~600K max with int8 KV (at 600K the estimated peak, ~455–460 GB, brushes the ~460 GB wired-memory limit); for a full 1M-token budget use the 3.5 bpw sibling.

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw \
  --kv-bits 8 --kv-group-size 64 --quantized-kv-start 4096 \
  --prompt "…"

# OpenAI-compatible server
mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw \
  --kv-bits 8 --quantized-kv-start 4096

The patched runtime quantizes only the MLA latent cache to int8 (the DSA indexer cache stays fp16) and dequantizes the latent on read inside MLA attention. At ≤128K context, fp16 KV is fine (a few GB); int8 KV extends the ceiling from ~350K (fp16) to ~500–600K.

Why int8 and not fp8? Measured on the actual latent (post-kv_a_layernorm, std ≈ 0.006, max ≈ 0.03): int8-affine (g64) cos 0.99998 vs MXFP8 (g32) 0.9997 vs plain fp8 e4m3 0.994 — int8 is 100–200× lower MSE on both the latent and the decoupled RoPE key. fp8's wider dynamic range doesn't help a narrow, well-conditioned distribution where precision (mantissa bits) is the binding constraint, and MLX dequantizes the latent to fp16 for the MLA matmul anyway. (Note this is about the KV cache; the expert weights are a different, heavy-tailed distribution — that's where a float4 format does win, hence nvfp4 experts + int8 KV in the same build.)

---

Native MTP — self-speculative decoding

GLM-5.2 ships a built-in MTP (multi-token-prediction / "nextn") layer that predicts token t+2 from the backbone's hidden state at t — DeepSeek-V3-style. This build includes it as model.layers.78.* (one extra shard, +4.5 GB). The MTP shard is reused unchanged from the 3.5 bpw sibling (experts 3-bit g64, attention 4-bit, indexer/norms/eh_proj bf16) — drafter precision was measured not to affect acceptance (bf16 ≡ 3-bit), so the drafter is deliberately kept small while the nvfp4 promotion applies to the backbone experts only.

Backward compatible: loaders without MTP support drop layer 78 in sanitize() and behave identically to a non-MTP checkpoint. With the fork, --mtp turns it on:

mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw \
  --mtp --prompt "…"                        # k=2 chained drafts (default)

mlx_lm.server --model avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw \
  --mtp --mtp-num-draft-tokens 2            # OpenAI-compatible serving
  # optional: --mtp-hybrid  (adds conservative prompt-lookup drafting
  #            for repetition-heavy workloads: long quotes, JSON, boilerplate)

MTP pays more on this build than on the 3-bit sibling. Speculation gains scale with target precision — a cleaner target accepts more drafts (its own outputs are less noisy) — measured across this quant family: 3-bit experts ≈ +2% (accept-len ~2.3, the 3.5 bpw sibling), 4-bit-class experts ≈ +11% (accept-len ~2.46, measured on a 4-bit sibling). Measured on this build (M3 Ultra 512 GB, single request, greedy, short context):

	plain	--mtp (k=2)	--mtp (k=3)
decode	16.34 tok/s	18.71 tok/s (+15%)	18.95 tok/s (+16%)
mean accepted length	—	2.61	2.87

It stays exactly lossless — at temperature 0 the output is the model's own greedy output; at temperature > 0 drafts are accepted only when they equal the target's sampled token (distribution-exact) — so enabling it costs nothing but the ~4.5 GB layer.

Long context: MTP works past index_topk (2048 tokens — the DSA-sparse regime); the fork's small-L gather path (commit 946c412) keeps the multi-token verify from paying the sparse-mask setup per iteration. Measured on this build at ~2.15K context, prefill-separated: plain 15.89 tok/s, --mtp k=1 ≈ 0.98× (accept-len 1.79), k=2 ≈ 0.86× (accept-len 2.31), k=3 ≈ 0.79×. Guidance: k=2–3 in the ≤2048 dense regime; at longer contexts prefer --mtp-num-draft-tokens 1 or plain — the residual gap is acceptance economics (accepted length vs the ~2.1+ break-even in the sparse regime), not path cost.

Notes for other integrators (the three footguns, all verified against vLLM's deepseek_mtp semantics):

1. eh_proj input concat order is [enorm(embed(t+1)), hnorm(h_t)] — embedding first.
2. The hidden fed to the MTP is the backbone's pre-final-norm residual stream.
3. For chained multi-token drafting, feed back the MTP's shared_head.norm-normalized hidden — chaining the raw layer output halves chained acceptance (measured 0.27 → 0.75 conditional at position 2).

---

How it compares

Within this two-build family, the 3.5 bpw sibling is the 1M-context build; this one is the quality build (~500–600K ctx). Against other public GLM-5.2 MLX builds in the 4.5–5 bpw tier, this one is the smallest on disk, and it is the only one of the compared builds shipping the native MTP layer, int8 MLA-KV, and the DSA correctness fixes:

	this build	3.5 bpw sibling	Q4.8-INF	DQ4plus-q8
effective bpw	~4.5 (experts nvfp4)	~3.5	~4.8	~5.0
on-disk	424 GB	333 GB	447 GB	465 GB
native MTP layer	✓ included	✓ included	✗ stripped	✗ stripped
DSA indexer	fp16	fp16	(custom fmt)	8-bit
int8 MLA-KV	yes	yes	no	no
max ctx in 512 GB	~500–600K (int8 KV)	~1M (int8 KV)	fp16 KV only — 1M ✗	fp16 KV only — 1M ✗

Per-component bit allocation was parsed from each build's config.json. The nvfp4 expert tier gives this build higher weight fidelity than Q4.8-INF/DQ4plus on the dominant parameter mass at 20–40 GB less footprint, plus the runtime correctness fixes the others lack.

---

Benchmarks

mlx_lm.perplexity (seq 2048, 50 samples, seed 123) — the same harness and settings as the sibling cards, against the author's earlier builds:

	GLM-5.1 · 2.7 bpw	GLM-5.2 · 2.56 bpw	GLM-5.2 · 3.5 bpw	GLM-5.2 · 4.5 bpw (this)
Perplexity (lower)	4.165	3.850	3.766	3.812 ± 0.031

On this small-sample chat corpus (allenai/tulu-3-sft-mixture, the mlx_lm.perplexity default) the two 512 GB-class siblings are within noise of each other; the higher-precision strided measurement in Quality above (−7.9% wikitext / −3.8% code, ~23–28K scored tokens per corpus) is the discriminating comparison. A 0-shot task-accuracy table (HellaSwag / PIQA / WinoGrande) may be added in a future card update.

---

Correctness (verified vs the HF reference)

glm_moe_dsa needs fixes beyond the stock mlx-lm port; this build was produced with a patched fork and validated:

• IndexShare — the DSA indexer runs only on "full" layers; "shared" layers reuse its top-k (index_topk_freq=4). The stock port built an indexer on every layer → missing-weights / wrong >2048-token output.
• Indexer RoPE / eps — non-interleaved (half-split) RoPE + LayerNorm eps 1e-6, distinct from the interleaved main attention. Baked into config.json (indexer_rope_traditional=false, indexer_norm_eps=1e-6); post-RoPE q matches the reference to ~1e-7.
• int8 MLA-KV — CacheList.to_quantized/offset + MLA dequant-on-read, so --kv-bits 8 actually engages for the latent cache (silently ignored on stock mlx-lm).
• Native MTP — layer 78 restored + --mtp self-speculative decoding in the fork; greedy output verified identical to non-speculative decoding, chained drafting matches vLLM's deepseek_mtp semantics (normed-hidden chaining).
• NVFP4 expert path — gather_qmm(mode="nvfp4") output matches the dequantized-weight reference exactly on the expert forward, at affine-parity speed; mixed nvfp4+affine per-layer config round-trips through save → load_model → generate (gate-checked before the build).

Validation (this build): full load under the wired-limit protocol (peak 424 GB, ~2.15K-ctx run); coherent greedy generation at short and ~2.15K context; MTP speculative path exercised at k=1–3 (accept-len 2.61–2.87 short / 1.79–2.55 long — greedy-identity is enforced by the acceptance rule, verified exhaustively on the sibling); per-layer conversion telemetry rel-err 0.1006–0.1035 across all 75 MoE layers (no outliers); config round-trip with 225 nvfp4 per-path overrides. Strided PPL vs the sibling in Quality above. Fork validated at branch head 2b2138f.

---

Usage

# requires the fork (glm_moe_dsa fixes + int8 MLA-KV + --mtp) on mlx ≥ 0.30.3
mlx_lm.generate --model avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw \
  --prompt "Write a quicksort in Python."

Hardware

Built for 512 GB Apple Silicon (M3 Ultra). Weights ~424 GB (395 GiB); with int8 KV a ~500K-token context runs comfortably in 512 GB (~600K max). For a full 1M-token context, or for more free headroom, use the 3.5 bpw sibling; for ≤256 GB machines, the 2.56 bpw build.

Credits

• Base model: Zhipu / Z.ai — GLM-5.2 (MIT).
• MLX & mlx-lm: Apple ml-explore (NVFP4 kernels since mlx 0.30.3).
• Mixed-precision quantization, glm_moe_dsa correctness fixes, int8 MLA-KV cache support, native-MTP restoration + speculative serving, and the nvfp4 expert promotion: Alis (avlp12).

Citation

Alis (avlp12) (2026). GLM-5.2-Alis-MLX-Dynamic-4.5bpw — 4.5 bpw (nvfp4-expert) MLX quantization of GLM-5.2. https://huggingface.co/avlp12/GLM-5.2-Alis-MLX-Dynamic-4.5bpw