canada-quant/GLM-5.3-Flash-W4A16-MTP

GLM-5.3-Flash — W4A16 (INT4) + BF16 MTP

INT4 weight-only quantization of zai-org/GLM-5.3-Flash with the BF16 MTP draft head preserved for speculative decoding. 177.7 GiB (−70%) — serves on 4× H100/H200, 4× RTX PRO 6000, or 2× DGX Spark at full 1M context.

🏆 Headline — where W4A16 wins (all measured, same-harness comparisons)

• −70% weights: 177.7 GiB vs ~599 GiB BF16 → 4 datacenter GPUs instead of 8, or a 2× DGX Spark desktop pair at full 1M context.
• vs NVFP4 (LibertAI, strongest community quant) — 2× DGX Spark: W4A16 serves 1,048,576 tokens there (KV pool 1.30×); the NVFP4 checkpoint OOM'd 9/9 boots at any context on identical nodes (dmesg-verified). Throughput @262K: +53% seq1 / +101% seq6.
• vs NVFP4 — H100 TP=4, matched MTP: wins c1 +4.8% · c8 +13.6% · saturated c32 +14.0% (vs NVFP4's best c32 config); mtp-off fixed-drafter c32 +22.4% — the batch edge is the marlin INT4 MoE GEMM, not the drafter.
• vs EXL3 (MiaAI-Lab 2×-Spark recipe) — independent matched-protocol head-to-head, FINAL: W4A16 leads single-stream +10.4% (33.0 vs 29.9 tok/s) and long-prefill +31–39% (~1.2K vs 864–914 tok/s @300K-class); KV capacity 1.30× vs 1.20×. The vendor README overstated decode ~2.2× and prefill ~1.2× vs independent measurement.
• Quality: AIME 2026 85.0% (102/120, n=120 final, max thinking) — beats the matched EXL3 arm 96/120 (80.0%) by +6 correct (+5.0 pt), FINAL 2026-09-06; AIME 2025 parity with NVFP4 on H100 (0.8833 vs 0.9000, 0.42σ); GSM8K 0.97; GPQA-Diamond within noise.
• SM120 (Blackwell): throughput parity with NVFP4 (±0.7%) with MTP paying at every concurrency — +69.9% c1 / +37.6% c8 / +47.8% c32.
• 4× H200 (SM90, 141 GB) — measured 2026-09-08 on AWS p5en.48xlarge, published recipe as-is (TP=4, MTP N=2): c1 200 · c8 672 · c32 1,234 · c64 1,507 · c128 1,751 · c256 ~1,970 output tok/s (8192/1024, thinking on), KV pool 6.19M tokens ≈ 23 concurrent 262K requests. Beats the best H100 row at every concurrency (+9% c1 / +10% c8 / +6% c32) and keeps scaling to c256. Prefix caching ON measured no gain on H200 (−2…−5%). 8× H200 TP=8: 210 · 823 · 1,636 · 2,063 · 2,385 · 2,674 (c1…c256), TPOT 4.5 ms single-stream. 2× H200 TP=2: 262K context with a 9.8× KV pool (c32 814 tok/s) and 1M context serving-validated (929K-token prompt admitted and decoded on two GPUs).
• Long context: single-stream decode is KV-length-insensitive to ≥486K (up to 357 tok/s, bf16 KV); long-prefill ceiling ~6.1–8.5K tok/s aggregate across the full 3-KV × c1–c32 matrix.
• Research-stack bug found and fixed: a 21-leg investigation root-caused a hang-forever admission wedge in the DFlash2 spec-decode stack and validated a fix to 256,388-token prompts (24/24 quality spot-checks pass).

Honest ledger: EXL3 leads mid-concurrency (c2 +67% / c4 +89%; ≈+81% cap-corrected c6); both sides saturate at c32 (EXL3 881–898 agg tok/s). SM120 vs NVFP4 is parity. The matched AIME-2026 quality arm vs EXL3 is FINAL 2026-09-06: W4A16 102/120 (85.0%) vs EXL3 96/120 (80.0%) — W4A16 wins. Results as of 2026-09-06; in-flight arms are labeled.

Why this quant exists

• BF16 GLM-5.3-Flash needs eight 80 GB GPUs per replica; this artifact cuts weights to 177.7 GiB — four GPUs, or a dual DGX Spark desktop at 1M context.
• Only the 36,288 routed-expert GEMMs are INT4 (symmetric, group-128, GPTQ); attention, router, shared experts, embeddings, the BF16 vision tower, and the BF16 MTP draft head (+40–50% single-stream) stay unquantized; canonical FP32 tensors preserved verbatim.
• Not a new model — all capability comes from the base model.

Hardware

The model's context is 1M tokens (1,048,576). Each row is the largest context serving-validated on that config — x86 rows are KV-memory-bound, not model-bound.

Config	GPUs	Validated context	Serving stack
DGX Spark GB10 (SM121), TP=2 + EP	2× 128 GB UMA	1M — KV pool 1,360,420 tok (1.30×)	DFlash2 drafter, fp8 KV — recipe below
RTX PRO 6000 (SM120), TP=4 + EP	4× 96 GB	512K (engine cap 524,288; 486K prompts measured)	MTP N=2, fp8 KV
H100 (SM90), TP=4 + EP	4× 80 GB	262K (256K prompts measured)	MTP N=2 or DFlash2, bf16 KV
H200 (SM90), TP=4 + EP	4× 141 GB	262K (engine cap; KV pool 6.19M tok = 23× a full 262K request)	MTP N=2, bf16 KV (fp8 KV impossible on Hopper for this NoPE model)
H200 (SM90), TP=8 + EP	8× 141 GB	262K (KV pool 7.79M tok = 30×)	MTP N=2, bf16 KV
H200 (SM90), TP=2 + EP	2× 141 GB	1M (KV pool 2.84M tok = 2.7× a 1M request; 929K-token prompt measured) or 262K with a 9.8× pool	MTP N=2, bf16 KV — recipe below

Quantization ran on 8× NVIDIA B300 (SM103) — see Quantization below; it is not a serving config.

Benchmarks vs competitors

Identical flags/prompts/hardware per row. Throughput = output tok/s, thinking ON.

vs EXL3 (MiaAI-Lab) — 2× DGX Spark, matched protocol, FINAL (2026-09-03)

• EXL3 was brought to full serving at 1M context on the same hardware (KV pool 1,200,348 = 1.20×; W4A16 1,360,420 = 1.30×).
• Like-for-like decode (TTFT excluded, 8K prompts, 256-token completions): W4A16 +10.4% c1 (33.0 vs 29.9 agg tok/s).
• EXL3 leads mid-concurrency: c2 +67.4% (59.6 vs 35.6), c4 +89.1% (112.7 vs 59.6), c6 ≈+80.9% after correcting its max-num-seqs=4 scheduler cap; it sustains ~29 tok/s per stream at every concurrency (W4A16 steps down 33.0→11.2 seq1→6 on the 1M serve; the 262K serve holds ~20 at seq4+).
• Saturated c32 (clean refire): EXL3 881–898 agg tok/s (32/32 streams, per-stream 27.4–28.1).
• W4A16 +31–39% long-prefill (~1.2K tok/s vs 864–914 @300K-class prompts).
• Vendor README overstated measured decode ~2.2× (claimed 62.9/51.7/37.1 per-stream c1/2/4 vs measured 26.6/21.2/17.0) and prefill ~1.2×.
• Two mid-bench hardware deaths on the competitor node were quarantined and never banked; the c32 set was re-run clean.
• Matched AIME-2026 quality arm: FINAL 2026-09-06 — W4A16 102/120 (85.0%) beats EXL3 96/120 (80.0%), +6 correct (+5.0 pt). EXL3 epochs 25/25/24/22, zero harness errors; 23/120 EXL3 rows hit the 131,072-token cap (W4A16: 16/18 misses were budget-exhaustion) — scored as written under the identical temp-0.0 / 131,072-token / n=120 config as our banked 85.0% arm.

4× H200 (SM90) — TP=4, published recipe, isl/osl 8192/1024, thinking ON (measured 2026-09-08, AWS p5en.48xlarge, image glm53-flash-x86_64-cu130)

Output tok/s, warm (second grid after boot — the first grid after boot under-reads by up to 2.2× at c8 on H200; cold rows are kept in the raws, never banked). Client: vllm bench serve --random-range-ratio 0.2 --ignore-eos, temp 1.0 / top_p 0.95, n = 8/32/64/256/512/768.

config	c1	c8	c32	c64	c128	c256	MTP accept	TPOT p50 ms (c1 → c256)
MTP N=2, prefix caching OFF (published) — FINAL, alone, mean of 3 reps	195	698	1,258	1,529	1,789	1,954	53–54%	4.5–4.9 / 10 / 20 / 38 / 66 / 118 — GSM8K-200 on this recipe on H200: 0.99
MTP N=2, prefix caching ON	188	660	1,236	1,482	1,771†	1,925†	52–53%	4.8 / 10.4 / 20.7 / 40.1 / 67.2 / 122
upstream nightly vllm-openai:nightly-x86_64, N=2, PC OFF	156	606	1,161	1,456	1,739	—	53–54%	5.9 / 11.7 / 22.4 / 35.8 / 58.6
MTP off (TP=4)	134	572	1,108	1,427	1,699	1,874	—	7.1 / 12.8 / 24.3 / 41.6 / 70.4 / 126
MTP N=1 · N=3 (TP=4)	167 · 196	670 · 685	1,181 · 1,142	1,473 · 1,448	1,718 · 1,679	1,954 · 1,811	65% · 42%	N=2 stays the recipe
8× H200, TP=8, N=2, PC OFF (alone, final reps)	217	843	1,642	2,042	2,399	2,681	53%	4.2 / 8.4 / 15 / 28 / 49 / 86 (c512: 2,834 but TTFT 33 s)
8× H200 as TWO TP=4 replicas, measured together (aggregate; per-replica c shown ×2)	391	1,363	2,486	3,067	3,550	3,911	53%	per-replica TPOT = the TP=4 row; +28% / +32% / +38% over TP=8 at equal total c128 / c256 / c512; zero interference
TP=4 long prompts 32K/1K · 131K/1K (c1 / c8 / c64 · c1 / c8 / c16)	165 / 385 / 510 · 98 / 136 / 138						54%	TTFT p50 1.5–5.4 s · 6.2–15.3 s; single-stream decode ~230 tok/s-class at 131K
TP=4 decode shape 1024/2048, N=2 (c1 / c8 / c32 / c128)	228 / 868 / 2,202 / 4,799						55%	TPOT 4.3 / 7.9 / 12.1 / 24.5; MTP +61% c1, +24% c32 vs MTP off
8× H200, TP=8, MTP off	141	692	1,439	1,931	2,342	2,633	—	6.8 / 10.5 / 18.7 / 30.7 / 50.9 / 89.2
8× H200, DP=2×TP=4 + EP, N=2 (probe)	142	275	1,134	1,713	2,018	2,412	n/a	not recommended — loses below c256
2× H200, TP=2, N=2, 262K mode	163	489	814	669			53%	5.5 / 14.3 / 31.5 / 91 (c64 queue-bound at mns 128)
2× H200, TP=2, N=2, 1M mode	160	476					55%	5.7 / 15.2; 131K prompt 64 tok/s c1 (TTFT 10 s); 929K prompt TTFT 97 s, TPOT 4.4 ms
DFlash2 K=7 · K=4 (incoai drafter) on upstream nightly + 2 patches, TP=4	207 · 205	661 · 702	1,135 · 1,263	1,395 · 1,483	1,490 · 1,687	1,530 · 1,750	21% · 33% of drafts	K=4 is the better depth but still −3…−10% vs MTP N=2 at c64–c256; +5% at c1
prefill shape 8192/128 (output tok/s · total tok/s), PC OFF	122 · 7.9K	246 · 16.0K	300 · 19.5K					TTFT p50 0.48 / 0.55 / 1.76 s

† measured with a second identical TP=4 serve benching concurrently on the other 4 GPUs (interference vs alone ≤ 5%, calibrated on the same recipe). 8-GPU guidance (measured): two TP=4 replicas behind a load balancer deliver 3,067 / 3,550 / 3,911 aggregate tok/s at total c128 / c256 / c512 versus 2,399 / 2,681 / 2,834 for one TP=8 endpoint (+28…38%), with zero cross-replica interference; TP=8 wins single-stream (217 vs 196 tok/s, TPOT 4.2 ms) and holds one 7.8M-token pool for very long requests. DP=2×TP=4 was measured and is not recommended. DFlash2 runs on H200 only on the upstream nightly with two small patches (maintained by the authors) and does not beat MTP N=2 on the pinned image. H100 (louse, same recipe, warm): c1 180 / c8 254 / c32 804; best H100 row ever (PC ON): 184 / 610 / 1,161 — H200 with PC OFF beats it at every point. Raws + scripts maintained by the authors; available on request.

vs NVFP4 — H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024

concurrency	W4A16 MTP N=2 (this)	NVFP4 MTP N=2	NVFP4 MTP N=5 (its best c32 config)
1	183.89	180.96	175.44
8	249.94	251.14	220.05
32	1161.13 †	770.66	1018.13
MTP acceptance	52–55%	~52%	~30%

• Headline: +4.8% c1 / +13.6% c8 / +14.0% saturated c32 vs NVFP4's banked N=5 config.
• † c=32 with --gpu-memory-utilization 0.92 (KV-pool capacity): 1161.13 tok/s (independent tuned run: 1145.05). The c=32 lever is KV-pool size, not a kernel gap.
• Fixed-drafter 2×2 re-confirm (both arms on the corrected #54374-port overlay): matched c1/c8 within noise (180.22/254.39 vs 182.13/252.89); saturated c32 win holds (+18.1%, 804.02 vs 680.99).
• MTP-off on both: W4A16 leads +2.7% / +10.5% / +22.4% at c1/c8/c32 — the c32 advantage is the marlin INT4 MoE GEMM at batch, not an MTP artifact.
• MTP adds +41.1%/+14.3%/−4.0% (W4A16) and +46.4%/+25.6%/−0.4% (NVFP4) at c1/c8/c32 on SM90.

vs NVFP4 — RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2

concurrency	W4A16 (this)	NVFP4	Δ
1	109.90	109.34	+0.5%
8	318.54	319.46	−0.3%
32	534.37	530.75	+0.7%

• Parity within noise (±0.7%); verbatim-methodology re-grid: c1 106.97 (−2.7%, 8-prompt high-variance class), c8 320.51 (+0.6%), c32 539.63 (+1.0%), MTP acceptance 50–53% — no stack drift.
• MTP pays at every concurrency on SM120: +69.9% c1 (106.97 vs 62.96), +37.6% c8 (320.51 vs 232.89), +47.8% c32 (539.63 vs 365.02) — a sign flip vs SM90 (drafter costs ~4% at saturated c32 there). Keep MTP on at every concurrency on SM120.
• KV-dtype at batch: fp8 vs bf16 costs nothing at c1/c8 (±0.05%) and ≈4.7% at c32 (365.02 vs 382.07) — lever ranking MTP ≫ KV dtype. (auto is the working bf16-KV path; bf16/bfloat16 variants fail on this stack.)

Long-context decode at batch (SM120, TP=4, 524K ctx, MTP N=2 — decode-isolated, TTFT excluded)

Per-stream decode tok/s (full-contention cohort bands — serialized prefills split stages into early/late cohorts, so bands, not blended medians):

KV length	KV dtype	c1	c4	c8	c32
8K (aggregate grid)	fp8	106.97	—	40.06	16.86
138.6K	fp8	143.94	2.92–7.53	2.49–3.74	2.24–2.72
138.6K	bf16 (auto)	152.19	3.23–4.21	2.53–2.64	2.45–2.63
277K	bf16 (auto)	177.15	2.23–6.15	2.28–4.32	2.17–2.79
486K	bf16 (auto)	357.34*	2.27–3.32	2.31–2.68	2.28–4.27†
486K	fp8	(degenerate‡)	2.31–3.26	2.37–3.62	2.03–3.93†

* n=1 short completion, noisy. † 21/32 rows — 11 bench-client TimeoutErrors disclosed in the authors' bench records. ‡ the fp8-KV c1 row recorded an absurd >480K tok/s (bench artifact) and is never cited. Additional bands: 277K c2 5.03, c6 2.29–3.38; 486K c6 2.34–3.68 (bf16) / 2.34–3.60 (fp8); 138.6K fp8 c2 9.58.

• Single-stream decode is KV-length-insensitive to ≥486K (143.94 → 177.15 → 357.34 tok/s at 138.6K/277K/486K bf16 — rises, if anything).
• At-batch collapse saturates with KV length: 13× per-stream drop at c8 vs the 8K grid, then flat ~2.2–3.7 tok/s/stream at c4–c32 from 138.6K through 486K — mechanism is target-model forward compute at batch × long-KV (4.4M tokens of live KV at c32), not the drafter.
• KV dtype is a c1-only ~5% lever (bf16 +5.7% @138.6K); at-batch bands overlap fp8/bf16 — the long-KV-amplifies-fp8-cost hypothesis is refuted.
• MTP acceptance never collapses: c1 ≈80–86% → c8 73.9% (n=1,286) → c32 71.6% (n=4,714) at 138.6K; 60.0% single-stream at 486K (small sample, disclosed). The saturated c32 stage decoded 2.53 tok/s/stream while the drafter accepted 71.6% — conclusively not acceptance-driven.
• At-batch lever ranking FINAL: MTP drafter ≫ everything measured; KV dtype ≈ c1-only ~5%; drafter acceptance is not the bottleneck.

Long-prefill at batch (SM120) — full 3-KV × c1–c32 matrix COMPLETE

• Single-stream prefill: ~6,100–7,100 tok/s for 277K–486K prompts (TTFT first request 39.2–79.3 s).
• At batch, long prefills are essentially serialized (stage wall ≈ N × single-stream time).
• Aggregate prefill ceiling ~6.1–8.5K tok/s — KV-length-insensitive AND batch-insensitive:
• 138.6K: 6,735 (c1) → 7,134 (c8) → 7,897 (c16) → 8,486 (c32) tok/s.
• 277K: 5,211 (c1) → 7,177 (c8) → 7,784 (c16) → 7,357 (c32) tok/s.
• 486K: 6,136 (c1) → 6,892 (c8) → 7,483 (c16) → 6,988.6 (c32) tok/s.
• Serialization ratios (wall / N×c1): 138.6K 0.94/0.85/0.79 · 277K 0.73/0.67/0.71 · 486K 0.89/0.82/0.88 (c8/c16/c32) — deeper-KV rows behave closer to strict serialization.
• mnb lever fully mapped: 8192 ✓ (validated production config) → 16384 ✓ (validated fallback, ceiling within ~1% at both 277K and 486K) → 32768 ✗ (CUDA OOM at 486K — 3.98 GiB needed vs 1.70 GiB free at mem-util 0.95). Ceiling is chunk-size-insensitive → attention/forward compute-bound, not scheduling-bound.
• Operational guidance: admission pacing / prefill scheduling is the only lever that moves long-context batch UX (mnb tuning and MTP do not address prefill serialization).

2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode

8K prompts, 256-token completions, aggregate tok/s:

config	seq1	seq2	seq4	seq6
W4A16+DFlash2 @262K (this)	33.29	42.01	80.98	121.81
NVFP4-MTP4 reference	21.8	41	66	60.5
Δ	+53%	+2.4%	+23%	+101%
W4A16+DFlash2 @1M serve	32.99	35.61	59.61	67.05

• 256K-token single prompt on the 1M serve: 30.85 tok/s decode (ttft 5.8 s).
• Structural: the NVFP4 checkpoint (90.67 GiB/node weights vs ~30 for W4A16) OOM'd all 9 boot attempts across 1M/300K/262K on the 121 GiB unified-memory nodes (dmesg-verified) — it cannot serve at any usable context on 2× Spark.

Quality

• AIME 2026 (30 problems, max thinking, 131,072-token budget, DFlash2 stack, 2× Spark): 102/120 = 85.0% — n=120 FINAL. Epochs 28/30, 25/30, 26/30, 23/30; 16/18 misses are budget-exhaustion truncations, 2/18 genuine wrong answers. The LibertAI NVFP4 external AIME-2025 reference (0.9000) remains +5.0 points ahead; the matched AIME-2026 EXL3 arm is FINAL 2026-09-06: 96/120 (80.0%) — W4A16 wins by +6 correct (+5.0 pt).
• AIME 2025 n=120, 4×30, max thinking, 131,072 budget — H100 (SM90): parity — W4A16 0.8833 (106/120) vs NVFP4 0.9000 (0.42σ). An earlier 0.7833 read was a serving-stack drafter-corruption bug (upstream vLLM #54374), since fixed.
• AIME 2025 — RTX PRO 6000 (SM120): raw-arm deficit real (~2σ; 0.8083/0.8000 pooled vs 0.9000), mechanism decomposed and fix VALIDATED. ≈63% of the deficit is a mechanical budget-wall (EMPTY-answer rate 11.7–14.2% vs NVFP4's 3.3%; doubling the budget recovered +5.8 pt); ≈37% is SM120 kernel numerics (near-tie argmax flips vs SM90, logit-quantified).
• Fix stack VALIDATED (zero GPU cost): whole-trace commit hook + thinking-budget force-commit (98,304-token cap) lifts the score to 0.8833 two-way / 0.8917 hook-credited — within noise of NVFP4 0.9000 (0.42σ / 0.21σ).
• Production-overlay pilot passed (committed 0.8917, zero modified non-empty responses across 120 requests); production adoption remains gated on the eval sign-off chain (owner decision pending).
• GSM8K: parity — 0.9704–0.9750 across all three quants on both x86 rigs.
• GPQA-Diamond n=198 @131k: 0.8586 vs 0.8687 NVFP4 (SM90), 0.8586 vs 0.8737 (SM120) — within noise.
• Matched AIME-2026 single-epoch arms on x86 (n=30, max thinking): W4A16 24/30 (RTX PRO 6000) and 22/30 (H100) vs NVFP4 23/30 — within noise at n=30.

Serving

The same checkpoint serves on every NVIDIA architecture from Hopper through Blackwell. Each recipe is the exact config the benchmarks ran.

SM90 — H100 / H200 (recommended: TP=4)

Re-download config.json if your copy predates 2026-09-08 (commit ece4697): older copies fail in vLLM with KeyError: 'layers.0.mlp.gate_up_proj.weight' — see the vLLM serving note under Details. Weights are unchanged.

Image vllm/vllm-openai:glm53-flash-x86_64-cu130 (CUDA 13.0). Gotcha: always pass --max-num-seqs ≤512 — the default 1024 exceeds this hybrid linear-attention model's 512 Mamba/KDA-state cache blocks.

docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 512 \
    --block-size 64 --gpu-memory-utilization 0.92 --no-enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8000

• MTP num_speculative_tokens: 2 is the sweet spot on Hopper (52–55% acceptance, best c1/c8); N=5 collapses acceptance (52→30%). H200 N=1 / N=3 / MTP-off and TP=8 rows are in flight (2026-09-08).
• H200 (141 GB) with the recipe above, unchanged — engaged: Using 'MARLIN' WNA16 MoE backend, FLASH_ATTN_MLA_SPARSE, SpeculativeConfig(method='mtp', num_spec_tokens=2), attention block auto-set to 1152 tokens (mamba page alignment; --block-size 64 is inert), GPU KV cache size: 6,194,528 tokens (76.5 GiB free after the 45.25 GiB/GPU weights). Prefix caching ON measured −2…−5% vs OFF at every concurrency → keep --no-enable-prefix-caching. Optional for c≳170 with N=2: --max-cudagraph-capture-size 1536 keeps large decode batches inside full CUDA graphs (default cap 512 tokens/step; costs ~0.4M tokens of KV). Same recipe also boots on upstream vllm/vllm-openai:nightly-x86_64 (v0.28.1rc1.dev516+, includes PR #53906): FLASHINFER_MLA_SPARSE_SM90 backend, ~6.06M-token pool, −22% c1 / −6% c32 / tie at c64–c128 vs the pinned image (and that backend IMAs on ≥131K prompts — pass --attention-backend FLASH_ATTN_MLA_SPARSE on the nightly).
• 8× H200 (TP=8): same recipe with --tensor-parallel-size 8 (engine init ~13 min; KV pool 7.79M tokens; attention block auto 640). Banked rows in the H200 table above.
• 2× H200 (TP=2) — validated 2026-09-09. Weights are 89.5 GiB/GPU, so the default CUDA-graph/chunked-prefill profiling OOMs; these flags boot:
  • 262K: recipe above with --tensor-parallel-size 2 --max-num-seqs 128 --max-cudagraph-capture-size 128 → KV pool 2,567,410 tokens (9.8× a 262K request); c1 163 · c8 489 · c32 814 tok/s.
  • 1M context: --tensor-parallel-size 2 --max-model-len 1048576 --max-num-seqs 16 --gpu-memory-utilization 0.95 --max-cudagraph-capture-size 64 --max-num-batched-tokens 4096 → KV pool 2,840,958 tokens (2.7× a full 1M request); a 929K-token prompt was admitted and decoded (TTFT 97 s, TPOT 4.4 ms). --max-num-batched-tokens 4096 costs prefill speed (~9.6K tok/s) and is what keeps profiling inside memory.

SM120 — RTX PRO 6000 (Blackwell)

Image cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 (SM120-patched build).

docker run --gpus '"device=4,5,6,7"' --ipc=host --network=host --rm \
  -v /models:/models cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 8192 \
    --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 --enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8001

• KV fp8 is required at 262K on 96 GB cards; --max-num-seqs 64 is bench-validated (48/36 fallback ladder); MTP N=2 matches NVFP4 to ±0.7%.

SM121 — 2× DGX Spark GB10 (desktop, 1M context)

Full 1,048,576-token context on two Sparks over RoCE (TP=2, expert-parallel) with the DFlash2 block-diffusion drafter. One launcher on both nodes; worker rank first, wait 25 s, then head rank (mandatory stagger).

Current production config (live since 2026-09-08): CUDA graphs ON + GMU 0.90. The published @1M numbers (decode 32.99/35.61/59.61/67.05, AIME26 102/120) were measured on the previous banked config (GMU=0.795, --enforce-eager); the current config re-measured equivalent within run-to-run noise (graphs-on vs eager decode wash), and the GMU raise itself was pre-validated (@262K staging: c2 +38.7%, c6 +10.6% vs 0.795).

Production launch (current):

# rank1 (worker) FIRST → wait 25 s → rank0 (head):
MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 GMU=0.90 EAGER=0 GRAPHS=1 bash launch_dflash2_tp2.sh <rank>

The full launcher is inlined below — the card is self-contained (the script previously lived out-of-tree). Launcher defaults = production for the remaining knobs: MAX_NUM_SEQS=6 BLOCK_SIZE=2304 SPEC_NUM_TOKENS=7.

Engine args the launcher produces (weights bind-mounted at /models/glm-5.3-w4a16-mtp; set MODEL_DIR to your local weights path):

vllm serve /models/glm-5.3-w4a16-mtp/ \
  --served-model-name glm-5.3-flash --host 0.0.0.0 --port 8000 --trust-remote-code \
  --tensor-parallel-size 2 --enable-expert-parallel \
  --gpu-memory-utilization 0.90 \
  --max-model-len 1048576 --max-num-seqs 6 --block-size 2304 \
  --speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":7}' \
  --kv-cache-dtype fp8_e4m3 --kv-cache-memory 9663676416 \
  --compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,24,32]}' \
  --tool-call-parser glm47 --enable-auto-tool-choice --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --distributed-executor-backend mp --nnodes 2 --node-rank <0|1> \
  --master-addr <HEAD_IP> --master-port 29521    # + --headless on rank1

Stack + topology:

• Image note (2026-09-11): the pinned SM121 image radixark/vllm-glm53-flash:sm121-v11-dflash2 is no longer published — its Docker Hub repo returns 404 as of 2026-09-11. Our fleet runs a locally cached copy (an aarch64/CUDA-13.0 SM121 vLLM build carrying the DFlash2 serving patches; the stock upstream DFlash2 image cannot boot this drafter — see the defect section below). Until an upstream stack lands, the reproducible paths are: (a) upstream vLLM main (≥ vllm-project/vllm#53906) serves these weights on Spark with MTP — without DFlash2; (b) the exact patched SM121 build recipe is maintained by the authors and is available to collaborators on request.
• Drafter incoai/GLM-5.3-Flash-DFlash2 mounted at /models/dflash2-draft; weights = this repo (178 GiB, bind-mounted into the container; config.json must be the current published one (the ignore list carries both HF and vLLM module spellings; older 765-entry copies fail to load)).
• Drafter internals: block 8, selector_rank 256, top_k 16, layers [5,14,24,33,42]; 7 speculative tokens (any other count boot-wedges).
• KV pool: 1,360,420 tokens fp8_e4m3 = 1.30× a full 1M request.
• RoCE fabric: NCCL_IB_HCA=rocep1s0f1, GID index 3, RoCEv2, 192.168.102.0/24 (enp1s0f1np1), NCCL_NVLS_ENABLE=0 — switch path at 97.98 Gbps line-rate (direct-attach measured worse).
• Container: --memory=118g --shm-size=32g --ipc=host --network=host, expandable_segments:True, TORCH_CUDA_ARCH_LIST=12.1a + FLASHINFER_CUDA_ARCH_LIST=12.1a, VLLM_ENGINE_READY_TIMEOUT_S=3600 + VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 (cold JIT otherwise kills boot), top-k patch bind-mount (sparse_attn_indexer_kpool.py), HF_HUB_OFFLINE=1.
• Cold boot ≈6–10 min; graceful docker stop -t 30 only — never rm -f a GPU-active container.
• Keep single prompts ≤ ~310K tokens (larger wedged a host twice).
• CUDA graphs (FULL_AND_PIECEWISE, capture sizes 1–32) are the current production config — adopted 2026-09-08 after re-measuring equivalent to eager; the historical eager banked config is noted under deltas below.

Full launcher (verbatim, sha256 440ca004b719…):

#!/usr/bin/env bash
# GLM-5.3-Flash W4A16 + DFlash2 speculative decoding — TP2 across two DGX Spark GB10 (SM121) nodes.
# Adapted from tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark/launch-glm53-vllm-tp2-dflash2.sh
# (their 46.9 tok/s config) for a two-Spark RoCE fabric:
#   - W4A16 weights (/home/pcozz/models/glm-5.3-w4a16-mtp/) instead of NVFP4
#   - fabric: rocep1s0f1 / enp1s0f1np1 / 192.168.102.0/24
#   - proven W4A16 serve args (EP on, fp8_e4m3 KV, block 2304, seqs 6)
#   - wedge-safety: graceful docker stop -t 30 (NEVER rm -f a GPU-active container), --memory=118g cap
# Usage: launch_dflash2_tp2.sh <0|1>  — worker (1) FIRST, wait 25s, then head (0).
# Env knobs (defaults = 262K serve):
#   MAX_MODEL_LEN  (default 262144; 1048576 = architecture max, needs KV_CACHE_MEM raise)
#   KV_CACHE_MEM   (default 3221225472 = 3 GiB -> 310,292 tok pool, ~10.38 KB/tok fp8;
#                   1M pool needs ~10.9 GiB -> use 11811160064 = 11 GiB -> ~1.09M tok;
#                   safer 1M value: 9663676416 = 9 GiB -> ~1.36M)
#   MODEL_DIR      (default /home/pcozz/models/glm-5.3-w4a16-mtp = W4A16)
set -euo pipefail
MAX_MODEL_LEN="${MAX_MODEL_LEN:-262144}"
KV_CACHE_MEM="${KV_CACHE_MEM:-3221225472}"
MAX_NUM_SEQS="${MAX_NUM_SEQS:-6}"
MODEL_DIR="${MODEL_DIR:-/home/pcozz/models/glm-5.3-w4a16-mtp}"
# DFlash2 drafter lever: override num_speculative_tokens (default 7; any other count boot-wedges).
SPEC_NUM_TOKENS="${SPEC_NUM_TOKENS:-7}"
# EAGER=1 (default) = --enforce-eager; current production passes EAGER=0 GRAPHS=1 (CUDA graphs ON).
EAGER="${EAGER:-1}"
if [ "$EAGER" = "1" ]; then EAGER_FLAG="--enforce-eager"; else EAGER_FLAG=""; fi
# Note: engine may raise block-size to >= mamba page size (see boot log).
BLOCK_SIZE="${BLOCK_SIZE:-2304}"
# Banked default 0.795; current production passes GMU=0.90.
GMU="${GMU:-0.795}"
# GRAPHS=1 + EAGER=0 = FULL_AND_PIECEWISE capture sizes 1-32 (current production).
GRAPH_ARGS=()
if [ "${GRAPHS:-0}" = "1" ]; then GRAPH_ARGS=(--compilation-config '{"cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[1,2,4,8,16,24,32]}'); fi

NODE_RANK="${1:?usage: launch_dflash2_tp2.sh <0|1>}"
[[ "$NODE_RANK" == "0" || "$NODE_RANK" == "1" ]] || { echo "rank must be 0 or 1" >&2; exit 2; }

IMAGE="radixark/vllm-glm53-flash:sm121-v11-dflash2"
NAME="vllm_node"
MODEL_HOST_PATH="$MODEL_DIR"
MODEL_PATH="$MODEL_DIR/"
DRAFTER_HOST_PATH="${DRAFTER_HOST_PATH:-/var/tmp/models/GLM-5.3-Flash-DFlash2}"
CACHE_HOST_PATH="/var/tmp/glm53-vllm-cache"
HEAD_IP="${HEAD_IP:-192.168.102.1}"
MPORT="${MPORT:-29521}"
PORT="8000"

case "$NODE_RANK" in
  0) HOST_IP="${HOST_IP_RANK0:-192.168.102.1}"; HEADLESS="" ;;
  1) HOST_IP="${HOST_IP_RANK1:-192.168.102.2}"; HEADLESS="--headless" ;;
esac

test -f "$MODEL_HOST_PATH/config.json"
test -f "$DRAFTER_HOST_PATH/config.json"
test -f "$HOME/patches/sparse_attn_indexer_kpool.py"
mkdir -p "$CACHE_HOST_PATH"

# Wedge-safe replace: graceful stop first (never rm -f a GPU-active container — UVM wedge).
if docker ps --format '{{.Names}}' | grep -qx "$NAME"; then
  echo "gracefully stopping running $NAME ..."
  docker stop -t 30 "$NAME" >/dev/null || true
fi
docker rm "$NAME" 2>/dev/null || true

docker run --gpus all -d \
  --name "$NAME" --restart no \
  --memory=118g \
  --network host --ipc host --shm-size 32g \
  --ulimit memlock=-1:-1 --cap-add IPC_LOCK \
  --device /dev/infiniband:/dev/infiniband \
  -v /home/pcozz/models:/home/pcozz/models \
  -v "$CACHE_HOST_PATH:/cache" \
  -v "$CACHE_HOST_PATH/flashinfer:/root/.cache/flashinfer" \
  -v "$CACHE_HOST_PATH/tilelang:/root/.tilelang" \
  -e VLLM_HOST_IP=$HOST_IP \
  -e HF_HOME=/cache/huggingface \
  -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1 \
  -e VLLM_ENGINE_READY_TIMEOUT_S=3600 \
  -e VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS="${VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS:-3600}" \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e TORCH_CUDA_ARCH_LIST=12.1a -e FLASHINFER_CUDA_ARCH_LIST=12.1a \
  -e FLASHINFER_DISABLE_VERSION_CHECK=1 \
  -e NCCL_NET=IB -e NCCL_IB_DISABLE=0 \
  -e NCCL_IB_HCA=rocep1s0f1 -e NCCL_IB_GID_INDEX=3 \
  -e NCCL_IB_ROCE_VERSION_NUM=2 -e NCCL_IB_ADDR_FAMILY=AF_INET \
  -e NCCL_IB_ADDR_RANGE=192.168.102.0/24 \
  -e NCCL_SOCKET_IFNAME=enp1s0f1np1 -e GLOO_SOCKET_IFNAME=enp1s0f1np1 \
  -e TP_SOCKET_IFNAME=enp1s0f1np1 -e MN_IF_NAME=enp1s0f1np1 \
  -e NCCL_NVLS_ENABLE=0 -e NCCL_CROSS_NIC=0 -e NCCL_IB_MERGE_NICS=0 \
  -e NCCL_CUMEM_ENABLE=0 -e NCCL_IGNORE_CPU_AFFINITY=1 -e NCCL_DEBUG=WARN \
  -e NCCL_BLOCKING_WAIT=0 \
  -e TORCH_NCCL_ASYNC_ERROR_HANDLING=1 \
  -e TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC=1800 \
  -e TORCH_NCCL_DISABLE_WATCHDOG=1 \
  -v $HOME/patches/sparse_attn_indexer_kpool.py:/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/sparse_attn_indexer_kpool.py:ro \
  -v $DRAFTER_HOST_PATH:/models/dflash2-draft:ro \
  "$IMAGE" \
  "$MODEL_PATH" \
  --served-model-name glm-5.3-flash \
  --host 0.0.0.0 --port "$PORT" \
  --trust-remote-code \
  --tensor-parallel-size 2 \
  --enable-expert-parallel \
  --gpu-memory-utilization "$GMU" \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs "$MAX_NUM_SEQS" --block-size "$BLOCK_SIZE" \
  --speculative-config '{"method":"dflash","model":"/models/dflash2-draft","num_speculative_tokens":'"$SPEC_NUM_TOKENS"'}' \
  --kv-cache-dtype fp8_e4m3 --kv-cache-memory "$KV_CACHE_MEM" \
  $EAGER_FLAG \
  "${GRAPH_ARGS[@]}" \
  --tool-call-parser glm47 --enable-auto-tool-choice \
  --reasoning-parser glm45 \
  --default-chat-template-kwargs '{"enable_thinking":true}' \
  --distributed-executor-backend mp \
  --nnodes 2 --node-rank "$NODE_RANK" \
  --master-addr "$HEAD_IP" --master-port "$MPORT" \
  $HEADLESS

echo "launched $NAME rank=$NODE_RANK host=$HOST_IP"
sleep 2
docker ps --format '{{.Names}} {{.Status}}' | grep "$NAME" || {
  echo "$NAME exited; inspect with: docker logs $NAME" >&2
  exit 1
}

The bind-mounted sparse_attn_indexer_kpool.py top-k patch is part of the authors' SM121 patch stack (available on request); the launcher tests for it before starting.

Labeled deltas — NOT the current production config:

• @262K staging (behind the 33.29/42.01/80.98/121.81 table row above): MAX_MODEL_LEN=262144 KV_CACHE_MEM=3221225472 → pool 310,292 tokens.
• Previous banked config (behind the published @1M table): GMU=0.795 + EAGER=1 (--enforce-eager), launched as MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 bash launch_dflash2_tp2.sh <rank>.

Docker images

Upstream note (2026-09-03): GLM-5.3-Flash support landed in vLLM main (vllm-project/vllm#53906, merged 16:40 UTC). The pinned images below remain the validated serving stacks for this quant; upstream-main validation is a separate, queued leg.

Architecture	Image
SM90 (H100/H200)	vllm/vllm-openai:glm53-flash-x86_64-cu130 (validated); vllm/vllm-openai:nightly-x86_64 ≥ 2026-09-08 also boots this checkpoint (upstream PR #53906) — second arm, see H200 table
SM120 (RTX PRO 6000)	cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1
SM121 (DGX Spark)	radixark/vllm-glm53-flash:sm121-v11-dflash2 — no longer published (Docker Hub 404 as of 2026-09-11); the patched SM121 build recipe is maintained by the authors, available to collaborators on request

Quick start

# 1. Download the weights (~178 GiB)
huggingface-cli download canada-quant/GLM-5.3-Flash-W4A16-MTP --local-dir /models/glm53-flash-w4a16-mtp

# 2. Serve (SM90 example — see recipes above)
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel --max-num-seqs 512 \
    --max-model-len 262144 --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code

# 3. Call it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "glm53-w4",
  "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}]
}'

Details

Field	Value
Base model	zai-org/GLM-5.3-Flash (BF16)
Architecture	Glm5NextForConditionalGeneration (glm5_next) — 45 decoder layers (+ MTP layer 45), hidden 4096, 288 routed experts (top-8) + 1 shared, MoE-intermediate 2048, KDA + DSA attention, 24-block vision tower, vocab 154,880
Weight quantization	W4A16, INT4, symmetric, group-size 128 (GPTQ, compressed-tensors pack-quantized), routed experts only
Quantized tensors	36,288 (42 MoE layers × 288 experts × 3 GEMMs) — verified exact
Kept in BF16	attention (KDA + DSA incl. indexer), dense prefix (layers 0–2), shared experts, router/gate, mHC tensors, embeddings, lm_head, norms, vision tower (348 keys), MTP layer 45 (889 keys)
Canonical FP32 preserved	A_log, dt_bias, e_score_correction_bias, hc_* — verbatim from source
Calibration	256 samples × 4096 tokens, in-distribution chat/code mix (glm53_mix), sequential per-layer GPTQ
Size	177.7 GiB (from ~599 GiB BF16)
License	MIT (inherited from the base model)

vLLM serving note (config.json updated 2026-09-08 — re-download config.json if your copy predates this): vLLM's Glm5NextForConditionalGeneration matches quantization_config.ignore against its own fused module names (language_model.model.layers.N.mlp.gate_up_proj, language_model.lm_head, MTP model.layers.45.*), not the HF names llm-compressor wrote. The ignore list therefore carries both spellings plus the fused gate_up_proj entries and a prefix-agnostic re:.*\.layers\.45\..* for the MTP head. With an older config.json (765 entries, sha256 92ae6f37…) vLLM fails at load with KeyError: '…mlp.gate_up_proj.weight'; the fix is the current config.json only — weights are unchanged.

Validation

Static gates at build time (2026-08-27), all PASS:

• exactly 36,288 packed tensors; zero compression outside routed experts;
• vision key-set 348/348 identical to source, zero compressed, all dtypes match;
• MTP layer present (889 keys), index rewrite purely additive;
• dtype drift vs source: zero (282 pipeline-drifted tensors restored verbatim);
• expert scale audit: no zero/collapsed scales.

Quantized on 8× NVIDIA B300, 2026-08-27. B300 smoke tests (transformers ≥5.16): loads as Glm5NextForConditionalGeneration (~110 s); text generation coherent; image captioning accurate.

Vision-quality note: the vision tower is BF16-passthrough but was not covered by the text-only calibration set. Image smoke tests pass; a vision eval leg (MMMU-class + OCR) is queued.

Research-stack known issue: DFlash2 admission wedge — root-caused across 21 legs; FIX VALIDATED (2026-09-04)

Affects the SM90 research stack only (x86_64, DFlash2 drafter n=7). The standard MTP serve path used by all production configs above is unaffected.

• Symptom: prompts above a block-size-dependent bound are never admitted — Waiting: 1 / Running: 0, 0% KV, retry-forever until timeout (no crash, no OOM).
• Measured bounds: block 2304 → wedges above ~15.5K tokens; block 1024 → ~17.4K; block 1536 → no wedge at all across the tested 16–24K class.
• Mechanism (pinned): spec-decode admission requires a full-sequence reservation against the unified KV pool (1,012 blocks); the binding group is the drafter's SlidingWindowManager (block-16) — a 15,523-token request needs exactly 1,011 blocks (passes, zero margin), a ~15,556-token request needs 1,013 (wedges).
• Root cause (pinned): the drafter SW manager allocates execution-side at full-sequence scale (cdiv(tokens,16)) while its true sliding-window need is only 130 blocks; the built-in skip-free mechanism fires but freed blocks are instantly re-consumed by the next full-sequence allocation.
• Fix VALIDATED: cap the SW manager's execution allocation at min(cdiv(tokens, bs), cdiv(sliding_window, bs) + 2) (= 130 blocks) — serves 8,528 / 15,787 / 25,344 / 63,734 / 127,952 / 256,388 tokens, all PASS at the shipping block-2304 config (pre-fix wedge at ≥15,537), zero engine errors.
• Quality spot-suite PASS: 24 deterministic wedge-class prompts, fixed stack vs spec-OFF control — 24/24 checks both arms, zero tokenization mismatch; no detectable quality impact at spot scope.
• Workarounds without the patch: block 1536 serves the whole tested 16–24K class (its SW cap saturates) and holds the largest KV pool (1.12× of a 262K request vs 0.78× at block 2304).
• Upstream status: the stock upstream DFlash2 image cannot boot this drafter at all, so the defect is provable only on the research overlay; the closest upstream PR (#50169) would not fix it (no execution-side cap). A fully-evidenced issue was FILED 2026-09-08 as vllm-project/vllm#55800 (file-with-disclosure); production adoption of the patch = owner decision.
• Full leg chain: findings/df2w*.md (DF2W → DF2W27).

SM121 kernel-level research findings (2026-09-09)

• Custom-kernel program COMPLETE — the stock marlin remains the serving kernel. A pre-registered kernel-optimization program for the SM121 W4A16 MoE GEMM (the GEMM measured at 84.2% of the c1→c6 step-time delta; 0.3604 ms/tok/layer slope) executed to exhaustion: a custom persistent-ticket-scheduler marlin ("E-K2b") was bit-correct everywhere (0 output flips vs the no-scheduler reference across the full M sweep) but never met the ≤1.10 scheduling-overhead adoption gate (best ≈1.29 at M≥64), and all three follow-up axes — O1 claim-batching, O2 barrier-elision, O3 claim-ahead pipelining — were cleanly falsified (overheads 1.29–1.36, monotone worse). No custom kernel code ships; everything on this card runs the stock moe_wna16_marlin heuristic.
• Upstream defect found and filed: vllm-project/vllm#56064. Forcing the marlin no-split-K (data-parallel, split_k=1) path produces a deterministic CUDA illegal memory access at M=256 on SM121/GB10 — clean at M≤128 on the same path, and the stock split-K-allowed call is clean at M=256. Reproduced on two independently built extension modules using the kernel's own determine_exec_config pick ⇒ the defect is in the shared marlin core's no-split-K path, not in caller scheduling. Serving is unaffected (the stock heuristic is the split-K path); this matters only to kernel researchers forcing DP/no-split-K execution (e.g. custom schedulers).
• Own-drafter program ("dspark2"): specs + Phase-0 reference stack done; not trained. A zero-reuse (no radixark code) own drafter for this checkpoint is fully specced (12-step build order; SWA-2048, fp8 KV parity, K-adaptive draft depth, cudagraph-safe) with an 18-module stdlib reference implementation (237 unit tests green). No trained weights exist yet; the drafter behind every number on this card remains the radixark DFlash2 drafter. Program target: closing the mid-concurrency gap vs EXL3 documented above.
• Evidence: every load-bearing result above is backed by a sha256-pinned artifact set (per-leg raw logs, verdict rows, and a consolidated findings document prepared for external review) maintained by the authors; the kernel-level defect reproduction is carried inline in the upstream issue #56064. Detailed artifacts are available to collaborators on request.

This section contributed via the Digby.ai coding harness from sha-pinned banked artifacts; reviewed by a human (the maintainer) before publication.

License

MIT, inherited from the base model. Follow the base model's usage terms.

---

Built, benchmarked, and documented with the Digby.ai coding harness, developed by CQL.ca.