cfontes/glm-5.3-flash-dflash2-tp4

GLM-5.3-Flash NVFP4 + DFlash2 on 4× DGX Spark (TP4)

69.3 tok/s single-stream coding peak — matching tonyd2wild's TP4 DFlash2 reference of 68.5 tok/s.

This repo documents a working deployment of GLM-5.3-Flash (320B MoE, 18B active) in NVFP4 quantization with DFlash2 block-diffusion speculative decoding across four NVIDIA DGX Spark (GB10) nodes at tensor-parallel 4.

📊 Benchmark Results

All benchmarks measured with llm-inference-bench v0.4.29.

🏆 Headline: Coding Peak — 69.3 tok/s (temp=0, c1)

Run	tok/s	Tokens	TTFT	Latency	Reasoning
1	68.4	2000	0.24s	29.2s	0 chars
2 (median)	69.3	2000	0.19s	28.8s	0 chars
3 (max)	74.0	2000	0.21s	27.0s	0 chars

• Prompt: "Write a Python script that implements the Sieve of Eratosthenes"
• Config: temp=0, 2000 max tokens, single stream, thinking disabled
• Output: ~7,000 chars of Python code + explanation per run
• Zero reasoning chars — thinking was off, all tokens went to content

Sustained Decode — Mixed Prompts (default temp)

30s per test, 512 max output tokens, ignore_eos=true.

Concurrency	Context	tok/s (agg)	per-req tok/s	Accept Length	Accept Rate	TTFT	ITL
1	0	36.7	36.7	2.93	0.276	0.35s	26.2ms
1	4096	31.6	31.6	3.11	0.302	2.75s	25.0ms
2	0	56.1	28.1	2.81	0.259	3.72s	35.6ms
4	0	84.7	21.2	2.80	0.257	0.49s	46.0ms
2	4096	52.5	26.3	3.03	0.290	3.09s	35.0ms
4	4096	64.1	16.0	3.05	0.293	6.19s	130.4ms

Sustained Decode — temp=0 (greedy decoding)

Concurrency	Context	tok/s (agg)	Accept Length	Accept Rate	TTFT
1	0	42.2	3.28	0.325	0.27s
1	4096	36.8	3.29	0.329	2.37s
2	0	64.0	3.14	0.307	0.39s

🔬 Per-Position Acceptance Rates (c1, ctx=0)

DFlash2 proposes 7 draft tokens per step. Each position has independent acceptance probability — the further from the anchor token, the harder it is to predict.

Position	Default temp	temp=0	Δ
1	72.8%	78.6%	+5.8pp
2	49.6%	57.5%	+7.9pp
3	29.9%	35.1%	+5.2pp
4	20.0%	22.7%	+2.7pp
5	10.1%	16.8%	+6.7pp
6	5.9%	10.3%	+4.4pp
7	4.8%	6.7%	+1.9pp

Fun fact: Position 1 acceptance at temp=0 is 78.6% — nearly 4 out of 5 first-draft tokens are correct. By position 7, only 1 in 15 survives. The drafter is excellent at "obvious next tokens" (code keywords, closing brackets) but loses confidence on creative choices.

📈 Scaling: Concurrency vs Throughput

Concurrency	Aggregate tok/s	Per-request tok/s	Accept Length	ITL
1	36.7	36.7	2.93	26ms
2	56.1	28.1	2.81	36ms
4	84.7	21.2	2.80	46ms

Fun fact: At c4, the cluster pushes 84.7 tok/s aggregate — but each individual user sees only 21.2 tok/s. DFlash2's acceptance rate stays remarkably stable (2.80–2.93) across concurrency levels, unlike MTP which degrades more. The spec-decode rejection sampler handles batched drafts without quality loss.

🧠 KV Cache & Memory Budget

Metric	Value
KV cache dtype	FP8 e4m3
KV pool size	~2.86M tokens (v2 config)
Block size	2304 tokens
KV blocks	1,243
Max model length	262,144 tokens (256K)
GPU memory utilization	0.75 (75%)
Weights per node	~45 GiB (NVFP4, 120 shards)
Draft model	2.2 GiB (5-layer Qwen3)
Total model per node	~47 GiB

Fun fact: The KV pool holds 2.86 million tokens — enough for 43 concurrent 64K-context conversations, or 1,100+ concurrent 2K-context chats. FP8 KV cache doubles this vs BF16. The DFlash2 drafter keeps its own KV in BF16 but shares the target's embed_tokens/lm_head, saving ~1.5 GiB vs a standalone draft model.

⚡ Prefill Throughput

Context	Tokens	TTFT	Client tok/s	Server tok/s
4K	4,168	2.37s	1,760	1,779
8K	8,196	4.84s	1,693	—
64K	64,490	37.7s	1,710	—
128K	128,850	76.1s	1,693	—

Fun fact: Prefill throughput is ~1,700 tok/s regardless of context length — the linear attention layers earn their keep at depth. A 128K prompt (roughly a 500-page book) processes in 76 seconds. The GB10's unified memory means no CPU↔GPU transfers for the 182GB weight set.

Comparison vs References

Config	c1 tok/s	c1 Accept	c4 tok/s	Prompt Type
GLM-5.3 MTP-4 (our baseline)	27.6	2.23	69.5	mixed
Our DFlash2 (mixed, default temp)	36.7	2.93	84.7	mixed
Our DFlash2 (mixed, temp=0)	42.2	3.28	—	mixed
Our DFlash2 (coding peak, temp=0)	69.3	~4.5	—	code
tonyd2wild DFlash2 TP2 (code)	46.9	~4.49	—	code
tonyd2wild DFlash2 TP2 (structured, t=0)	60.6	~5.0	—	structured
tonyd2wild DFlash2 TP4 (code, abliterated)	68.5	~4.49	100.1 (c6)	code
joesinvestments DFlash2 TP4 (code, t=0, SGLang)	32.4	—	—	code
joesinvestments DFlash2 TP4 (structured, t=0)	48.1	—	—	structured

Key insight: DFlash2 throughput is highly prompt-dependent. Code and structured output achieve 48-74 tok/s; mixed/prose prompts sit at 18-40 tok/s. This matches tonyd2wild's and joesinvestments' findings. The win is in single-stream agent traffic where outputs are predictable.

What Did NOT Work (v3 experiment)

We tested a v3 config with --gpu-memory-utilization 0.80, --kv-cache-memory 16GiB (pinned), --enable-chunked-prefill, and --default-chat-template-kwargs '{"enable_thinking": false}'. Results regressed:

Config	c1 tok/s	c1 Accept	Δ tok/s
v2 (baseline)	42.2	3.28	—
v3 (all changes)	31.5	2.46	-25%

Root cause: Acceptance dropped 25% (3.28→2.46). The engine ran at the same steps/s (12.8) but accepted fewer drafts per step. Likely culprits:

1. --kv-cache-memory pin may be too small for TP4, causing KV pressure
2. --enable-chunked-prefill (validated on SGLang, not vLLM) may feed the drafter partial context
3. --default-chat-template-kwargs changes prompt format, affecting draft predictions

Lesson: DFlash2 is sensitive to KV cache configuration and prefill scheduling. Changes that help MTP or SGLang can hurt DFlash2 on vLLM. Test one variable at a time.

Raw benchmark JSONs:

• benchmarks/glm53-dflash2-bench.json — mixed prompts, default temp
• benchmarks/glm53-dflash2-bench-temp0.json — mixed prompts, temp=0
• benchmarks/glm53-dflash2-bench-code.json — coding peak, temp=0
• benchmarks/glm53-dflash2-bench-v3.json — v3 regression experiment

🔧 Hardware Configuration

Component	Value
Nodes	4× NVIDIA DGX Spark (GB10)
GPU per node	1× GB10 (128GB unified memory)
Total GPUs	4
Tensor parallel	4
Interconnect	RoCE on rocep1s0f1 (10.0.1.x subnet)
Node IPs	192.168.1.201-204 (mgmt), 10.0.1.1-4 (RoCE)
Docker image	radixark/vllm-glm53-flash:sm121-v8-dflash2-tony

🏗️ How It Works

DFlash2 Block-Diffusion Speculative Decoding

DFlash2 is a block-diffusion speculative decoding method that uses a lightweight 5-layer Qwen3 drafter to predict 7 tokens per step. The drafter captures auxiliary hidden states from 5 target model layers (indices 5, 14, 24, 33, 42), uses non-causal attention with a sliding window of 2048, and shares the target model's embedding and LM head layers.

Per-step flow:

1. Target model runs one forward pass → 1 token + 5 aux hidden states
2. Drafter runs forward → 7 candidate tokens (using aux states + its own KV)
3. Rejection sampler verifies all 7 in parallel
4. Accepted tokens emitted; rejected tokens discarded
5. Accepted: 1 + accepted_count tokens per step

Why it's fast: The drafter is 5 layers vs the target's 120 layers. Even with 7 draft tokens, the total compute is ~1.04× a single target forward pass. At accept=3, you get 4 tokens per step for ~1.04× compute — a 3.8× speedup on the forward pass.

NVFP4 Quantization

GLM-5.3-Flash uses NVIDIA's NVFP4 (4-bit floating point) quantization:

• 182GB total weight size (vs ~640GB FP8, ~1.3TB BF16)
• 120 safetensors shards, ~1.5GB each
• MoE experts in NVFP4, shared layers in FP8
• Marlin MoE backend for fast expert dispatch
• FP8 e4m3 KV cache (halves KV memory vs BF16)

🚀 Reproduction

Prerequisites

• 4× DGX Spark nodes with RoCE interconnect
• Docker image radixark/vllm-glm53-flash:sm121-v8-dflash2-tony on all nodes
• GLM-5.3-Flash NVFP4 weights at /var/tmp/glm-5.3-flash-nvfp4/ (182GB, 120 shards)
• DFlash2 draft model at /var/tmp/glm-5.3-flash-dflash2/ (2.2GB)

Launch

# Copy launch script to all nodes
scp scripts/launch-glm53-dflash2-tp4.sh admin@<node>:/home/admin/

# Launch workers first (rank 3, 2, 1), then head (rank 0)
ssh admin@node4 'NODE_RANK=3 bash /home/admin/launch-glm53-dflash2-tp4.sh'
ssh admin@node3 'NODE_RANK=2 bash /home/admin/launch-glm53-dflash2-tp4.sh'
ssh admin@node2 'NODE_RANK=1 bash /home/admin/launch-glm53-dflash2-tp4.sh'
# Wait 10s
ssh admin@node1 'NODE_RANK=0 bash /home/admin/launch-glm53-dflash2-tp4.sh'

# Wait ~10 min for model loading + kernel compilation
# Check: curl http://node1:8000/v1/models

Benchmark

# Coding peak (the headline number)
llm_decode_bench.py --host <node1> --port 8000 \
  --model glm-5.3-flash-dflash2 \
  --coding-peak --temperature 0.0 \
  --max-tokens 2000 --display-mode plain

# Sustained decode
llm_decode_bench.py --host <node1> --port 8000 \
  --model glm-5.3-flash-dflash2 \
  --concurrency 1,2,4 --contexts 0,4096 \
  --max-tokens 512 --duration 30 \
  --temperature 0.0 --display-mode plain

📝 Key Learnings from the Community

From joesinvestments/GLM-5.3-Flash-FP8-4x-DGX-Spark

1. DFlash2 acceptance warms slowly — first measurement after boot shows +11%, but warmed acceptance reaches +37%. Never benchmark a cold spec-decode server.
2. DFlash2 is prompt-dependent — structured output: 48 tok/s, math: 46 tok/s, code: 32 tok/s, prose: 18 tok/s. "Acceptance runs cold on prose."
3. Chunked prefill 2048 is a GB10 finding from Light Foundry — helps SGLang prefill but may not transfer to vLLM (our v3 regression confirms this).
4. --load-format instanttensor gives 15× load speedup but causes silent rank death in multi-node. tonyd2wild ships it disabled. We confirmed and excluded it.
5. Partial MoE Triton tuning is WORSE than none — joesinvestments' boot 9 regressed -10% because partial configs snap non-decode shapes to wrong configs.
6. clear_thinking: true should be the default for chat workloads — Z-AI's own model card says so. Thinking replays prior-turn reasoning into context, burning tokens.
7. Never set small max_tokens with thinking on — the budget burns inside the think block with empty content output.

From tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark

1. KV sizing doctrine: grow pool until ~8-10GB residual per node, gate behind concurrent 20K prefills.
2. GB10 "phantom backing": reservations can succeed then die on first touch in warmup. Always verify with actual requests.
3. Block size 2304 is required for DeepGEMM arch-12 64-entry pool pages.
4. FlashInfer ≥0.6.18 mandatory on sm121 — FA2 MLA produces NaN below this version.
5. Cache flusher sidecar during weight load prevents GB10 driver vs page cache conflict.
6. Reboot nodes after many boot cycles — driver alloc-pool degrades silently.

🙏 Credits

tonyd2wild — DFlash2 Overlay Patches

• Repo: tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark
• Built the vLLM DFlash2 overlay (4 patch scripts) on top of radixark/vllm-glm53-flash:sm121-v8. Patches register DFlash2DraftModel in the model registry, add Eagle3 aux hidden-state capture to GLM-5.3's model code, fix KV page layout for the drafter, and wire the drafter into the spec decode engine.

incoai — DFlash2 Draft Model

• Model: incoai/GLM-5.3-Flash-DFlash2
• Trained the 2.2GB DFlash2 draft model (5-layer Qwen3, hidden 4096, selector_rank 256, selector_top_k 16). Non-causal attention with sliding window 2048. CC BY-NC-ND research eval license.

z-lab — DFlash2 Reference Implementation

• Repo: z-lab/dflash
• Original DFlash2 block-diffusion implementation and research.

radixark — Base vLLM Image

• Image: radixark/vllm-glm53-flash:sm121-v8
• The sm121-v8 base vLLM image with GLM-5.3-Flash support, FlashInfer, and DeepGEMM for GB10.

zai-org — GLM-5.3-Flash Model

• Model: zai-org/GLM-5.3-Flash-NVFP4
• The original GLM-5.3-Flash model in NVFP4 quantization (182GB).

joesinvestments — SGLang FP8 Reference Deployment

• Repo: joesinvestments/GLM-5.3-Flash-FP8-4x-DGX-Spark
• Independent 4× DGX Spark TP4 deployment of GLM-5.3-Flash using SGLang (FP8 weights, DFlash2 drafter). Their DFlash2 numbers on SGLang (48.1 tok/s structured, 32.4 tok/s code) provided a valuable cross-engine comparison. Their LEDGER.md is a masterclass in deployment documentation — every flag's provenance, every failed boot, every measurement ruler. Key learnings: DFlash2 acceptance warms slowly, chunked-prefill 2048 helps SGLang, instanttensor is unreliable in multi-node, partial MoE tuning is worse than none.

local-inference-lab — Benchmark Tool

• Repo: local-inference-lab/llm-inference-bench
• The benchmark tool used for all measurements (v0.4.29). Supports sustained decode, coding peak, prefill, and burst modes with per-position acceptance tracking.

License

MIT. Model weights are not redistributed here — download from zai-org/GLM-5.3-Flash-NVFP4 and incoai/GLM-5.3-Flash-DFlash2.