cfontes/glm53-spark-tp4

GLM-5.3 on four DGX Sparks - serving kit + incremental KV prefix cache

A reproducible recipe for serving zai-org/GLM-5.3 - an EXL3 MixedK 3.38 bpw quantization of Z.ai's 753B MoE - on four NVIDIA DGX Sparks at a 1,048,576-token context, plus the incremental KV prefix cache that makes long agentic sessions affordable.

Weights: vcruz305/GLM-5.3-EXL3-3.38bpw (public, 297.1 GiB, 58 shards). This repository is the serving kit and the cache patch series around it.

Everything is in this repository: the weights, the patched TensorFold engine, the DFlash2 drafter, and the launcher. No other download, no other repo.

# 4x DGX Spark, same user, passwordless ssh from rank 0 to the other three
git clone https://huggingface.co/cfontes/glm53-spark-tp4
cd glm53-spark-tp4
bash fetch_weights.sh        # verifies every shard the index names is present AND complete
bash serve.sh                # TP4 + DCP4 + DFlash2 + prefix cache; READY in ~14 min
bash verify.sh               # checks a completion, not just /health

fetch_weights.sh parses each shard's safetensors header, so a truncated shard or an LFS pointer fails there rather than mid-boot. The endpoint is http://<rank0>:8300/v1, served model id glm-5.3-exl3 (both overridable; see DEPLOY.md).

---

Why this exists

Four Sparks give 512 GB of unified memory. GLM-5.3 at BF16 does not fit; at 3.38 bpw it does, and it leaves room for a 1M-token KV cache. The hard part was never the weights - it was making long-context serving fast, and making multi-turn work without re-prefilling the whole conversation every turn.

That second part is what kv_cache/ adds.

Performance

Decode throughput, 320 new tokens, temperature 0, n=6, p50. Same lane: 4 Sparks, TP4, DCP=4, DFlash2 drafting.

Workload	tok/s
Prose	55.7
Code	69.2
Structured	73.6

Method: the banked bench3.sh against /v1/chat/completions. The model page's own figures (up to 69 tok/s peak, 41.1 tok/s mean over 6 real prompts) come from a different harness with 512 new tokens and long contexts. Do not compare the two tables.

Incremental prefix cache - the agentic win

Six-turn agent conversation appending ~2.7k tokens per turn.

Turn	Prompt tokens	Cached	Per-turn wall	No-cache baseline	Speedup
1	2,670	0	13.0 s	15.5 s	1.2x
3	7,974	5,120	17.2 s	44.5 s	2.6x
5	13,278	10,240	17.7 s	72.0 s	4.1x
6	15,930	12,288	19.4 s	85.3 s	4.4x

Without a cache, per-turn cost grows linearly with the prompt, so a session is O(n^2). With it, per-turn cost is flat - prefill is bounded by the new tokens, not the history. A 19,215-token prompt extended by one word:

	Wall
cold	46.4 s
reused (19,212 cached)	0.41 s

Quality

Benchmark	Result	Method
GSM8K, full 1,319	96.7% (1271 / 1315 committed)	runeval.py, T=0
GPQA-Diamond, stratified 50	87.8% (36 / 41 committed)	T=0.6, thinking_budget=12000

Read the denominators. GPQA answers 41 of 50; the other 9 are the model degenerating into repetition loops on hard physics/chemistry items, caught by the loop detector rather than silently dropped. The quant pack's own held-out fidelity (KL 0.0948, top-1 agreement 92.98%, perplexity +2.1%) is measured by the pack author; see its card.

Hardware and topology

4x NVIDIA DGX Spark (GB10), 128 GB unified memory each	512 GB total
Tensor parallel	4
Decode context parallel	4 (position p lives on rank p % 4)
Context	1,048,576 tokens
KV cache	fp8 latent, DCP-sharded
Drafting	DFlash2, depth up to 13, top2 relaxed acceptance
Interconnect	one QSFP switch, RoCE

The KV prefix cache in one paragraph

kv_cache/ reuses the KV of a conversation's longest common prefix and prefills only the new suffix into the same buffer. The subtle part is that chunking must be a function of position only: the original code sized prompt chunks from the total prompt length, so two requests of different length computed the same prefix rows with different tiling, and the inherited KV was not what a cold run would have produced. Fixed-grid chunking plus a bound at the canonical prefill frontier - never reuse decode-written rows - makes reuse exact. Design notes, the seven patches and the acceptance tests are in kv_cache/.

Disable anytime: TF_GLM53_PREFIX_CACHE=0 restores the byte-identical pre-cache path (sha=e38c30fdc5e4).

Caveats - read these

• Single request at a time. The lane is single-flight; concurrent requests queue and cost 1.5-2.9x more per item. Do not point a parallel agent fleet at one lane.
• "draft": false is pathological on this build (~9x slower, with a misleading tok/s). Leave drafting on.
• Reasoning workloads need temperature >= 0.6 and a thinking_budget. At T=0 the model degenerates into verbatim repetition on long chains; with no budget it can think to the token cap and never answer.
• The weight pack has no MTP layer; a separate DFlash2 drafter checkpoint is required for drafting.
• The bundled drafter is CC BY-NC-ND 4.0 upstream. drafter/ is a recalibrated derivative of incoai/GLM-5.3-DFlash2. Do not redistribute it, and do not use it commercially; see NOTICE.
• Everything here was measured on one 4-Spark lane. Treat the numbers as that lane's, not a guarantee.

Credits

• Model: Z.ai - GLM-5.3. License glm-5.3.
• Quantization: the SAGE MixedK 3.38 bpw pack by its author - vcruz305/GLM-5.3-EXL3-3.38bpw.
• Runtime: TensorFold, Apache-2.0. The GLM-5.3 tensor-parallel engine is PR #159 by @drowzeys.
• Original Spark recipe: vcruz305/GLM-5.3-EXL3-DGX-Spark-recipe.
• KV prefix cache, the launcher, and the serving/benchmark work in kv_cache/: this repository.

License

The weights are governed by the GLM-5.3 license. The runtime is Apache-2.0. The patches and scripts here are released under Apache-2.0. The drafter is a derivative of incoai/GLM-5.3-DFlash2 (CC BY-NC-ND 4.0) and is included only for reproduction; it is not yours to redistribute. Nothing in this repository relicenses the base model, the quant pack, or the drafter.