Instructions to use canada-quant/glm-5.3-w4a16-mtp with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use canada-quant/glm-5.3-w4a16-mtp with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="canada-quant/glm-5.3-w4a16-mtp")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("canada-quant/glm-5.3-w4a16-mtp")
model = AutoModelForMultimodalLM.from_pretrained("canada-quant/glm-5.3-w4a16-mtp", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use canada-quant/glm-5.3-w4a16-mtp with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "canada-quant/glm-5.3-w4a16-mtp"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "canada-quant/glm-5.3-w4a16-mtp",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/canada-quant/glm-5.3-w4a16-mtp

SGLang

How to use canada-quant/glm-5.3-w4a16-mtp with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "canada-quant/glm-5.3-w4a16-mtp" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "canada-quant/glm-5.3-w4a16-mtp",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "canada-quant/glm-5.3-w4a16-mtp" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "canada-quant/glm-5.3-w4a16-mtp",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use canada-quant/glm-5.3-w4a16-mtp with Docker Model Runner:
```
docker model run hf.co/canada-quant/glm-5.3-w4a16-mtp
```

GLM-5.3-Flash — W4A16 (INT4) + BF16 MTP

INT4 weight-only quantization of zai-org/GLM-5.3-Flash with the BF16 multi-token-prediction (MTP) draft head preserved for speculative decoding. 177.7 GiB (−70%) — serves on 4× H100/H200, 4× RTX PRO 6000, or 2× DGX Spark at full 1M context.

Headline	Measured
Weight footprint	177.7 GiB vs ~599 GiB BF16 (−70%)
Minimum hardware	4× H100 · 2× H200 (fit-estimated) · 4× RTX PRO 6000 · 2× DGX Spark
Context on 2× DGX Spark	1,048,576 tokens served (1.36M-token KV pool); the NVFP4 comparison checkpoint OOM'd 9/9 boots at any context on this hardware
Throughput vs NVFP4 (2× Spark)	+53% seq1 / +101% seq6 @262K
Throughput vs NVFP4 (H100, TP=4)	wins c=1; edges c=8 vs NVFP4's banked config; NVFP4 wins saturated c=32
AIME 2026 (max thinking, no truncation)	86.7% (26/30)
GSM8K	0.97 — parity with the FP8 baseline

All benchmark numbers measured on live serving stacks; the one fit-estimated hardware row is labeled. Results as of 2026-08-31; in-flight arms are labeled.

Why this quant exists

GLM-5.3-Flash in BF16 needs eight 80 GB GPUs per replica. This artifact cuts weights to 177.7 GiB so it serves on four datacenter GPUs — or a dual DGX Spark desktop pair at full 1M context — while keeping everything quality-sensitive at full precision. Only the 36,288 routed-expert GEMMs are INT4 (symmetric, group-128, GPTQ); attention, router, shared experts, embeddings, the BF16 vision tower, and the BF16 MTP draft head (+40–50% single-stream) ride unquantized, with canonical FP32 parameters preserved verbatim. It is not a new model — all capability comes from the base model.

Hardware

Config	GPUs	Context	Notes
H100/H200 (SM90), TP=4	4× 80 GB	262K, bf16 KV	bench-validated recipe
H200 (SM90), TP=2	2× 141 GB	~64–128K, fp8 KV	fit-estimated (not rig-measured)
RTX PRO 6000 (SM120), TP=4	4× 96 GB	262K, fp8 KV	bench-validated
DGX Spark GB10 (SM121), TP=2	2× 128 GB UMA	1M, fp8 KV	DFlash2 stack — recipe below
B300 (SM103)	8×	—	quantization node (transformers smoke-tested)

Benchmarks

Head-to-head against the strongest community NVFP4 quant (LibertAIDAI/GLM-5.3-Flash-NVFP4), identical flags/prompts/hardware per row. Throughput = output tok/s, thinking ON.

H100/H200 (SM90), TP=4, MTP both sides, isl/osl 8192/1024

concurrency	W4A16 MTP N=2 (this)	NVFP4 MTP N=2	NVFP4 MTP N=5 (its best c32 config)
1	183.89	180.96	175.44
8	249.94	251.14	220.05
32	689.46	770.66	1018.13
MTP acceptance	52–55%	~52%	~30%

W4A16 leads the interactive regime (+4.8% c=1, +13.6% c=8 vs NVFP4's banked N=5 config) on much higher MTP acceptance. NVFP4 leads saturated c=32 — its kernel keeps ~12% pure-kernel edge at full batch. Closing c=32 is active work.

RTX PRO 6000 (SM120, Blackwell), TP=4, matched MTP N=2

concurrency	W4A16 (this)	NVFP4	Δ
1	109.90	109.34	+0.5%
8	318.54	319.46	−0.3%
32	534.37	530.75	+0.7%

Parity within noise (±0.7%).

2× DGX Spark GB10 (SM121), TP=2, DFlash2 block-diffusion spec-decode

8K prompts, 256-token completions, aggregate tok/s:

config	seq1	seq2	seq4	seq6
W4A16+DFlash2 @262K (this)	33.29	42.01	80.98	121.81
NVFP4-MTP4 reference	21.8	41	66	60.5
Δ	+53%	+2.4%	+23%	+101%
W4A16+DFlash2 @1M serve	32.99	35.61	59.61	67.05

256K-token single prompt on the 1M serve: 30.85 tok/s decode (ttft 5.8 s). The comparison is also structural: the LibertAI NVFP4 checkpoint (90.67 GiB/node weights vs ~30 for W4A16) OOM'd all 9 boot attempts across 1M/300K/262K on the 121 GiB unified-memory nodes (dmesg-verified) — it cannot serve at any usable context on 2× Spark.

Quality

AIME 2026 (30 problems, max thinking, no truncation, 131,072-token budget, DFlash2 stack on 2× Spark): 26/30 = 86.7%. All 4 misses are budget-exhaustion (finish_reason=length at exactly 131,072 tokens), not wrong answers; a 1M-budget rescue re-run and a 4-epoch n=120 arm are in flight.

GSM8K: parity — 0.9704–0.9750 across all three quants on both x86 rigs (quant-insensitive). GPQA-Diamond n=30 @131k: 0.8586 vs 0.8687 NVFP4 (SM90), 0.8586 vs 0.8737 (SM120) — within noise.

AIME 2025 n=120 — the one honest deficit. At 4 epochs × 30, max thinking, 131,072 budget, NVFP4 leads: 0.9000 vs 0.7833 (SM90, 2.5σ) and 0.9000 vs 0.8000 (SM120, 2.19σ). Per-sample audit of all 480 completions: genuine math errors at parity — the gap is thinking-budget exhaustion. W4A16's chains run longer (NVFP4's tail ≥2.1× lighter) and hit the cap unanswered ~4× more often (15–17/120 vs 3–4/120). Spec-decode is exonerated; frequency-penalty mitigation was rejected; budget-doubling is a partial mitigation (rescues convergent chains, not loop-attractor chains). Answer-commit/loop-salvage is the active lever. (AIME25 ran via lm-eval, which has no aime2026 task; the 2026 arm uses a custom harness — matched AIME26 arms on the x86 rigs are in flight.)

Serving

The same checkpoint serves on every NVIDIA architecture from Hopper through Blackwell. Each recipe is the exact config the benchmarks ran.

SM90 — H100 / H200 (recommended: TP=4)

Image vllm/vllm-openai:glm53-flash-x86_64-cu130 (CUDA 13.0). Gotcha: always pass --max-num-seqs ≤512 — the default 1024 exceeds this hybrid linear-attention model's 512 Mamba/KDA-state cache blocks.

docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 512 \
    --block-size 64 --gpu-memory-utilization 0.92 --no-enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8000

MTP num_speculative_tokens: 2 is the sweet spot on Hopper (52–55% acceptance, best c1/c8); N=5 helps only at c=32. TP=8 works too.

SM120 — RTX PRO 6000 (Blackwell)

Image cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 (SM120-patched build).

docker run --gpus '"device=4,5,6,7"' --ipc=host --network=host --rm \
  -v /models:/models cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1 \
  vllm serve /models/glm53-flash-w4a16-mtp \
    --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel \
    --max-model-len 262144 --max-num-seqs 64 --max-num-batched-tokens 8192 \
    --kv-cache-dtype fp8 --gpu-memory-utilization 0.92 --enable-prefix-caching \
    --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code --port 8001

KV fp8 is required at 262K on 96 GB cards; --max-num-seqs 64 is bench-validated (48/36 fallback ladder); MTP N=2 matches NVFP4 to ±0.7%.

SM121 — 2× DGX Spark GB10 (desktop, 1M context)

Full 1,048,576-token context on two Sparks over RoCE (TP=2) with the DFlash2 block-diffusion drafter:

Image radixark/vllm-glm53-flash:sm121-v11-dflash2; drafter GLM-5.3-Flash-DFlash2 (block 8, selector_rank 256, top_k 16, layers [5,14,24,33,42]).
MAX_MODEL_LEN=1048576 KV_CACHE_MEM=9663676416 → KV fp8_e4m3 pool 1,360,420 tokens (1.30× a full 1M request); staging @262K = MAX_MODEL_LEN=262144 KV_CACHE_MEM=3221225472.
--max-num-seqs 6 --block-size 2304 --enforce-eager; 7 speculative tokens (any other count boot-wedges).
Worker rank first, 25 s, then head rank; cold boot ≈6–8 min.
NCCL on the RoCE switch path (97.98 Gbps line-rate; direct-attach measured worse).
VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3600 (cold JIT otherwise kills the boot).
Keep single prompts ≤ ~310K tokens (larger wedged a host twice).

Docker images

Architecture	Image
SM90 (H100/H200)	`vllm/vllm-openai:glm53-flash-x86_64-cu130`
SM120 (RTX PRO 6000)	`cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1`
SM121 (DGX Spark)	`radixark/vllm-glm53-flash:sm121-v11-dflash2`

Quick start

# 1. Download the weights (~178 GiB)
huggingface-cli download canada-quant/glm-5.3-w4a16-mtp --local-dir /models/glm53-flash-w4a16-mtp

# 2. Serve (SM90 example — see recipes above)
docker run --gpus '"device=0,1,2,3"' --ipc=host --network=host --rm \
  -v /models:/models vllm/vllm-openai:glm53-flash-x86_64-cu130 \
  vllm serve /models/glm53-flash-w4a16-mtp --served-model-name glm53-w4 \
    --tensor-parallel-size 4 --enable-expert-parallel --max-num-seqs 512 \
    --max-model-len 262144 --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
    --reasoning-parser glm45 --tool-call-parser glm47 --enable-auto-tool-choice \
    --trust-remote-code

# 3. Call it (OpenAI-compatible)
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "glm53-w4",
  "messages": [{"role": "user", "content": "Prove there are infinitely many primes."}]
}'

Details

Field	Value
Base model	zai-org/GLM-5.3-Flash (BF16)
Architecture	`Glm5NextForConditionalGeneration` (`glm5_next`) — 45 decoder layers (+ MTP layer 45), hidden 4096, 288 routed experts (top-8) + 1 shared, MoE-intermediate 2048, KDA + DSA attention, 24-block vision tower, vocab 154,880
Weight quantization	W4A16, INT4, symmetric, group-size 128 (GPTQ, compressed-tensors `pack-quantized`), routed experts only
Quantized tensors	36,288 (42 MoE layers × 288 experts × 3 GEMMs) — verified exact
Kept in BF16	attention (KDA + DSA incl. indexer), dense prefix (layers 0–2), shared experts, router/gate, mHC tensors, embeddings, lm_head, norms, vision tower (348 keys), MTP layer 45 (889 keys)
Canonical FP32 preserved	`A_log`, `dt_bias`, `e_score_correction_bias`, `hc_*` — verbatim from source
Calibration	256 samples × 4096 tokens, in-distribution chat/code mix (`glm53_mix`), sequential per-layer GPTQ
Size	177.7 GiB (from ~599 GiB BF16)
License	MIT (inherited from the base model)

vLLM serving note: the checkpoint's quantization_config.ignore already excludes the vision tower and re:model\.language_model\.layers\.45\..*, so the MTP head loads BF16 for speculative decoding out of the box.

Validation

Static gates at build time (2026-08-27), all PASS:

exactly 36,288 packed tensors; zero compression outside routed experts;
vision key-set 348/348 identical to source, zero compressed, all dtypes match;
MTP layer present (889 keys), index rewrite purely additive;
dtype drift vs source: zero (282 pipeline-drifted tensors restored verbatim);
expert scale audit: no zero/collapsed scales.

Quantized on 8× NVIDIA B300, 2026-08-27. B300 smoke tests (transformers ≥5.16): loads as Glm5NextForConditionalGeneration (~110 s); text generation coherent; image captioning accurate.

Vision-quality note: the vision tower is BF16-passthrough but was not covered by the text-only calibration set. Image smoke tests pass; a vision eval leg (MMMU-class + OCR) is queued.

License

MIT, inherited from the base model. Follow the base model's usage terms.

Built, benchmarked, and documented with the Digby.ai coding harness, developed by CQL.ca.

Downloads last month: 7

Safetensors

Model size

50B params

Tensor type

F32

I32

BF16

Model tree for canada-quant/glm-5.3-w4a16-mtp

Base model

zai-org/GLM-5.3-Flash

Quantized

(74)

this model