EigenLabs/GLM-5.3-Flash-MLX-NVFP4-mtp

GLM-5.3-Flash — MLX NVFP4 hybrid with embedded MTP

An Apple-silicon (MLX) packaging of Z.ai's GLM-5.3-Flash, built from NVIDIA's NVFP4 checkpoint nvidia/GLM-5.3-Flash-NVFP4. It targets the Darkbloom native glm5_next runtime. The text model, the vision tower and the multi-token prediction (MTP) layer are all in one indexed checkpoint: one download. The MTP head (layer 45, 2,641 arrays) ships in these shards, so no separate draft model or MTP file is needed.

Status: qualified for single-Mac serving on the Darkbloom network (catalog id glm-5.3-flash, 128K context, text, image and video input). Measured on one M5 Ultra (256 GB) with the Darkbloom native runtime, embedded MTP on by default: prefill 1,897 / 2,042 / 2,042 tok/s at 2.7K / 10.5K / 31.6K prompt tokens, about 1,660 tok/s at 128K; decode 62 / 59 tok/s for 256 / 1,024 tokens at one request (MTP acceptance about 0.6–0.8 per proposed token). Needle retrieval passes at 32K, 60K, 128K and 190K tokens. The record is docs/reports/2026-10-21-glm53-flash-single-mac-qualification.md in Layr-Labs/d-inference.

Formerly EigenLabs/GLM-5.3-Flash-MLX-4bit-mtp; the old name redirects here. This repository previously held a uniform affine 4-bit build (commit efb9557e38d2b6e6bbec1b514b19b529f3fef42e). That revision stays available by commit hash; the current files replace it.

What is in it

Part	Storage	Source
Routed experts, layers 3–44 (36,288 projections)	MLX nvfp4 (E2M1 values, FP8-E4M3 scale per 16, FP32 per-tensor scale)	NVIDIA's NVFP4 weights, repacked losslessly (bytes unchanged)
Dense MLP, layers 0–2 (9 projections)	MLX nvfp4	NVIDIA's NVFP4 weights, repacked losslessly
MTP layer 45 routed experts (864 projections)	MLX nvfp4	BF16 in NVIDIA's checkpoint; quantized here with MLX round-to-nearest nvfp4 so every routed expert has one format
Attention / linear attention / indexer projections, shared experts, lm_head, embeddings, MTP eh_proj (534 matrices)	MLX affine 4-bit, group 64	BF16 (NVIDIA keeps these in BF16); quantized here
Router, norms, hc_*, A_log, dt_bias, conv1d, indexer k_norm/kpool, vision tower (1075 tensors)	as stored (BF16/F32)	BF16 source, byte-identical

Routed experts are NVFP4 and the remaining matrices affine 4-bit; routers, norms, scalar and state parameters and the whole vision tower stay in source precision.

Field	Value
Architecture	glm5_next (Glm5NextForConditionalGeneration): 45 target layers (linear attention + DeepSeek sparse attention), 288 routed experts (8 active), mHC, vision tower, 1 MTP layer
Safetensor files	181,709,576,492 bytes (169.23 GiB) in 35 shards; index total_size (tensor payload) 181,693,841,436
Indexed arrays	114,160, including 2,641 MTP and 347 vision entries
Weight bytes read per decoded token (B=1, 8 of 288 experts, MTP excluded)	9.55 GB (the previous affine build: 9.55 GB)
License	MIT — see License and attribution

NVFP4 and affine g64 both cost 4.5 bits per weight, so decode bandwidth matches the earlier all-affine build while the experts keep NVIDIA's calibrated NVFP4 values. NVIDIA reports BF16-level accuracy for its NVFP4 checkpoint (for example GPQA Diamond 0.9211 vs 0.9217 BF16, MMMU-Pro 0.763 vs 0.769). Those numbers are for NVIDIA's checkpoint on NVIDIA GPUs; the extra 4-bit quantization of the non-expert matrices here is ours and has not yet been measured end to end.

NVFP4 storage

For each NVFP4 module <base>:

Tensor	dtype	Shape	Meaning
<base>.weight	U32	[out, in/8]	E2M1 codes, MLX nvfp4 packing (element k at bits 4k of the little-endian word stream)
<base>.scales	U8	[out, in/16]	FP8 E4M3 block scales
<base>.weight_scale_2	F32	[]	per-tensor scale (ModelOpt weight_scale_2)

W = e2m1(code) × e4m3(scale) × weight_scale_2. NVIDIA's ModelOpt U8 packing and MLX's U32 packing are the same byte stream, so .weight, .scales and .weight_scale_2 are byte-for-byte NVIDIA's tensors (108,891 tensors, SHA-256 matched per tensor). NVIDIA's activation input_scale values (36,297) are GPU-only and dropped. config.json declares the default {"bits": 4, "group_size": 64, "mode": "affine"} and one {"bits": 4, "group_size": 16, "mode": "nvfp4"} override per NVFP4 module.

Note for other MLX loaders: mx.dequantize(..., mode="nvfp4") without global_scale returns e2m1 × e4m3; multiply by weight_scale_2. MLX's own global_scale argument means amax (the kernel multiplies by global_scale / 2688), which is not the stored quantity.

Affine non-expert rounding

The non-expert matrices use the same deterministic CPU recipe as the earlier EigenLabs affine build: for each group of 64 weights, MLX round-to-nearest plus 74 clipped-range candidates with 2 least-squares scale/bias refinements, lowest squared error wins (RTN on ties). The sampled outputs are byte-identical to the previous build's tensors. Relative Frobenius error against BF16 (weight statistics only, not a quality measurement):

Class	Matrices	MSE search	RTN reference
kda_main	136	0.0815	0.0927
mla	60	0.0810	0.0928
shared	129	0.0820	0.0943
kda_gate	170	0.0811	0.0913
indexer	36	0.0826	0.0941
eh_proj	1	0.0863	0.0923
embed	1	0.0807	0.0909
lm_head	1	0.0813	0.0915

Multi-token prediction (MTP)

• Embedded: the trained MTP layer is in this checkpoint as model.language_model.layers.45.* (num_nextn_predict_layers: 1 in config.json): enorm, hnorm, eh_proj, shared_head.norm, sparse attention with indexer, router, 288 routed experts and a shared expert. It reuses the target model's embedding and output head. There is no separate draft model to download.
• Darkbloom: the native glm5_next runtime serves GLM-5.3-Flash with MTP on by default; no flags or environment variables are needed. Proposed tokens are always verified by the target model before they are emitted.
• Other MLX runtimes: stock mlx-lm / mlx-vlm have not been evaluated with this checkpoint and are not expected to load it unmodified (per-expert tensor names instead of stacked switch_mlp, NVFP4 per-tensor scales, MTP stored as layer 45). Most generic loaders drop or ignore MTP weights.
• vLLM / SGLang (NVIDIA GPUs): use NVIDIA's original checkpoint nvidia/GLM-5.3-Flash-NVFP4, not this MLX repack, and enable MTP with --speculative-config '{"method":"mtp","num_speculative_tokens":N}' (pick a small N such as 1–3).

MTP acceptance on these weights has not been measured yet. The MTP experts were re-quantized from BF16 (RTN nvfp4, mean relative weight error 0.0946), so acceptance may differ slightly from the BF16 MTP layer.

Chat template

chat_template.jinja is Z.ai's template from zai-org/GLM-5.3-Flash-BF16 at a5b45eb4 with four edits for the runtime's Swift Jinja (lines 81, 85, 103, 234: x.0 → x[0]); under Jinja2 the two compile to identical code.

	SHA-256
Template in this repository	46b2e2137e2759355b3e0e3e90cf4bd172b568437a75588af5c457cc8c095b2e
Z.ai template (BF16 repo a5b45eb4)	0c4099f3382d6c92700dfb99725025360966fd73032f0ecf32377c0d9e6309c5
Template in NVIDIA's NVFP4 repo (an older Z.ai revision, before the "early break" fix)	34d5ee66b12fa6446cdae131c352b8f68cd85369e0e6fda115583805fada3891

Tokenizer, tokenizer configuration, processor and generation configuration are byte-identical to zai-org/GLM-5.3-Flash-BF16@a5b45eb4 (the tokenizer and processor files are also identical to NVIDIA's). config.json is Z.ai's configuration plus quantization / quantization_config.

Download

from huggingface_hub import snapshot_download

model_path = snapshot_download(
    repo_id="EigenLabs/GLM-5.3-Flash-MLX-NVFP4-mtp",
    revision="<full commit hash>",
)

Pin a full commit hash for reproducible deployments.

Validation and limitations

• Source integrity: all 33 NVIDIA weight shards were verified against their published LFS SHA-256 at da920bb0b9f4a06727223a349e55468e38352348 before conversion; metadata files were verified against a pinned API snapshot.
• Lossless NVFP4 repack: per-tensor SHA-256 of every repacked tensor equals NVIDIA's. For 35 sampled modules (routed experts across layers and projections, all dense MLP projections, MTP experts) MLX 0.32.2 dequantize(mode="nvfp4") and a NumPy implementation of ModelOpt's formula agree bit for bit.
• Semantics check: NVIDIA's NVFP4 dense-MLP weights dequantize to within 0.090–0.094 relative error of Z.ai's BF16 weights (a swapped nibble order would give 1.41+).
• BF16 agreement: every BF16 non-expert tensor in NVIDIA's checkpoint equals Z.ai's BF16 weights (byte-identical; NVIDIA's BF16 copies of hc_*_base/scale are exact BF16 roundings of the F32 originals, which are kept here; NVIDIA's F32 conv1d weights are exact upcasts of the BF16 originals, kept as BF16). Sampled MTP expert weights are byte-identical to Z.ai's BF16 weights.
• Native load and serving (2026-10-21, one M5 Ultra 256 GB): the native loader verifies every manifest file before and after load; greedy output, tool calls (tool_choice: auto), image input (16–8,000 visual tokens per image, known-content OCR, counting and shapes) and video input (2 FPS sampling; known-content motion direction, colour order and frame counters) pass the Darkbloom qualification checks; teacher-forced perplexity on a 15,909-token panel is 7.314 under the serving policy.

Not measured: KL divergence vs BF16 and broad task benchmarks. These checks do not establish BF16 equivalence or quality on any workload.

License and attribution

• Original model and training: Z.ai, released under the MIT License. LICENSE is Z.ai's MIT license, copied byte-for-byte; keep it with any copy or substantial portion of these weights.
• NVFP4 quantization of the routed experts and dense MLPs: NVIDIA, with TensorRT Model Optimizer (ModelOpt 0.47, recipe nvfp4_experts_dense_mlp-kv_fp8_cast), published as nvidia/GLM-5.3-Flash-NVFP4 under the MIT license ("ready for commercial or non-commercial use"). NVIDIA states the model is not owned or developed by NVIDIA (third-party model).
• MLX packaging, non-expert 4-bit quantization and MTP-expert re-quantization: EigenLabs. This is not an official Z.ai or NVIDIA release.

artifact-provenance.json and SHA256SUMS.json record provenance and file hashes; nonexpert-quantization.json has per-tensor statistics for the affine matrices.