GLM-5.3-Flash-Alis-MLX-8bit

Part of the GLM-5.3-Flash · Alis MLX collection.

Status: public. Weights and card are the verified 8-bit teacher pack (2026-08-29). Image→text generate is proven on the 4-bit sibling via the campaign mlx-vlm path. Stock mlx-lm is still text-only.

Apple Silicon (MLX) 8-bit quantization of zai-org/GLM-5.3-Flash — a 320B-parameter (~18B active) Mixture-of-Experts model with a hybrid KDA linear-attention / DSA sparse-attention stack (glm5_next), 288 routed experts, and a 1M-token context window.

This is the fidelity tier of a three-build set (8-bit / 6-bit / 4-bit): every quantized tensor is 8-bit affine, group size 64, and every architecturally sensitive tensor group is kept at source precision (see Recipe). 346.7 GB on disk (322.96 GiB), ~8.63 effective bits/param over the 321.3B stored parameters. It also served as the teacher for the 4-bit chain's paired-KL evaluation and DWQ pass (the DWQ was reverted on held-out regression; see the 4-bit card).

⚠️ Requires a ported runtime. glm5_next is not in stock mlx-lm (as of 2026-08-28) — loading there fails with an unsupported-model-type error. This set was built and verified with a drop-in port (mlx_lm/models/glm5_next.py, pinned md5 695c6376914ec0bd958f72b1044c885c, plus hyper_connection.py).

• Campaign repo (port, conversion chain, receipts): https://github.com/avlp12/local-llm-serving
• mlx-vlm processor PR: https://github.com/Blaizzy/mlx-vlm/pull/2091 (architecture already in #2030)
• mlx-lm fork branch + upstream PR: pending — glm5_next.py is still untracked on the local mlx-lm tree.

At a glance

Build	Disk	Effective bits/param	Status
8-bit (this repo)	346.7 GB (322.96 GiB)	~8.63	strict-load + Metal forward + generation smoke ✓
6-bit	268.1 GB (249.66 GiB)	~6.67	strict-load (recipe audit) ✓
4-bit	189.8 GB (176.8 GiB)	~4.73	final = RTN pack — QUASAR-init/AWQ/ALIS-DWQ all rejected under pre-registered gates; PROFILE-MATCH q4 + Metal forward ✓

Source pin for the whole set: zai-org/GLM-5.3-Flash revision 84c6a6aa9497188e15a635ba793b0f95a79b1033 (FP8 e4m3 with 128×128 block scales).

Architecture

GLM-5.3-Flash (Glm5NextForConditionalGeneration, model_type: glm5_next) is a hybrid-attention MoE:

• 45 text-backbone layers, hidden 4096, plus one MTP layer (layer 45) declared only in the weight index.
• 34 KDA layers (every layer with ℓ % 4 != 3): Kimi-Delta-Attention linear attention — short q/k/v conv, delta-rule recurrent state, learned forget gate (gate_lower_bound = −5). Fixed-size state; no growing KV cache on these layers.
• 11 MLA + DSA layers (ℓ = 3, 7, …, 43): DeepSeek-style compressed-latent attention (kv_lora_rank 512, q_lora_rank 1536, head_dim 256, NoPE — qk_rope_head_dim = 0) with a DeepSeek-Sparse-Attention lightning indexer (index_topk 2048) and IndexPool key pooling (4→1, select_k 512).
• MoE: 288 routed experts, top-8, sigmoid routing with an fp32 correction bias, routed scaling 2.5, plus 1 shared expert (moe_intermediate_size 2048). Layers 0–2 are dense (intermediate_size 12288). SwiGLU with a pre-activation clamp (swiglu_limit 10).
• mHC: multi-stream hyper-connections (4 residual streams, collapsed by unweighted mean before the head).
• Untied embedding and lm_head; vocab 154,880; context max_position_embeddings 1,048,576.
• Generation defaults: temperature 1.0, top_p 0.95.

Long-context memory is dominated by the 11+1 MLA layers' compressed latent cache; the 34 KDA layers carry fixed-size recurrent state — which is what makes the 1M window cheap on this architecture.

Vision: image→text generate proven on the 4-bit pack (custom runtime)

The upstream checkpoint is a VLM (glm5_next_vision tower). This repo preserves all 347 model.visual.* tensors byte-identical to upstream (bf16/fp32 passthrough, 1.13 GB — never quantized), so nothing was destroyed in conversion.

• Tower smoke: 347/347 tensors loaded; a 448×448 image maps to a (256, 4096) feature sequence (q8 visual shard, 2026-08-27).
• Full image→text generate: proven on the 4-bit RTN pack of this set (Epsilon, 2026-08-29). A Catalina Coast photograph was described as a coastal/island scene with dark water and rocky shoreline; a text-only control answered "Paris".
• Not in stock mlx-lm or stock mlx-vlm yet. python -m mlx_lm generate still drops model.visual.*. Image input uses the campaign mlx_vlm/models/glm5_next package (run_vlm_generate.py in avlp12/local-llm-serving), not mlx_vlm.load().
• Video is implemented in the port and has not been run on real video.
• Decode on that VLM path measured 0.55 tok/s (greedy) — not the text-serving path.

The vendor MTP layer (layer 45) is likewise fully preserved (quantized per the same recipe; its eh_proj/enorm/hnorm/shared_head.norm stay bf16). The current port ignores it at load — self-speculative decoding is future work, not a shipped feature of this build.

Recipe

MLX affine quantization, group size 64 throughout (fp16 scales+biases ⇒ ~+0.5 effective bpw over the nominal bit-width). Bits are spent by sensitivity; discrete control paths are never quantized:

Component	This build	Why
Routed experts gate/up (86 tensors) + down (43)	8-bit g64	~97% of parameters — the bulk
Shared expert + dense L0–2 MLP (129)	8-bit g64	per-token path
MLA/DSA attention projections (36)	8-bit g64	12 layers incl. MTP
kv_a_proj_with_mqa (12)	8-bit g64	latent chokepoint
embed_tokens / lm_head (2)	8-bit g64	distribution-sensitive
Router mlp.gate (43)	8-bit g64 (QuantizedMoEGate, fp32 e_score_correction_bias kept alongside)	kimi / house convention: quantize the projection, keep the bias exact
Skipped — source precision (1040 tensors)	bf16 (conv/A_log/dt_bias fp32)	see below

Held at source precision in every tier of this set, deliberately:

• all KDA attention tensors — q/k/v/o projections, q/k conv kernels, forget-gate and output-gate projections, b_proj, A_log, dt_bias, o_norm. The upstream FP8 release itself excluded every KDA tensor from FP8 (no weight_scale_inv), and recurrent-state parameters gate a state that carries the whole sequence — cheap in bytes, high in leverage.
• DSA lightning indexer — drives discrete top-k selection.
• kv_b_proj (stored split as embed_q / unembed_out) — upstream left it BF16.
• mHC mixing tensors and every RMSNorm.
• MTP extras (eh_proj, enorm, hnorm, shared_head.norm).
• the entire vision tower (byte-identical passthrough).

The skip set costs ~9 GB versus quantizing it — noise next to ~300 GB of experts, and it is the quality knob the upstream FP8 release already voted for.

Verified allocation (strict-load audit of the shipped config.json against the actual tensors): skip=1040, expert:8=86, expert_down:8=43, shared:8=129, router:8=43, kv_a:8=12, attn:8=36, head:8=2, other:8=9 → PROFILE-MATCH q8. 2479 tensors in 177 shards; index total_size 346,669,456,248 bytes.

Verification

Run on an M3 Ultra 512 GB (build host), 2026-08-27, with the pinned port:

Gate	Result
Strict lazy load + per-tensor recipe audit	PROFILE-MATCH q8 (histogram above)
Single-token Metal forward	OK — logits (1, 1, 154880), finite; 45.97 s cold load+forward
Tokenizer suite (154,856 entries; specials, chat template incl. tools, GLM tool XML, encode/decode roundtrips)	PASS
Generation smoke — factual	PASS ("Paris")
Generation smoke — Korean	PASS ("서울")
Generation smoke — long-context needle (~3.8K tok)	PASS ("PALANTIR-7X" retrieved)
Generation smoke — code	judge-fail on truncation only: the 192-token cap landed inside the <think> block; reasoning was coherent, no repetition pathology

Two port defects were found and fixed by this real-weight verification (both in the pinned port): the router leaf is not an nn.Linear (fixed with QuantizedMoEGate), and the indexer mistook mlx-lm's (S, S) causal mask for an HF (B, S) padding mask.

Not yet measured (planned before public flip): corpus perplexity / KL vs source, decode/prefill throughput, serving. No standard benchmarks (MMLU etc.) were run; none should be inferred.

Reproduction

FP8 → bf16 → affine, streaming, without ever materializing the ~640 GB bf16 model:

1. Pin zai-org/GLM-5.3-Flash @ 84c6a6aa9497188e15a635ba793b0f95a79b1033 (62 FP8 shards, 328.3 GB).
2. Block-dequant each FP8 e4m3 tensor with its 128×128 weight_scale_inv to bf16, one tensor at a time. The e4m3 decode is bit-exact against a NumPy reference, and the MLX dequant path matched NumPy at maxabs 0.0 on a real shard before the batch ran.
3. Sanitize to the MLX layout: strip the model.language_model. prefix, stack the 288 per-layer expert projections into single switch_mlp tensors, split kv_b_proj into embed_q (64, 512, 256) / unembed_out (64, 256, 512), keep e_score_correction_bias / conv kernels / dt_bias / A_log fp32, pass model.visual.* through byte-identical.
4. Quantize per-tensor with the recipe predicate above (affine g64), write complete-only with resume; finalize assembles the 177-shard index and verifies the shipped config.json quantization map against the actual tensors (the PROFILE-MATCH gate).

The source tree is read-only throughout. House conventions for recipe gating, receipts, and DWQ sequencing: avlp12/alis-dwq. Campaign scripts: avlp12/local-llm-serving (mlx-lm fork PR pending).

Usage

Until the fork branch/PR above is published, the port is a drop-in file pair (glm5_next.py, hyper_connection.py) on any current mlx-lm checkout:

# with the glm5_next port installed in an mlx-lm fork/checkout:
PYTHONPATH=/path/to/mlx-lm python -m mlx_lm generate \
  --model avlp12/GLM-5.3-Flash-Alis-MLX-8bit \
  --prompt "Explain how a hybrid linear/sparse attention stack changes long-context memory cost." \
  --max-tokens 1024

The chat template opens a <think> reasoning block by default — budget --max-tokens generously; a tight cap can be spent entirely inside the reasoning block (that is what the code-smoke judge-fail above was).

Hardware

Weights alone are 322.96 GiB: this tier is for 512 GB Apple Silicon (M3 Ultra class). For smaller machines, take the 6-bit (still 512 GB-class) or wait for the 4-bit.

Limitations

• Multimodal: image generate proven, not stock. Vision tensors are preserved (bf16, 347). End-to-end image→text was verified on the 4-bit pack via the campaign mlx-vlm package. Stock mlx-lm generate remains text-only. Video is untested.
• MTP not wired. Layer 45 ships in the weights; the port ignores it. No speculative decoding yet.
• No quality numbers yet. Strict-load, Metal forward, tokenizer, and generation smokes are the verification so far; PPL/KL and throughput are pending (this build served as the paired-KL teacher for the 4-bit chain).
• Requires the port. Stock mlx-lm cannot load glm5_next as of 2026-08-28.

License and attribution

MIT, inherited from zai-org/GLM-5.3-Flash (© 2026 Z.AI Co., Ltd). This repository redistributes quantized weights derived from that checkpoint; all model credit belongs to the Z.AI team. Conversion, port, recipe, and verification by avlp12.

Citation

Alis (avlp12) (2026). GLM-5.3-Flash-Alis-MLX-8bit — 8-bit MLX quantization of GLM-5.3-Flash. https://huggingface.co/avlp12/GLM-5.3-Flash-Alis-MLX-8bit