GLM-5.3-Flash-AZ

GLM-5.3-Flash with half the routed experts per token, healed by self-distillation — ~20 % faster decoding at the same tool-calling quality.

~324 B total parameters, ~12.5 B active per token (vs ~17 B with 8 experts). The "50 B" shown by the Hub counts the packed INT4 weights as stored 32-bit integers (8 weights each) and includes the quantization scales; Intel's INT4 release shows the same figure.

This is Intel/GLM-5.3-Flash-W4A16-AutoRound (INT4 AutoRound quantization of zai-org/GLM-5.3-Flash) with two changes:

1. Top-k 8 → 4. Each token is routed to 4 of the 288 experts per MoE layer instead of 8 (num_experts_per_tok: 4), which halves the routed-expert weights read per token.
2. Healed shared experts. Cutting experts shifts the model's outputs, so the always-active shared expert of every MoE layer (layers 3–44) was retrained to make the 4-expert model reproduce the original 8-expert model's next-token distribution (KL self-distillation).

Everything else is byte-for-byte Intel's release: the INT4 routed experts, attention, embeddings, head, router and the MTP (multi-token prediction) layer. Only the 126 shared-expert tensors (BF16, 1.06 B parameters) differ, and they sit in the same files, under the same names, at the same offsets as in the original, so the repository has the same layout and index as Intel's and loads like it.

Results

Measured on 2× NVIDIA GB10 (128 GB each), tensor-parallel 2, with the myllmbox GLM-5.3-Flash cluster recipe (myllmbox/glm53-flash-cluster-vllm:v1, vLLM 0.31, MTP speculative decoding at its default settings). Both models were served with the identical recipe; the only difference is the weights. Benchmarks: tool-eval-bench 2.7.0.

These numbers were measured on the epoch-2 midpoint checkpoint (update 1200) of this training run. The published weights are the end of epoch 2 (update 1546), which is 96.9 % top-1-identical to it; numbers for the final weights will replace these.

Tool calling (tool-eval-bench, 69 scenarios, 3 full runs each):

	GLM-5.3-Flash INT4 (8 experts)	GLM-5.3-Flash-AZ (4 experts)
Score (mean ± sd of 3 runs)	93.3 ± 1.5	94.3 ± 1.5
Runs	93 / 92 / 95	93 / 94 / 96
Total tokens per run	449 K	442 K

Decode speed, single stream (tool-eval-bench --perf-only, pp2048 + tg128, concurrency 1):

Context depth	8 experts (tok/s)	AZ (tok/s)	Speed-up
0	43.5	52.9	+22 %
4 K	41.8	49.0	+17 %
8 K	42.3	46.8	+11 %

Speculative decoding, effective tok/s (tool-eval-bench --spec-bench, depth 0, mean of 3):

Workload	8 experts	AZ	Speed-up
Code	44.1	53.5	+21 %
Structured output	48.7	58.6	+20 %
Free text	42.3	48.9	+16 %

Prefill is also ~10 % faster (pp2048 at depth 0: 1,652 vs 1,507 tok/s).

Long Coding Strings Performance (one long code-generation request of 2,047 output tokens, single stream, streaming decode speed, 2 passes after a warm-up):

	8 experts	AZ	Speed-up
Decode tok/s (median of 2 passes)	47.1	63.7	+35 %
Decode tok/s (best pass)	47.5	64.8	+36 %
Time for 2,047 tokens (best pass)	43.1 s	31.6 s	−27 %

Limitations

• Fewer active experts is not free. Tool calling and everyday tasks are on par with the 8-expert model in our testing, but on hard multi-step coding and expert-level reasoning problems our internal (non-public) evaluation still shows a clear gap to the original 8-expert model. If those workloads matter most to you, use the original.
• The MTP draft layer was not retrained; its acceptance rate is ~1–3 points lower than with 8 experts, which the faster main model more than compensates for.
• Only tested with the myllmbox image above. Any runtime that loads Intel's INT4 AutoRound checkpoint should load this one the same way (same files, names and index), with num_experts_per_tok read from config.json.
• The base model's limitations apply: it can produce incorrect, biased or offensive output; test before deploying.

Usage (myllmbox cluster recipe, 2× GB10)

git clone https://github.com/myllmbox/glm53-flash-cluster-recipe && cd glm53-flash-cluster-recipe
# in recipe.yaml set:  server.model: azampatti/GLM-5.3-Flash-AZ
# then follow the recipe's README (setup.sh once for the two boxes, then run.sh);
# run.sh downloads the weights on the first run and syncs them to the worker

The server exposes an OpenAI-compatible API (served name zai-org/GLM-5.3-Flash unless you change served-model-name). GLM-5.3-Flash always reasons; control the depth with reasoning_effort.

How it was made

• Teacher: the same INT4 checkpoint at top-k 8 with its original shared experts. Student: top-k 4, only the shared experts trainable (gate/up/down of layers 3–44); routed experts stay frozen INT4 (dequantized to BF16 inside each forward pass, bit-exact with auto-round).
• Loss: KL divergence between the teacher's and the student's next-token distributions (teacher top-64) on the completion tokens only.
• Data: 14,142 prompts up to 9,728 tokens — math 30 %, tool use 22 %, code 22 %, general 20 %, long context 6 % — with completions written by GLM-5.3-Flash itself (so the model learns its own behaviour, not another model's style); ~17.5 M tokens per epoch.
• Optimisation: AdamW, LR 5e-5 constant after a 75-update warm-up, aprox 22 K tokens per update, gradient clip 2.5 (never reached), 2 epochs = 1,546 updates (about 93 h), pipeline-parallel across two GB10 nodes (layers 0–23 / 24–44, 1F1B).
• Held-out KL to the 8-expert model (255 unseen prompts): 0.120 unhealed → 0.065 after epoch 1 → 0.061 after epoch 2 (−49 %). Served top-1 agreement with the 8-expert model (12 reference conversations, completion tokens): 92.8 % unhealed → 95.7 %.
• Weight movement of the shared experts vs Intel's release: 5.9 % (relative RMS).

License and credits

MIT, as the base model — please also follow the license of GLM-5.3-Flash. Built on the work of Z.ai (GLM-5.3-Flash), Intel (auto-round INT4 quantization) and myllmbox (cluster serving recipe).