cfao/GLM-5.2-Int4-MSE-Int8Mix

GLM-5.2 — INT4 (MSE) experts + INT8 attention mix, for CPU/GPU hybrid serving

A quantized build of GLM-5.2 tuned for CPU/GPU hybrid MoE serving on a single GPU that can't hold all 256 experts/layer at higher precision: routed MoE experts are self-quantized to INT4 (compressed-tensors, per-group MSE-calibrated), and the attention path (q_b_proj, o_proj, and the shared-expert MLP in most layers) is swapped to an INT8 mix, freeing enough memory to run a substantially larger KV-cache pool and a weight-staging overlap system without giving up single-stream decode speed.

This is the weight side of a larger effort — see the paired serving-stack forks for the runtime work that makes this checkpoint fast (not the vanilla upstream projects, which lack the fixes below): cfao-new/sglang (branch bisect-no58) and cfao-new/ktransformers (branch kt064-glm52). Those READMEs document the actual measured wins (sparse attention restoration, native fp8 MLA decode, weight-staging overlap, and the bugs fixed along the way); this card focuses on the checkpoint itself.

What's in this checkpoint

Component	Precision	Method
Routed MoE experts (all layers)	INT4, group size 32	Custom per-group MSE-calibrated quantization (compressed-tensors, symmetric)
q_b_proj, o_proj, shared-expert MLP (layers 3-77)	INT8, group size 128	Donor tensors from QuantTrio/GLM-5.2-Int4-Int8Mix
Everything else (embeddings, norms, layers 0-2, indexer)	BF16	Unchanged from the base release

The INT4 expert quantization was run in-house against the official zai-org/GLM-5.2-FP8 release using a custom MSE grid-search calibrator built on compressed-tensors; it is not a redistribution of a third-party quantization. The INT8 attention/shared-expert tensors are QuantTrio's own public release, mixed in via the module-level scheme their build uses.

Why this specific mix

A straight INT4 quantization of everything leaves accuracy on the table on a few precision-sensitive attention/expert-selection paths. Swapping just the attention projections and shared-expert MLP to INT8 — the QuantTrio donor set — recovers that margin while keeping the bulk of the parameter count (the routed experts, which dominate total size) at INT4. The net effect versus an all-INT4 build: comparable weight footprint, meaningfully better numerical headroom on the attention path, and — combined with the serving stack's own weight-staging optimizations — enough freed memory to run this model with a much larger KV-cache pool than the unmixed build supports.

Serving

Built and validated against the paired sglang + kt-kernel forks linked above. Key serving flags (see the forks' READMEs for the complete, currently-adopted configuration):

--attention-backend nsa            # native sparse attention (this is a trained-in DSA model)
--kv-cache-dtype fp8_e4m3          # native SM90 fp8 MLA decode
--max-total-tokens 524288          # funded by the memory this quantization mix frees up

Attribution and license

• Base model: zai-org/GLM-5.2-FP8, Zhipu AI, MIT license.
• INT8 attention/shared-expert donor tensors: QuantTrio/GLM-5.2-Int4-Int8Mix, MIT license.
• This checkpoint (the INT4 expert requantization and the assembly of the above into one build): released under the same MIT license, in keeping with both upstream sources.

See the included LICENSE file for the full MIT text and copyright notice.