GLM-5.3-INT4-FP8Mix

A mixed INT4/FP8 quantization of GLM-5.3, stored in the compressed-tensors format. Most routed experts use symmetric INT4 weights with BF16 activations (W4A16); selected projections and the final main-model expert layer use block FP8.

• Base model: zai-org/GLM-5.3-BF16
• Quantization: symmetric INT4 (group size 128) + FP8 E4M3 (128 × 128 blocks)
• Serving: SGLang / vLLM

Quantization

Layer numbers below are zero-based.

Modules	Weight format	Activations
Routed experts in layers 3–76	Symmetric INT4, group size 128, packed	BF16
Routed experts in layer 77	FP8 E4M3, 128 × 128 blocks	Dynamic FP8, group size 128
Configured attention, dense MLP, and shared-expert projections in layers 1–77	FP8 E4M3, 128 × 128 blocks	Dynamic FP8, group size 128
Routed experts in MTP layer 78	Symmetric INT4, group size 128, packed	BF16
Layer 0, routing gates, attention indexers, non-expert MTP modules, and other unquantized parameters	Original precision	Original precision

The exact module selection is defined by targets and ignore in config.json. The top-level format is mixed-precision, with pack-quantized and float-quantized formats specified per group.

Evaluation

Benchmark	GLM-5.3-INT4-FP8Mix	GLM-5.3
GPQA-Diamond avg@3	92.421	92.421
GPQA-Diamond details	92.42/91.41/93.43	-
AA-LCR v1.12 avg@3	79.67%	80%3
AA-LCR v1.1 details	80/78/81	-

1. The highest GPQA-Diamond score for the INT4/FP8 model was obtained with SGLang. Artificial Analysis reports 92% for GLM-5.3 (max), consistent with the reproduced baseline score of 92.42% shown above.

2. The INT4/FP8 model's AA-LCR v1.1 evaluation used NVIDIA's NeMo Gym evaluation framework, with GLM-5.3-Flash as the judge.

3. The 80% AA-LCR v1.1 reference score is sourced from Artificial Analysis's GLM-5.3 (max) evaluation.

Quick Start

The examples use a local checkpoint path and tensor parallelism across eight GPUs. Adjust parallelism and context length to fit the available GPU memory. Both servers below listen on port 8000.

Prepare SGLang

Two patch bundles are provided for SGLang 0.5.18 and 0.5.21. Choose the bundle matching your source version and adapt the fixes to your deployment requirements. The installers require the exact commits below; for other revisions, review upstream changes and update the patches against those sources.

SGLang version	Required commit	Patch instructions
0.5.21	e00930c5489053f26d86b179cee0d087f846acbb	sgalng-0.5.21-patches
0.5.18	71de97b264b04dcd514cf904003028aefe9775c8	sgalng-0.5.18-patches

Apply the bundle for your source version:

# SGLang 0.5.21
bash sgalng-0.5.21-patches/apply.sh --check /path/to/sglang-v0.5.21
bash sgalng-0.5.21-patches/apply.sh /path/to/sglang-v0.5.21

# SGLang 0.5.18
bash sgalng-0.5.18-patches/apply.sh --check /path/to/sglang-v0.5.18
bash sgalng-0.5.18-patches/apply.sh /path/to/sglang-v0.5.18

Serve with SGLang

Always include --disable-shared-experts-fusion. Without this flag, SGLang cannot correctly identify the precision of this checkpoint's shared experts. (Several community-released mixed-quantization models also include this flag by default; this behavior is not specific to this model.) The shared experts use FP8 while most routed experts use INT4, so their quantization schemes must be handled separately.

python -m sglang.launch_server \
  --model-path /path/to/GLM-5.3-INT4-FP8Mix \
  --served-model-name GLM-5.3-INT4-FP8Mix \
  --port 8000 \
  --tp 8 \
  --disable-shared-experts-fusion \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Serve with vLLM

vllm serve /path/to/GLM-5.3-INT4-FP8Mix \
  --served-model-name GLM-5.3-INT4-FP8Mix \
  --port 8000 \
  --tensor-parallel-size 8 \
  --reasoning-parser glm45 \
  --tool-call-parser glm47 \
  --enable-auto-tool-choice

Call the server

curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "GLM-5.3-INT4-FP8Mix",
    "messages": [{"role": "user", "content": "Write a Python function that checks whether a number is prime."}],
    "max_tokens": 1024
  }'

Citations

If you use this checkpoint, please cite:

@misc{glm53int4fp8mix2026,
  author = {AlayaNeW},
  title = {{GLM-5.3-INT4-FP8Mix}},
  year = {2026},
  url = {https://huggingface.co/AlayaNeW/GLM-5.3-INT4-FP8Mix},
  note = {Mixed INT4/FP8 quantization of GLM-5.3}
}