L-Alchemyst/GLM-5.3-GGUF

GLM-5.3

Imatrix quants best-in-class for ik_llama.cpp

NOTE: ik_llama.cpp is highly versatile and runs mainline llama.cpp quants such as existing GGUFs from bartowski, unsloth, mradermacher, Aes Sedai etc

⚠ Work in progress — IQ3_K uploading now. IQ2 and IQ4 variants coming soon. Do not download yet.

General Specs

	Value
Architecture	GLM-DSA (MoE, 256 experts × 22B, 8 active)
Params	744B total / 40B active
Context	1,048,576
Base	zai-org/GLM-5.3-BF16
Imatrix	Aes Sedai — converted for ik_llama.cpp
Calibration	calibration_datav5.txt

Quant Collection

IQ3_K  3.670 BPW · 321.83 GiB

Evaluated on wiki.test.raw, 512 ctx, 565 chunks, against BF16 reference logits.

Metric	Value
PPL (quant)	2.820603 ± 0.014842
PPL (BF16 ref)	2.674315 ± 0.013811
(PPL(Q)/PPL(base)) - 1	+5.4701% ± 0.1627%
KLD	0.108800 ± 0.000809
Same top-p	90.352 ± 0.078 %
Δp RMS	12.780 ± 0.070 %

Quantization recipe

# Attention blk\..*\.attn_k_b\.weight=q8_0 blk\..*\.attn_v_b\.weight=q8_0 blk\..*\.attn_kv_a_mqa\.weight=q8_0 blk\..*\.attn_q_a\.weight=iq6_k blk\..*\.attn_q_b\.weight=iq6_k blk\..*\.attn_output\.weight=iq6_k

# Router blk\..*\.ffn_gate_inp\.weight=f32

# First 3 Dense Layers blk\..*\.ffn_down\.weight=iq6_k blk\..*\.ffn_(gate|up)\.weight=iq5_ks

# Shared Expert Layers blk\..*\.ffn_down_shexp\.weight=iq6_k blk\..*\.ffn_(gate|up)_shexp\.weight=iq6_k

# Routed Experts blk\.(78)\.ffn_down_exps\.weight=q6_K blk\.(78)\.ffn_(gate|up)_exps\.weight=q6_K blk\..*\.ffn_down_exps\.weight=iq4_ks blk\..*\.ffn_(gate|up)_exps\.weight=iq3_ks

# Indexer blk\..*\.indexer\.k_norm\.(weight|bias)=f32 blk\..*\.indexer\.proj\.weight=q8_0 blk\..*\.indexer\.attn_k\.weight=q8_0 blk\..*\.indexer\.attn_q_b\.weight=q6_K

# MTP block - eh_proj is the throughput-critical tensor per ik's finding blk\..*\.nextn\.eh_proj\.weight=q4_0 blk\..*\.nextn\.enorm\.weight=f32 blk\..*\.nextn\.hnorm\.weight=f32 blk\..*\.nextn\.shared_head_norm\.weight=f32

token_embd\.weight=iq5_k output\.weight=iq6_k

Quick Start

Requires ik_llama.cpp. Standard llama.cpp will not run these quants.

# Clone and build git clone https://github.com/ikawrakow/ik_llama.cpp cd ik_llama.cpp cmake -B build -DCMAKE_BUILD_TYPE=Release -DGGML_CUDA=ON cmake --build build --config Release -j $(nproc)

Download

pip install huggingface_hub hf download L-Alchemyst/GLM-5.3-GGUF --repo-type model --include "IQ3_K/*" --local-dir ./glm5.3-iq3k

Hybrid CPU+GPU — adjust --n-cpu-moe and -cram to your RAM

./build/bin/llama-server
--model GLM-5.3-IQ3_K.gguf
-muge --merge-qkv
--ctx-size 131072
-ctk f16 -mla 3 -amb 1536
-ngl 999
--n-cpu-moe 50
--parallel 1
--threads 96 --threads-batch 128
--host 127.0.0.1 --port 8080
--no-mmap
-cram 8192
--jinja

CPU-only: use -ctk q8_0 and prefix with numactl -N ${SOCKET} -m ${SOCKET}.

Credits
Imatrix & reference logits: Aes Sedai
Base model: zai-org/GLM-5.3-BF16
Runtime: ik_llama.cpp