GLM-5.3-SLIM-GGUF (E192, UD-IQ2_M)

A GLM-5.3-class model that runs on a single NVIDIA B200, or on a Mac Studio with 192 GB or more of unified memory (with mlx-serve). At 179.9 GB it is 24.6% smaller than the 256-expert UD-IQ2_M (238.6 GB), which does not fit in the memory of one B200 or a 192 GB Mac. On one B200 the whole model sits on the GPU and about 100K tokens of KV cache are left for up to 10 parallel requests; in our tests it scores 92.7% on HumanEval.

GLM-5.3 with a 192-expert MoE architecture found by neural architecture search (NAS): every MoE layer keeps 192 of the original 256 routed experts. The model is built from Unsloth's UD-IQ2_M GGUF quantization.

	Unsloth GLM-5.3 UD-IQ2_M	This model
routed experts per MoE layer	256	192
parameters	753B	563B
file size	238.58 GB (222.19 GiB)	179.86 GB (167.51 GiB), -24.6%
average bits per weight		2.56
MTP (next-token prediction) layer	yes	removed
HumanEval pass@1 (greedy, thinking off)	not measured	92.7%
GPQA-Diamond (T=1.0, reasoning_effort=high, 65,536 tokens)	not measured	84.8%

How it was made

• Architecture (NAS). The 192-experts-per-layer architecture is the one found by the neural architecture search behind GLM-5.3-SLIM-E192. Each MoE layer (3-77) keeps the experts chosen by the search; the other 64 routed experts per layer are removed.
• No re-quantization. In GGUF the expert axis is the slowest dimension of ffn_{gate,up,down}_exps, ffn_gate_inp and exp_probs_b, so each expert is one contiguous byte range. The kept experts are copied byte-for-byte, in their original order, as experts 0-191. Every tensor keeps the quant type Unsloth chose; all other tensors (attention, DSA indexer, shared experts, dense layers 0-2, embeddings, output) are byte-identical to the source.
• MTP layer removed (blk.78), as in SLIM. Do not use MTP speculative decoding with this model.
• Metadata changes: glm-dsa.expert_count 256 -> 192, glm-dsa.block_count 79 -> 78, glm-dsa.nextn_predict_layers 1 -> 0, general.size_label 256x22B -> 192x22B. Tokenizer and chat template are unchanged.
• No retraining, no recalibration.

Quant types (unchanged from UD-IQ2_M):

tensors	size	types
ffn_down_exps	65.9 GiB	IQ3_XXS (71 layers), IQ4_XS (4)
ffn_gate_exps, ffn_up_exps	2 x 43.7 GiB	IQ2_XXS (74), IQ2_S (1)
attn_output, attn_q_a	5.7 GiB	Q5_K, Q6_K
attn_q_b, attn_k_b, attn_v_b, attn_kv_a_mqa, indexer	4.7 GiB	Q8_0
shared experts	1.9 GiB	Q5_K / Q6_K / Q8_0
token_embd / output	0.6 / 0.5 GiB	Q5_K / Q4_K

Files

file	size
GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf	metadata only
GLM-5.3-UD-IQ2_M-E192-00002-of-00006.gguf ... 00006	38.1 / 37.5 / 37.5 / 37.5 / 29.1 GB
imatrix_unsloth_E192.gguf	0.86 GB. Unsloth's imatrix with the same expert slicing applied, for re-quantizing this 192-expert model to other types. Not needed to run the model.
eval/serve_b200.sh	starts llama-server on one B200 with the settings used for the evaluation
eval/gpqa.py	GPQA-Diamond client with KV-pool admission control (see step 7 of the deployment guide)
eval/humaneval.py	HumanEval pass@1 client (greedy, thinking off); runs the generated code against the official tests

Deploying on a single NVIDIA B200

Everything below was run on one NVIDIA B200 (183,359 MiB reported by nvidia-smi) with llama.cpp commit 8e5a58d (2026-10-24), NVIDIA driver 595.91 and CUDA 13.0.

1. Requirements

• One NVIDIA B200. The weights alone need about 167 GiB of GPU memory, so smaller GPUs need a multi-GPU split (not tested here).
• An NVIDIA driver that supports CUDA 13, and the CUDA 13 toolkit (nvcc) to build llama.cpp.
• 180 GB of free disk space for the GGUF files.
• For the evaluation script: Python 3.10+ and pip install aiohttp datasets.

2. Build llama.cpp with CUDA

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
git checkout 8e5a58d              # tested commit; newer commits may work but were not tested
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=100 -DCMAKE_BUILD_TYPE=Release
cmake --build build --target llama-server llama-cli -j

CMAKE_CUDA_ARCHITECTURES=100 builds only for the B200 (sm_100), which keeps the build short.

3. Download

pip install -U huggingface_hub
hf download Scantist/GLM-5.3-SLIM-GGUF --include "GLM-5.3-UD-IQ2_M-E192-*.gguf" "eval/*" --local-dir GLM-5.3-E192

Always pass the first split (...-00001-of-00006.gguf) to llama.cpp; it loads the other splits itself.

4. Memory budget

item	GPU memory
weights (-ngl 99)	170,896 MiB (token_embd, 624 MiB, stays in host memory)
compute buffer (-ub 512)	~512 MiB
KV cache, f16	93 KiB per token (MLA 87.75 KiB + DSA indexer 5.25 KiB)
KV cache, q4_0 (-ctk q4_0 -ctv q4_0)	26.2 KiB per token

Measured total use after start-up, with the unified KV pool:

-c (KV pool, tokens)	GPU memory used	free
102,400	181,678 MiB	~1.7 GB	recommended
110,592	182,454 MiB	~0.9 GB	too little room for temporary buffers
343,040 with q4_0 KV	182,226 MiB	~1.1 GB	5 requests of 68,608 tokens at once

So a B200 holds about 100K tokens of KV cache in f16, or about 340K tokens in q4_0, shared by all requests.

5. Start the server

eval/serve_b200.sh starts llama-server with the settings used for the GPQA run (10 slots, 102,400-token shared pool, up to 68,608 tokens per request):

cd GLM-5.3-E192/eval
LLAMA_SERVER=/path/to/llama.cpp/build/bin/llama-server ./serve_b200.sh

This is the same as:

llama-server -m GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf \
  -ngl 99 -kvu -c 102400 --kv-unified-per-slot 68608 -np 10 -bs \
  --host 127.0.0.1 --port 8080 --metrics

flag	meaning
-ngl 99	put all layers on the GPU
-kvu -c 102400	one KV pool of 102,400 tokens shared by all slots
--kv-unified-per-slot 68608	limit per request: the longest GPQA prompt (2,821 tokens) + 65,536 generated tokens
-np 10	10 parallel slots
-bs	sample on the GPU (see the notes below)

Change any value with an environment variable (CTX, NP, SLOT_CTX, HOST, PORT, MODEL); DRY_RUN=1 prints the command without running it. Start-up takes about 30 s when the files are in the page cache; the first start also reads 180 GB from disk.

Check that the server is up, then send a request:

curl http://127.0.0.1:8080/health          # {"status":"ok"}

curl http://127.0.0.1:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Write a Python function that checks whether a string is a palindrome."}],
  "temperature": 1.0, "top_p": 0.95, "top_k": 0, "min_p": 0,
  "chat_template_kwargs": {"reasoning_effort": "high"},
  "max_tokens": 8192
}'

For an interactive session instead of a server:

llama-cli -m GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf -ngl 99 -c 16384 --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0

6. Choosing -c and -np

With -kvu, the per-slot limits may add up to more than the pool. If the pool runs out, llama-server fails all running requests with Context size has been exceeded and clears them. Size the setup by the total number of tokens that are live at the same time:

• Short requests (chat, code): more slots are fine as long as slots x longest request stays below the pool, e.g. -np 32 with -c 102400 when no request goes past ~3K tokens. Total generation speed rises with the number of parallel sequences (see Throughput below).
• Long reasoning (up to 65K tokens per request): one such request alone uses two thirds of the pool. Either run few slots, or let the client control admission, as eval/gpqa.py does: it starts a new request only when the pool has room, and when the pool is nearly full it aborts the youngest request and runs it again later.

7. Run GPQA-Diamond

With the server from step 5 running:

cd GLM-5.3-E192/eval
python gpqa.py --url http://127.0.0.1:8080 --pool 102400 --max-conc 10 --out gpqa_results.jsonl
python gpqa.py --summary --out gpqa_results.jsonl

--pool and --max-conc must match the server's -c and -np. The script prints one line per finished question (accuracy so far, active requests, pool use, tokens per second, number of preemptions). Results are appended to --out after every question, so a stopped run continues where it stopped when you start it again.

For long-reasoning runs like this one, use 5 slots with a 4-bit KV cache instead. The pool then holds 5 requests of full length, so the client never has to abort anything:

CTX=343040 NP=5 ./serve_b200.sh -ctk q4_0 -ctv q4_0
python gpqa.py --url http://127.0.0.1:8080 --max-conc 5 --pool 1000000 --reserve 0 --out gpqa_results.jsonl

(--pool 1000000 --reserve 0 turns the client-side pool control off; it is not needed when the server pool holds --max-conc full-length requests.) Our run took about 13.5 hours: the first 166 questions ran with the 10-slot f16 setup above, and once only the long, hard questions were left the client aborted and restarted requests 275 times, which wasted most of that time. The last 32 questions ran with the 5-slot q4_0 setup in 2.3 hours, without aborts.

8. Notes

• Reasoning. The chat template always opens a <think> block. Set the effort per request with "chat_template_kwargs": {"reasoning_effort": "low" | "high"}; anything else means max. --reasoning-budget 0 turns thinking off.
• Sampling. The recommended setting is temperature=1.0, top_p=0.95; llama-server takes these defaults from the GGUF metadata, but it also applies top_k=40 and min_p=0.05 by default. Pass top_k=0, min_p=0 to turn them off.
• -bs (backend sampling, experimental) moves sampling to the GPU: +23% throughput at 10 slots in our test (66.9 -> 82.5 tok/s). The sampled distribution matched the CPU sampler (two-sample chi-square, p = 0.88 / 0.19 / 0.18 on three prompts), but the token probabilities returned with n_probs were not accurate. Leave out -bs if you need logprobs.
• No MTP. The model has no MTP layer, so do not enable MTP speculative decoding.

9. Troubleshooting

symptom	cause and fix
ggml_cuda_init: failed to initialize CUDA: system has unsupported display driver / cuda driver combination, while nvidia-smi works	The loader picks the libcuda.so.1 of the CUDA forward-compatibility package instead of the driver's. Put the directory with the driver's libcuda.so.<driver version> first in LD_LIBRARY_PATH.
out of memory at start-up	Lower -c (each 10K tokens of pool is about 0.9 GiB).
Context size has been exceeded, and all running requests fail	The shared pool ran out; see step 6.
a reply stops with finish_reason: length before max_tokens	Prompt + max_tokens is larger than --kv-unified-per-slot; raise it.
GGML_ASSERT(*cur_backend_id != -1) while the context is created	No GPU is in use (CPU-only build, or CUDA failed to start). At the tested commit the CPU backend cannot run this architecture; the original 256-expert UD-IQ2_M fails the same way.

Throughput (one B200, llama-batched-bench)

Prompt 512 tokens, 128 generated tokens per sequence, unified KV, f16 or 4-bit (-ctk q4_0 -ctv q4_0) KV cache:

parallel sequences	1	4	8	16	32	64
generation, tok/s (total), f16 KV	34.5	91.4	110.4	130.3	164.3	203.3
generation, tok/s (total), q4_0 KV	40.5	91.1	109.4	128.9	162.6	200.2
prompt processing, tok/s, f16 KV	648	672	643	596	524	573
prompt processing, tok/s, q4_0 KV	668	673	644	597	526	573

Long prompts:

setting	f16 KV	q4_0 KV
16K-token prompt, 1 sequence, generation tok/s	37.4	34.9
16K-token prompt, 4 sequences, generation tok/s (total)	81.9	70.6
1 sequence at 1K tokens of context, generation tok/s	39.9	37.4
1 sequence at 32K tokens of context, generation tok/s	36.5	32.4
1 sequence at 63K tokens of context, generation tok/s	34.6	27.8
prompt processing at 32K-63K tokens, tok/s	550	550

• With short contexts the KV type makes no difference. With long contexts the 4-bit cache is slower per sequence: -11% at 32K and -20% at 63K tokens for one sequence, -14% for four sequences at 16K.
• In exchange, the 4-bit cache needs 26.2 KiB instead of 93 KiB per token, so five requests of 68K tokens fit on one B200 instead of one. For long-reasoning workloads this gives much more total throughput.
• Single-sequence numbers vary by about +-10% between runs (f16, 1K context: 34.5 and 39.9 tok/s in two runs).
• In llama-server the speed can be lower than in llama-batched-bench: in the GPQA run (5-slot q4_0 setup), the last remaining request generated about 17 tok/s at ~56K tokens of context.
• These speeds are limited by the llama.cpp kernels for these low-bit types, not by the GPU's memory bandwidth.

Evaluation

All results below come from our own scripts against llama-server with the settings shown. They are not comparable to the numbers in the official GLM-5.3 card. We did not run the original 256-expert UD-IQ2_M or the full-precision model on the same harness, so the cost of the expert reduction itself is not measured here; for that, see the A/B results of GLM-5.3-SLIM-E192 (FP8) against GLM-5.3 (FP8).

benchmark	setting	result
GPQA-Diamond (198 questions)	T=1.0, top_p=0.95, top_k/min_p off, reasoning_effort=high, max 65,536 new tokens, 1 sample	84.8% (168/198)
HumanEval (164), pass@1	greedy, thinking off, max 2048 new tokens, one request at a time	92.7% (152/164)
HumanEval (164), pass@1	same, 8 concurrent requests	89.6% (147/164)
MMLU, 570-question subset (10 per subject, seed 0)	zero-shot, greedy, thinking off, answer letter only	71.4% (407/570)

• HumanEval: answers were run against the official tests in a subprocess. The two runs differ only through batch numerics; the difference is not significant (exact McNemar test on the 9 discordant problems, p = 0.18). 11 of 164 serial answers reached the 2048-token limit (5 of them still passed).
• GPQA-Diamond: questions from the public mirror fingertap/GPQA-Diamond (same 198 questions, options already shuffled). Prompt: "Answer the following multiple choice question. The last line of your response should be of the following format: 'Answer: $LETTER' ... Think step by step before answering." The answer is parsed from the final response only; a response that hits the token limit counts as wrong. The run used eval/gpqa.py with the server from eval/serve_b200.sh (step 5 and 7 of the deployment guide); requests that were aborted to keep the KV pool from overflowing were restarted from the beginning with a new seed, so every question has exactly one complete sample (48 of the 198 answers are such a later attempt). The first 166 questions ran with an f16 KV cache (10 slots, 142/166 correct); the last 32 with a q4_0 KV cache (5 slots, 26/32 correct). 9 answers reached the 65,536-token limit and 1 ended without a parsable answer; all 10 count as wrong.

Reproducing

HumanEval: the numbers above were produced with this server (8 slots of 8,192 tokens, thinking off, CPU sampling) and one request at a time:

llama-server -m GLM-5.3-UD-IQ2_M-E192-00001-of-00006.gguf -ngl 99 -c 65536 -np 8 --reasoning-budget 0 --port 8080
python eval/humaneval.py --url http://127.0.0.1:8080 --concurrency 1 --out humaneval_results.json

humaneval.py runs model-generated code on your machine; use a sandbox or a throw-away container. With other server settings the score can move by a few problems, because batch composition changes the numerics (compare the serial and the 8-concurrent run above).

GPQA-Diamond: start the server with eval/serve_b200.sh and run eval/gpqa.py (steps 5 and 7 of the deployment guide).

Integrity checks

• All 1,782 tensors were compared with the source UD-IQ2_M files: non-expert tensors are byte-identical, and every expert-axis tensor equals the source tensor restricted to the kept experts, in their original order.
• The expert mapping was cross-checked against an independent NVFP4 checkpoint built with the same expert selection (from nvidia/GLM-5.3-NVFP4): the kept router rows (ffn_gate_inp, exp_probs_b) are bit-identical in all 75 MoE layers, and dequantized experts match with cosine similarity 0.94 (IQ2_XXS), 0.98 (IQ3_XXS) and 0.994 (IQ4_XS), against < 0.003 for a neighbouring expert.

Limitations

• Two losses stack: 25% of the routed experts are removed, and the weights are at ~2.6 bits per weight.
• SLIM's A/B tests of the same expert set (FP8) show a clear loss on Chinese knowledge QA and a smaller loss on science QA; expect the same here.
• With greedy decoding and thinking off, a few percent of long answers wander or repeat (in HumanEval, 1 of 164 answers looped on <code>), and the model sometimes uses <code> tags instead of Markdown code fences.
• No MTP layer, so no MTP speculative decoding.

License and credits

Released under the GLM-5.3 License, the same terms as the base model. Weights by Z.ai; quantization and imatrix by Unsloth; expert selection from GLM-5.3-SLIM-E192; runtime by llama.cpp.