anthori/GLM-5.3-Flash-EXL3-Q8Core-Q3Q4Q6

GLM-5.3-Flash-EXL3-256K-Q8Core-Q3Q4Q6 (4.59 bpw)

This is a custom mixed-precision EXL3 conversion of zai-org/GLM-5.3-BF16, built for ExLlamaV3. It preserves selected high-sensitivity tensors at higher precision while fitting a 256K-token FP16 KV-cache deployment on a two-GPU system.

Overall quantization: 4.59 bits per weight (bpw), as reported by the EXL3 checkpoint metadata.

This is an EXL3 checkpoint. It is intended for ExLlamaV3 and compatible frontends such as TabbyAPI; it is not a Transformers or vLLM checkpoint.

Quantization

The complete tensor mapping is recorded in quantization_config.json. The model's reported average EXL3 bitrate is 4.59 bpw.

Component	EXL3 precision
Attention projections, dense MLP layers, shared experts, and output head	Q8
Routed expert down_proj	Q6
Routed expert gate_proj	Q4
Routed expert up_proj	Q3, promoted to Q4 in selected layers
MTP module	Q4

The Q4 routed up_proj promotions are layers 3, 11, 16-18, 21, 26, 28-32, 34-35, and 37-44. They include the final sparse layer and additional layers selected from the importance data used by the original mixed-precision plan.

Non-projection tensors retain their appropriate original storage formats. See quantization_config.json for the authoritative per-tensor representation.

Tested Deployment

This release targets a two-GPU, 192GB-class VRAM configuration. It is not a single-GPU release and it does not use tensor parallelism; ExLlamaV3 places consecutive layers across the two GPUs.

Item	Tested configuration
GPUs	2 x NVIDIA RTX PRO 6000 Blackwell Workstation Edition
Available VRAM	97,887 MiB per GPU, 195,774 MiB aggregate (approximately 192 GB / 191.2 GiB)
Power limit	450 W per GPU
Runtime	ExLlamaV3 1.4.8 through TabbyAPI
Context and main KV cache	262,144 tokens, FP16, GPU-resident
Main-model layer placement budget	GPU 0: 89.75 GiB; GPU 1: 94.0 GiB
MTP placement budget	GPU 0: 4.5 GiB; GPU 1: 0 GiB
MTP	Enabled, 3 draft tokens
Vision	Enabled, BF16 vision weights offloaded to pinned system RAM
CPU MoE or KV offload	Disabled
Batch size	1

The placement budgets above are the explicit ExLlamaV3 split configuration, not a promise of a fixed nvidia-smi reading. Allocator and workspace usage vary by driver, prompt length, and request state.

Measured Interactive Performance

At the 450 W-per-GPU limit above, interactive requests on 69.8K-75.0K-token conversations with 98-100% prompt-cache hits produced:

Measurement	Observed result
Output throughput with MTP	82.7-125.6 tokens/s
Cached incremental prefill	802-1,186 tokens/s
First-token latency	0.23-1.45 s
MTP draft-token acceptance	47-88%

The prefill number is the rate for the 714-1,708 newly added tokens in otherwise cached conversations. It is not an isolated cold-prefill benchmark and should not be read as a full-context prefill claim.

Runtime Notes

• The checkpoint's 256K profile uses an FP16 KV cache; FP16 is the cache format exposed by ExLlamaV3.
• The model supports vision. The tested deployment keeps vision weights in system RAM with vision_offload: true.
• GPU placement is hardware dependent. The split above was selected to retain long-prompt workspace on GPU 1.

An example TabbyAPI starting point is:

model:
  model_dir: /path/to/parent-directory
  model_name: GLM-5.3-Flash-EXL3-256K-Q8Core-Q3Q4Q6
  backend: exllamav3
  max_seq_len: 262144
  cache_size: 262144
  cache_mode: FP16
  vision: true
  vision_offload: true

draft_model:
  draft_mode: mtp
  draft_cache_mode: FP16
  draft_num_tokens: 3

License and Attribution

This is a derived quantization of Z.AI's GLM-5.3-BF16 release. The upstream license is included in LICENSE and governs use and redistribution of this checkpoint. Please cite and attribute the original GLM-5.3 release when using this model.

Conversion

Converted with ExLlamaV3 1.4.8. This repository contains the complete, self-contained checkpoint: 30 safetensor shards, model and tokenizer configs, and processor assets. There are no external weight links or runtime download requirements.