GLM-5.3-Flash-EXL3 (4.0 bpw)

GLM-5.3-Flash — 153 GB on disk (down from ~643 GB BF16) — EXL3 quantization (exllamav3 trellis + random Hadamard) at 4.0 bits per weight. For the exllamav3 runtime on Hopper (H100/H200) and Blackwell (incl. DGX Spark).

• Source: zai-org/GLM-5.3-Flash — glm5_next: a multimodal hybrid model (vision tower + mixed linear/full attention + MoE), 46 text layers, 288 routed experts top-8, MTP head.
• Quantization: EXL3 (quant_method: exl3), 4.0 bpw decoder, 6 bpw head, 4 bpw MTP, mul1 codebook, calibrated on the exllamav3 standard corpus (128×2048). Built with exllamav3 1.4.4 (dev).
• Vision tower kept at 16-bit (-vb 16) — full multimodal quality retained.
• MTP included (layer 45) — speculative decoding ready.
• Verified: all 46 LM layers present, MTP + vision preserved (150,894 tensors, 23 shards).

Requirements

Requires ExLlamaV3 ≥ 1.4.5 or the dev branch (adds the glm5_next architecture). On DGX Spark (aarch64/GB10) the extension needs the small ARM CPU stubs — see the exllamav3 issues / DGX-Spark community recipes.

Usage (exllamav3 / TabbyAPI)

from exllamav3 import Config, Model, Cache, Tokenizer
config = Config.from_directory("JANGQ-AI/GLM-5.3-Flash-EXL3")
model = Model.from_config(config)
# ... standard exllamav3 load; see the exllamav3 examples

Or serve with TabbyAPI pointed at this model directory.

Method

Quantized on a single NVIDIA DGX Spark (GB10, sm_121) with exllamav3's layer-streamed converter — the model is processed module-by-module so the ~106B hybrid MoE fits the 120 GB unified memory. Routed experts + attention projections are EXL3-coded; norms, router gates, embeddings, the vision tower, and MTP glue are kept at source precision per the exllamav3 strategy. expandable_segments was required to avoid an OOM at the hyper-connection (hc_expand) step.

---

Quantized by JANGQ-AI.