brandonmusic/GLM-5.3-Flash-tr3-4bpw

GLM-5.3-Flash TR3 4bpw — current SM120 runtime

This is the uniform-K4 EXL3/TR3 routed-expert checkpoint for GLM-5.3-Flash. The current v84 runtime supports three explicit TP2/EP2/DCP2 profiles on two SM120 GPUs: multimodal DFlash2, language-only DFlash2, and language-only MTP3. All use calibrated NVFP4 MLA KV and CUDA graphs. This is a custom vLLM/B12X build and is not compatible with stock upstream vLLM.

Encoder reproducibility closure

The repository now contains the complete, hash-verified R10 Python encoder closure used by the EXL3/MCG adapter, including r7_encoder/r10_codec.py (R10TrellisCodec) and the pinned lineage/encode_tr3_v31.py numeric core. It is published under reproducibility/r10/ with a per-file SHA-256 manifest and an offline verifier:

python3 reproducibility/r10/verify_bundle.py

The bundle is byte-identical to the immutable prior-control source at Hugging Face revision 7c73450f05a151439d0f184f216b1eefcc394a31. It contains the portable Python/numeric source, not a compiled exllamav3_ext; that binary must still be built for the target PyTorch, CUDA, and SM ABI and is independently hash-bound by the adapter. See the bundle README for the exact adapter paths, lineage boundary, and licensing.

Pick a serving profile

Goal	Launcher	Extra checkpoint	Measured KV tokens
Images plus fastest measured C1 decode	compose.sm120-tp2.yaml	DFlash2-7	129,473
Text-only DFlash2 decode	compose.sm120-tp2-language-only-dflash2.yaml	DFlash2-7	184,619
Text-only capacity/default	compose.sm120-tp2-language-only.yaml	none; built-in MTP3	1,376,256

The MTP3 option means the model's built-in MTP head only: it does not load or mount the external DFlash checkpoint. Choose DFlash2 when its modest C1 decode gain matters more than resident context/concurrency; choose MTP3 for the normal text-only daily driver.

Run the current image

verdictai/glm53-flash-exl3-k4:r19-sm120-tp2-ep2-dcp2-v84-dflash2
OCI digest: sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692

The runtime image does not contain either checkpoint. Download/mount this EXL3 model and incoai/GLM-5.3-Flash-DFlash2 separately. The DFlash2 checkpoint is distributed under CC-BY-NC-ND-4.0; review its license before use.

Docker Compose:

curl -L -o compose.sm120-tp2.yaml \
  https://raw.githubusercontent.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/main/runtime/compose.sm120-tp2.yaml

GLM53_MODEL_PATH=/absolute/path/to/GLM-5.3-Flash-tr3-4bpw \
GLM53_DFLASH_PATH=/absolute/path/to/GLM-5.3-Flash-DFlash2 \
docker compose -f compose.sm120-tp2.yaml up -d

curl http://127.0.0.1:8012/v1/models

Standalone serve script:

curl -L -o serve-glm53-sm120-tp2.sh \
  https://raw.githubusercontent.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/main/runtime/serve-glm53-sm120-tp2.sh
chmod +x serve-glm53-sm120-tp2.sh

MODEL=/absolute/path/to/GLM-5.3-Flash-tr3-4bpw \
DFLASH_MODEL=/absolute/path/to/GLM-5.3-Flash-DFlash2 \
GPU_DEVICES=0,1 \
./serve-glm53-sm120-tp2.sh

The published profile has a 98,304-token request ceiling and allocated 129,473 KV tokens on the qualified pair. Its hybrid Mamba/DFlash rollback layout has room for one full resident request; additional requests queue. C2/C4 rows in the raw benchmark are therefore capacity-limited and are not throughput claims.

Language-only profile

For text serving, use the language-only profile. It disables the vision tower and uses the built-in MTP3 head by default, avoiding a second external checkpoint and leaving substantially more room for KV cache:

curl -L -o compose.sm120-tp2-language-only.yaml \
  https://raw.githubusercontent.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/main/runtime/compose.sm120-tp2-language-only.yaml

GLM53_MODEL_PATH=/absolute/path/to/GLM-5.3-Flash-tr3-4bpw \
docker compose -f compose.sm120-tp2-language-only.yaml up -d

Standalone:

curl -L -o serve-glm53-sm120-tp2-language-only.sh \
  https://raw.githubusercontent.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/main/runtime/serve-glm53-sm120-tp2-language-only.sh
chmod +x serve-glm53-sm120-tp2-language-only.sh

MODEL=/absolute/path/to/GLM-5.3-Flash-tr3-4bpw \
GPU_DEVICES=0,1 \
./serve-glm53-sm120-tp2-language-only.sh

To keep DFlash2 while disabling vision, use the separate speed-first launcher:

curl -L -o compose.sm120-tp2-language-only-dflash2.yaml \
  https://raw.githubusercontent.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/main/runtime/compose.sm120-tp2-language-only-dflash2.yaml

GLM53_MODEL_PATH=/absolute/path/to/GLM-5.3-Flash-tr3-4bpw \
GLM53_DFLASH_PATH=/absolute/path/to/GLM-5.3-Flash-DFlash2 \
docker compose -f compose.sm120-tp2-language-only-dflash2.yaml up -d

Its standalone equivalent is serve-glm53-sm120-tp2-language-only-dflash2.sh.

The language-only alias points to the same tested v84 code digest:

verdictai/glm53-flash-exl3-k4:r19-sm120-tp2-ep2-dcp2-v84-language-only
OCI digest: sha256:0f1cdcc8891f1cc3a444121eb61d366289a1cbba285f0892dcbb24bc94961692

Measured capacity on the same two 96 GB GPUs at 300 W each:

Runtime profile	Vision	Speculator	KV tokens	Concurrency at tested ceiling
Multimodal DFlash2-7, 98,304 max	on	external 7-layer draft	129,473	1.32x
Language-only DFlash2-7, 98,304 max	off	external 7-layer draft	184,619	1.88x
Language-only MTP3, 131,072 max	off	built-in head	1,376,256	10.50x

Turning vision off raises the DFlash KV token pool by 42.6%. The much larger 7.45x language-only gain comes from using the built-in MTP head instead of keeping the external DFlash2-7 model resident. It is not a vision-only gain. The language-only profiles do not accept image inputs. The reported KV-token pool is total allocated capacity, not a promise that every request can use the entire pool; the configured per-request ceiling and scheduler concurrency still apply.

Current measured results

Qualified on two RTX PRO 6000 Blackwell Workstation Edition GPUs (96 GB each), TP2/EP2/DCP2, NVFP4 MLA KV, prefix cache off, and DFlash2-7. The current quick speed pass used 600 W limits and +6000 MHz memory offsets. Generation uses the model defaults (temperature=1.0, top_p=0.95); the acceptance comparison uses reasoning_effort=max.

Measurement	Result
Cold prefill, 32K	6,225 client / 6,277 server tok/s
Cold prefill, 64K	6,083 client / 6,130 server tok/s
C1 decode, empty context	145.5 tok/s
C1 decode, 32K context	147.2 tok/s
C1 decode, 64K context	151.5 tok/s
DFlash2 acceptance, 5 distinct GSM8K prompts	5.428 mean / 5.441 token-weighted; 5/5 correct
DFlash2 acceptance, GSM8K first 16	5.739 mean / 5.550 token-weighted
DFlash2 acceptance, published reference	5.78 mean over 128 samples
Image smoke	pass — correctly identified a mallard

The clean C1 decode run used a 4,096-token completion budget so the client did not roll into the next prefill request. A prior 60.1 tok/s row was a harness rollover artifact and is excluded. The DFlash acceptance fix is material: the partially ported Triton mask scored 1.017 weighted. Restoring the reference semantics—full bidirectional visibility inside the draft block with a backward-only historical window—raised the same five-seed probe to 5.068 and the exact GSM8K sample to 5.739. Synthetic padded long-context decode accepts roughly 2.8–3.0 tokens/step, while five distinct GSM8K reasoning prompts accepted 4.89–6.03 and all answered correctly; acceptance is workload-dependent.

Receipts: 600 W prefill JSON, 600 W prefill TUI, clean 600 W C1 decode JSON, clean 600 W C1 decode TUI, earlier C1-C4 benchmark, acceptance rows, distinct-prompt acceptance, language-only capacity, and release validation.

Quality and KLD

v84 changes draft speculation, Triton draft-attention semantics, and vision packaging; it does not change target-model weights, EXL3 kernels, calibrated MLA KV scales, or target logits. The current target-quality receipts therefore remain the repeatedly qualified v75 measurements:

Test	Result
FP8 MLA KV KLD, five-run full 2,047-position mean	0.024610591221
NVFP4 MLA KV KLD, five-run full 2,047-position mean	0.054757372223
Estonia 10x, NVFP4	10/10
LAVD-low 10x, FP8	8/10 accepted
LAVD-low 10x, NVFP4	3/10 accepted — failed quality gate
Needle through 500K, NVFP4	17/18 raw; final cell passed on longer retry

KLD was measured in eager/no-speculation mode against the sealed BF16 teacher over every causal position in the 2,048-token window. Draft acceptance does not alter that target-logit measurement. Hotel was explicitly stopped and is not presented as a current result.

Receipts: v75 KLD and quality evidence. Older tuning history is retained in the historical model card, not mixed into the current launch path.

Vision and implementation notes

The image fixes a packaging defect where GLM-5.3 vision RoPE unconditionally imported vllm.vllm_flash_attn.layers.rotary even when a custom wheel shipped only the compiled flash-attention extensions. It now uses native PyTorch RoPE as a correctness fallback. Cold multimodal warmup and a real remote-JPEG chat request both passed.

The target path remains the fused uniform-K4 EXL3 route-128 SMEM/register kernel. SM120 in this build does not use a TMEM/TCGEN path. DFlash uses Triton attention because its noncausal sliding-window semantics are now tested there.

Provenance and attribution

The image embeds /opt/glm53/PROVENANCE.json and OCI source, author, documentation, revision, checkpoint, and validation labels. The manifest binds the runtime source and benchmark artifacts with SHA-256 hashes. This is a transparent provenance fingerprint: there is no telemetry, callback, hidden output watermark, or inference modification.

curl -L -o verify-provenance.sh \
  https://raw.githubusercontent.com/brandonmmusic-max/glm-5.3-flash-exl3-4bpw/main/runtime/verify-provenance.sh
chmod +x verify-provenance.sh
./verify-provenance.sh

This checkpoint is distributed under the ShapleyMCG License 1.0 in [LICENSE](LICENSE https://huggingface.co/brandonmusic/GLM-5.3-Flash-tr3-4bpw/blob/main/LICENSE Local Inference Lab contributors for the runtime foundation.