cont1037/GLM-4.6V-Flash-FP8-DYNAMIC

GLM-4.6V-Flash — FP8_DYNAMIC (judge VLM)

Derivative work. Modified from zai-org/GLM-4.6V-Flash (MIT). See LICENSE for the original licence and NOTICE for the changes made. Quantization only — no fine-tuning, no training data.

FP8 W8A8 quantization of the VLM that decides duels in the 404-GEN judge (glm-4.6v-flash). Per-channel FP8 weights, per-token dynamic FP8 activations, vision tower left in bf16.

• base: zai-org/GLM-4.6V-Flash @ 411bb4d77144a3f03accbf4b780f5acb8b7cde4e
• built by quant/quantize_fp8.py --scheme FP8_DYNAMIC --device cpu (17 s, no calibration)
• 12.45 GB on disk, 11.73 GiB resident under vLLM 0.23.0 (bf16: 20.6 GB / 19.29 GiB)

The ignore list is the load-bearing detail

llm-compressor expands ignore into concrete HuggingFace module names (model.visual.blocks.0.attn.qkv, 124 of them). vLLM's GLM-4V implementation names the same module visual.blocks.0.attn.qkv — no model. prefix — so the match fails and vLLM quantizes the bf16 vision tower at load. Nothing errors: text generation stays perfect and only image features are destroyed, so a schema-constrained judge returns correct JSON keys with garbage content.

This checkpoint ships prefix-agnostic regexes instead, which match under either naming:

["lm_head", "re:.*lm_head.*", "re:.*visual\\..*", "re:.*vision_tower\\..*"]

Verify before trusting any quality number from a rebuild:

/opt/vllm-glm-env/bin/python quant/probe_dtypes.py <ckpt-dir>
# must print: vision tensors stored as FP8: 0

This build reports 156 vision tensors bf16, 160 language tensors float8_e4m3fn, 0 vision FP8.

Every public compressed-tensors checkpoint of this model has the bug — cyankiwi/GLM-4.6V-Flash-AWQ-4bit (173 concrete vision entries) and alecccdd/GLM-4.6V-Flash-W8A8-INT8 (124), neither with a single re: pattern.

Serving

vllm serve <this-repo> --served-model-name glm-4.6v-flash \
    --max-model-len 81920 --max-num-seqs 196 --gpu-memory-utilization 0.9 --trust-remote-code

Do not pass --revision when serving a local directory.

Measured on 1x H200 (SM90), vLLM 0.23.0

Throughput, 401 frozen judge requests (~3,958 prompt tokens, ~69 completion tokens each):

	bf16	FP8_DYNAMIC
req/s @ concurrency 32	4.85	6.03 (+24%)
req/s @ concurrency 64	12.77	12.71 (−0.5%)
TPOT @ 32	69.5 ms	55.7 ms
model weights	19.29 GiB	11.73 GiB
GPU KV cache	2,460,880 tok	2,660,048 tok

The gain disappears at concurrency 64: the workload is vision-encoder-bound there, and this arm leaves the encoder in bf16.

Fidelity, 896 real duels across 7 complete matches, re-judged through the judge's own evaluate_duel and compared to the recorded competition outcomes:

	duel agreement	flips	mean |Δmargin|	max |Δmargin|	7% bar crossings
bf16	90.40%	86	0.0368	0.0547	1 of 7
FP8_DYNAMIC	80.02%	179	0.0458	0.1172	1 of 7

bf16 does not reproduce itself: dynamic batching changes GEMM reduction order, so re-judging the same duels moves match margins by 0.037 on average and flips one of seven qualification decisions. FP8 roughly doubles that. Read the FP8 row against the bf16 row, never against 100%.

Alternatives measured and rejected

arm	per-call direction agree	req/s @32	verdict
FP8_DYNAMIC	—	6.03	best of the quantized arms
W8A8-INT8	88.22%	6.17	same structure, ~5 pts worse — INT8's uniform grid fits activation outliers badly
W4A16-AWQ (192 calib rows)	70.95%	4.93	no speedup at all, penalty MAE 1.03 vs a 0.7-wide draw band
FP8_DYNAMIC + vision tower	—	6.14 / 11.02 @64	slower at production load and less accurate

Speed comes from 8-bit activations; fidelity comes from 8-bit weights. FP8 is the only scheme on the good side of both.

Note for anyone re-running AWQ here: GLM-4.6V-Flash uses GQA with 2 KV heads, so v_proj emits 256 channels against o_proj's 4096 input and that mapping is inexpressible — llm-compressor skips 40 of them, one per decoder layer. AWQ gets 2 of 4 mappings on this model, not 3.