DevQuasar

/

zai-org.GLM-5.3-Flash-GGUF

Like 11

Follow

DevQuasar 454

Thanks for these builds — the expert path in them is unusually clean, and that turns out to matter a lot for CPU-offloaded serving.

Inspecting the tensor tables (HTTP range reads, no download), your Q2_K and Q3_K_M are 0% i-quant across the expert tensors:

That's the interesting property. On a CPU-offloaded MoE the bottleneck is weight reconstruction rather than bandwidth, and K-quant blocks are roughly 2.1× cheaper to dequantise than i-quant blocks of similar size. Most GLM-5.3-Flash GGUFs in this size class are i-quant-dominant regardless of what the filename suggests, so a genuinely K-quant expert path at ~100 GiB is rare and worth having.

Also worth noting for anyone else reading: the Q2_K build declares block_count = 45 and is self-consistent — it is not a 46-block model with a missing block, so nothing to trip a loader.

Edited to correct the original request. I first asked you to preserve the blk.45.nextn.* tensors (nextn_predict_layers=1, block_count=46), on the assumption that dropping them ruled out speculative decoding. That was wrong, and I'd rather fix it here than leave a misleading ask on your tracker.

Checking the engine side: an external DFlash2-style drafter does not need the target's nextn head at all. It reads hidden states from three ordinary layers of the target, with the layer indices supplied by the drafter's own metadata — only the built-in MTP drafter requires nextn embeddings from the target. So your builds are already usable with external speculative decoding exactly as they are.

That leaves the nextn head as a nice-to-have rather than a blocker: it would only matter to someone wanting built-in MTP specifically. Please treat it as low priority, or skip it — I don't want to send you after a rebuild on a mistaken premise.

The K-quant expert path is the valuable part, and it's already there. Happy to report numbers once I've benchmarked Q2_K against the i-quant builds on CPU-offloaded hardware.