Qwen3.8-Flash-Next-Q4_K_XL-DN4

This is unsloth's UD-Q4_K_XL with one change: the 48 expert down projections are IQ4_NL instead of Q5_1. That takes the file from 111.3 GB to 102.5 GB. On the same text it measures as the same model.

I built it because I run this model on a 128 GB AMD Strix Halo (Ryzen AI MAX+ 395, Radeon 8060S) under Windows with llama.cpp on Vulkan. On that machine one process can put about 78 GiB on the GPU. UD-Q4_K_XL does not fit that unless you page or move expert layers to the CPU, and the next size down, UD-IQ4_XS, is a worse model. Nobody ships the size in between, so I made it. If your machine has room for something between 94 GB and 111 GB, this is for you.

Every number below was measured on that machine. The logs are in the method notes.

What changed

tensor group	size	UD-Q4_K_XL	this file
expert gate and up (ffn_gate_exps, ffn_up_exps)	44.4 GB	Q4_K, layer 2 at Q5_K	same
expert down (ffn_down_exps, 48 layers)	27.1 GB to 22.6 GB	Q5_1, 5 layers at Q8_0	IQ4_NL
n-gram table (per-layer embeddings)	28.8 GB	IQ4_NL	same
attention, linear attention, hyper-connections, shared expert, embeddings, output	5.5 GB	Q8_0	same
total		111.3 GB	102.5 GB

1,224 tensors in four shards, same split layout as the source. 48 tensors differ. Every other byte is unsloth's.

Why the down projections, and why IQ4_NL: ffn_down_exps is 640 values wide. That is not a multiple of 256, so none of the K-quants or I-quants with 256-value blocks can touch it. unsloth had to leave it at Q5_1, a 32-block type at 6 bits, which makes it the one big tensor group in the file with more bits than it needs. The 32-block family also has IQ4_NL at 4.5 bits. Moving the downs there saves 8.9 GB and changes nothing else. The same rule is why the 28.8 GB table (rows 160 wide) cannot get any smaller.

Quality

Same engine, same text (wikitext-2 test, 24 chunks of 2048 tokens, -b 2048 -ub 1024 -fa on), same machine.

file	perplexity	KL divergence vs UD-Q4_K_XL	same top-1 token
unsloth UD-Q4_K_XL, 111.3 GB	2.9117 ± 0.0363	reference	reference
this file, 102.5 GB	2.9165 ± 0.0363	0.028 ± 0.001	94.5 %
unsloth UD-IQ4_XS, 93.7 GB	2.998 ± 0.037	0.089	90.75 %

Perplexity moves 0.17 %, which is inside the error bar. The KL divergence is the number that matters: median 0.005, 99th percentile 0.34, one token in eighteen changes its top pick. IQ4_XS changes one in eleven and its tail is three times worse.

Two checks on that number. The reference had to run with eight expert layers on the CPU, because 111 GB does not fit my GPU. Scoring this file the same way gives 0.0277 and 94.8 %, so the CPU kernels add nothing and the 0.028 is the quantization itself. And for scale: the q4_0 KV cache that I run every day puts 0.026 between this file and itself, and I have never noticed it.

The imatrix made no difference

The downs here were requantized from unsloth's Q5_1 without an importance matrix. I expected that to be the weak point, so I tried it: the same 48 tensors requantized with unsloth's published imatrix_unsloth.gguf_file, everything else identical.

downs requantized	KL divergence vs UD-Q4_K_XL	same top-1
without the imatrix (this file)	0.0282 ± 0.0007	94.50 %
with the imatrix	0.0277 ± 0.0007	94.60 %

Inside one error bar on every statistic except the single worst token. From a 6-bit source the IQ4_NL grid sets the error and the importance weighting has nothing left to steer. I deleted the imatrix build and kept this one. I guess a rung quantized from the BF16 checkpoint with the imatrix would still be the right way to make this file, though.

Running it on a 128 GB Strix Halo, Windows, Vulkan

This is what serves on my machine every day, at 262144 context, the whole transformer on the GPU, no expert layers on the CPU:

llama-server -m Qwen3.8-Flash-Next-Q4_K_XL-DN4-00001-of-00004.gguf \
  -c 262144 -ngl 99 -fa on -ctk q8_0 -ctv q8_0 -ub 512 -np 1 \
  -lm mmap --lazy-mode on --cache-ram 0 --ctx-checkpoints 8 --jinja \
  -md mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf --spec-type draft-mtp,ngram-mod \
  --spec-draft-n-max 4 --spec-draft-p-min 0.75 \
  --spec-ngram-mod-n-min 16 --spec-ngram-mod-n-max 128 --spec-ngram-mod-n-match 48

Memory mode: dynamic settings with the shared memory slider at its 124 GB maximum. The driver then lets one process address about 78 GiB on the GPU. This file lands at 77.9 to 78.5 GiB with the q4_0 cache and the Q4_K_M draft head. If it ever dies at warm-up, -ub 256 frees 1.7 GiB. I tested that fallback and it works.

The table streams from the file. -lm mmap --lazy-mode on keeps the 28.8 GB n-gram table out of committed memory and reads rows as needed. It measured the same as a resident table. mlock cannot pin a region this big on Windows, so do not bother.

The draft head is unsloth's mtp-Qwen3.8-Flash-Next-shared-Q4_K_M.gguf. Those are Qwen's own MTP weights, exported by unsloth. The head borrows embeddings from the main model by shape, so it pairs with this file unchanged. It is worth about 50 % more speed on fresh text. The n-gram drafter adds about 13 % on edits.

The KV cache. I serve it at q8_0 on both sides. Against an f16 cache that costs 0.012 of KL divergence at 2048 tokens; q4_0 on both sides costs 0.026 and keys q4_0 with values q8_0 costs 0.019. I measured all three at depth under real work, attended and with a power log beside them, and none of them changes decode speed: about 19.6 / 18.5 / 17.1 / 14.8 tokens per second in the under-25k, 25 to 50k, 50 to 100k and 100k+ bands for both q8_0 and q4_0. q8_0 needs 1.5 GiB more than q4_0 at 262k, which puts this file at 79.9 GiB on my GPU, over the nominal 78.4 GiB budget, and it serves. If yours does not, q4_0 is the smaller cache and -ub 256 frees 1.7 GiB.

One warning from that testing that has nothing to do with the cache: this tablet cuts its power limit to about 15 W whenever nobody is using it, which halves everything. If your Strix Halo runs this as a server over Remote Desktop, keep the session attended or disable Modern Standby, and never trust a benchmark taken while you were away.

Speed on my machine. Prefill about 180 tokens per second at shallow depth, 120 with the MTP head on a 22k prompt. Decode 28 to 32 on fresh text with the head, 45 to 50 on edits with both drafters, 11 to 13 past 200k tokens of context. On fresh text this file runs about 20 % slower than the source did in a fixed 96 GB carve-out, because IQ4_NL kernels cost more per step on this driver than Q5_1. That is the one price.

Vision. unsloth's mmproj for this model works with it. On this driver run the projector on the CPU with --no-mmproj-offload.

Engine used throughout: unsloth's llama.cpp fork build b10715 with PR 27836 and unslothai#142 for MTP. Mainline llama.cpp loads the file the same way, as long as your build knows the qwen4exp architecture.

Files

file	bytes	sha256
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00001-of-00004.gguf	10,946,624	4448186216b3af4cc558bbce2c3213f01608f8f8b2e5267a9767971dd3ec8082
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00002-of-00004.gguf	47,447,858,336	9904da25dd2dc97c1756ac88e0d56ff30012b65e890161b90b14b378a7d3d883
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00003-of-00004.gguf	44,552,691,904	2685325f5dfab3e4abad9f9519d1236b31b112b2bd7e5a3ff0afefd34e91b4cd
Qwen3.8-Flash-Next-Q4_K_XL-DN4-00004-of-00004.gguf	10,462,690,720	e656c8d08713915c9005ee2a631e8c02efa6c5914cc5c492c97e2a1c5ff82f92

Total 102,474,187,584 bytes, 102.47 GB, 95.43 GiB. Shard 1 holds the metadata and the split header only, as in the source.

How I built it, and how to build the next one

method/ next to this card has the three scripts (plain Python, no numpy) and the exact commands. The short version:

1. gguf_tensor_map.py reads a GGUF's real composition from its header alone, from a local file or straight from a Hugging Face URL. Judge a quant by this map, never by its name.
2. gguf_extract.py pulls the 48 down tensors out of the four source shards into one loadable GGUF. 29 GB, three minutes.
3. llama-quantize --allow-requantize --tensor-type ffn_down_exps=iq4_nl downs.gguf downs-iq4nl.gguf IQ4_NL, about 25 minutes. The explicit --tensor-type is not optional. With the IQ4_NL file type alone, llama-quantize picks q5_K for the downs, hits the 256-block rule, and silently falls back to q5_1. You get the same file back and a success message. Read the log.
4. gguf_splice.py writes a new shard set. Every tensor is copied byte for byte except the 48 replacements, with offsets recomputed. Eleven minutes. The originals are never touched.

The same recipe works on any tensor whose width is not a multiple of 256, on any model.

Credits and license

Qwen made the model. It is under the Qwen Community License 1.0, included here as LICENSE, as the license asks for derivatives.

unsloth made UD-Q4_K_XL, which is every tensor in this file but 48, and exported the MTP head. Their choice to keep attention, the linear attention blocks and the hyper-connections at Q8_0 is what makes this model's 4-bit quants good. My measurements say those small tensors decide quality more than the expert bits do.

I built and measured this on my own machine. I am not affiliated with Qwen or unsloth.