TonyMFJ

/

Qwen3.8-Flash-Next-WinVulkan-GGUF

Like 0

Hi Eugene — thanks for the kind words, and sorry for the slow reply. Here are the real numbers from the reference machine (ROG Flow Z13, Ryzen AI MAX+ 395, 128 GB LPDDR5X, 96 GB VRAM carve-out, Windows 11, Vulkan backend).

One caveat up front: all of these were measured with the APU package power limited to 80 W. On a higher power profile expect a bit more, on a lower one less.

Generation (decode)

• 14.5 t/s with the shipped config (PLE table on disk, ubatch 2048) and MTP speculative decoding enabled — acceptance ≈ 67%
• ~28 t/s with everything resident in VRAM (short context, 93/96 GB used)
• Context length matters a lot: at 90K tokens you get ~14 t/s if the KV cache sits in host RAM, but ~30 t/s once the KV cache is in VRAM (-ctk q8_0 -ctv q8_0). That was the single biggest long-context lever we found.

Prefill: ~299 t/s at 24K tokens, ~219 t/s at 90K tokens.

Two requirements to actually hit those numbers:

1. --load-mode auto is mandatory on this fork under Windows — none crashes the process.
2. --ubatch-size 2048 is the ceiling with a 96 GB carve; 4096 fails (host malloc failure, or MTP draft buffer OOM).

Full environment (driver 32.0.31032.1003, w64devkit GCC 16.2.0, CMake 4.4.0, Vulkan SDK 1.4.309, exact build commands, pitfall list): https://github.com/TonyMFJ/Qwen3.8-Flash-Next-WinVulkan/blob/main/ENVIRONMENT.md

Hope that helps — happy to answer anything more specific.

Hi Eugene — following up with the shape of the curve, not just single numbers.

Reference machine: ROG Flow Z13 (Ryzen AI MAX+ 395, 128 GB LPDDR5X, 96 GB VRAM carve-out), Windows 11, Vulkan backend, APU package power limited to 80 W. Config: -c 102400 -b 2048 -ub 2048 -ctk q8_0 -ctv q8_0 (KV cache in VRAM) + MTP speculative decoding (adaptive n-min 2 / n-max 4) + PLE n-gram table on disk.

Prefill / decode curve (same session, one fresh random prefix per run):

Model size for those runs: huihui-ai/Huihui-Qwen3.8-Flash-Next-abliterated, UD-Q4_K_XL (Unsloth dynamic) — 4 shards, 111.3 GB total (49.86 + 49.38 + 12.09 GB + a 10.9 MB index shard), plus a 0.91 GB mmproj. The PLE n-gram table inside it is 28.8 GB and stays on disk via --ngram-on-disk, which is what keeps the whole thing inside 96 GB of VRAM (peak 90.4 GB).

The kit uploaded in this repo is a different quant of the same base (ROCmFP4-FAST-v2-ple16): 87.06 GB main + 20.86 GB PLE sidecar + 0.84 GB mmproj + 2.59 GB MTP draft = 111.35 GB total — same ballpark, ~28 % smaller main file.

Worth knowing before you compare against your own numbers:

• MTP is the decode lever, not a gimmick: without the draft model decode drops to ~6-8 t/s. It costs ~10 % prefill, because the draft context forces ubatch 4096 -> 2048 (3072 and 4096 both fail to allocate).
• Acceptance in the table is flattered by highly repetitive test text. On natural text we measure 59-83 % acceptance and 17-35 t/s decode.
• Prefill decays monotonically with context (~403 t/s at 4.5K -> ~221 t/s at 90K) — that is attention cost, nothing to tune away.
• Both a higher power profile and a lighter quant will move these numbers; pin the wattage before comparing.

Full environment (driver 32.0.31032.1003, w64devkit GCC 16.2.0, CMake 4.4.0, Vulkan SDK 1.4.309, build commands, pitfall list): https://github.com/TonyMFJ/Qwen3.8-Flash-Next-WinVulkan/blob/main/ENVIRONMENT.md
Tuning levers (MTP, KV placement, ubatch ceiling): https://github.com/TonyMFJ/Qwen3.8-Flash-Next-WinVulkan/blob/main/TUNING_NOTES.md

Happy to answer anything more specific — context length, quant, flag sets, whatever is useful.