Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED-GGUF

The balanced Qwen3.8-27B quant for AMD Strix Halo (Ryzen AI Max+ 395 / gfx1151). Measured, not marketed: one 15 GB file that serves 128K of context on the iGPU — full attention, no offload flags, speculative decoding built in.

Brand: PaoAI · community quant, not affiliated with Alibaba or the Qwen team.

Updated 2026-09-17 — engine update, weights unchanged. Re-measured on the Strix-Halo-tuned Vulkan fork (LaurentZuijdwijk/llama.cpp, commit 11bfe8a): prefill is +11 % to +33 % across 8K→128K at matching filled context, chain quality is unchanged (three runs, 100/100/100 · task 100), and the 128K window holds with integrity at every point. The model file itself did not change — same sha256, same 15.01 GB. See Update log. Prefill is ahead at every depth — that is the solid result of this update.

---

What is this? (30 seconds)

Qwen3.8-27B is a hybrid: 48 cheap linear-attention layers and only 16 full-attention layers that actually re-read your context. We compressed it organ by organ — precision where it matters (attention K/V, the MTP draft head, every norm), 4-bit FP4 where it doesn't. One 15 GB file, no offload flags, speculative decoding built in, intact from 8K to 128K of filled context.

The recipe (what's inside)

Model part	What it does	Format
Full attention Q/K/V/O (16 layers)	reads your prompt, tracks context	Q4_0_ROCMFP4_STRIX (~4.5 bpw)
⚡ Linear attention (48 gated-delta layers)	the fast path — constant-time state	ROCmFP4 (qkv, type 100)
FFN gate/up + down (dense)	the knowledge (biggest block: 9.2 GB)	ROCmFP4 (~4.5 bpw, imatrix)
✍️ Draft head (blk.64)	MTP speculative guessing	Q8_0 + FP32 (kept sharp)
📖 Token embeddings	vocab lookup	Q6_K (~6.6 bpw)
Norms, gates, glue	keep signals correct	FP32 (never touched)

Measured tensor sizes: FFN gate/up 6.16 GB · FFN down 3.08 GB · linear-attn qkv 1.42 GB · GDN state ops 0.81 GB · token embeddings 1.04 GB · output 0.68 GB · full-attn 0.62 GB · draft head 0.10 GB · norms <0.01 GB → 15.01 GB total.

Chain-test results (N=3, graded by execution)

We run a fixed coding exam where tasks are graded by execution, not opinion. Three identical runs per engine; we report medians.

Task (what it measures)	Published engine	Vulkan fork
Hunt — read lying code: trace it + find the planted bug	100 · 100 · 100	100 · 100 · 100
Automaton — implement novel rules + exact state trace	100 · 100 · 100	100 · 100 · 100
Ring buffer — multi-threaded code under stress	100 · 100 · 100	100 · 100 · 100
Task median	100	100

A same-session reference run of the published engine also scored 100/100/100 · task 100. Zero protocol events on either engine — quality is a straight tie.

Speed (decode, tokens/second, per phase)

Phase	Published	Fork	Ours, same session	Fork vs ours
Spec (planning)	19.2	20.4	19.3	+6 %
Hunt (analysis)	21.6	26.3	24.7	+6 %
Automaton (code writing)	23.2	29.8	26.1	+14 %
Ring buffer (code writing)	20.7	22.9	19.8	+16 %
Publish (long output)	23.2	27.5	25.2	+9 %

*The published column is from September. The defensible comparison is the last column — same session, same flags, engine the only variable: +6 % to +16 %.*

Does quantization hurt? (perplexity + KL test — measured 2026-09-17, published 2026-09-18)

Plain words: the chain test above can only tell you if the model still passes. This test asks a different question: did compressing the weights change what the model actually predicts? We ran the industry-standard check — llama.cpp's own llama-perplexity tool — comparing this 15 GB FP4 file against a full-precision F16 copy of the same model (built by us from Qwen's official BF16 release, sha-verified shards), reading the same ~295,000 tokens of standard Wikipedia text through both at 8K windows.

Metric	What it means	Result
Perplexity delta	how much worse the quant predicts text vs its own F16 parent (lower = better)	−1.0 % (6.587 vs 6.654 — statistically identical)
Mean KL divergence	average "extra surprise" per token vs the parent — 0 = identical	0.086 (median 0.029)
Same top token	how often both models pick the exact same next word	88.8 % of positions

Plain read: on standard text, the FP4 cut is statistically indistinguishable from the full-precision model — no measurable quality loss. The small difference that exists is concentrated in a tail of rare positions (~1 % of tokens shift noticeably), and the chain test above shows those shifts didn't cost task success. Not lossless in the mathematical sense — lossy in a way we cannot measure on standard text.

Reproducible: llama.cpp llama-perplexity, engine commit 11bfe8a, wikitext-2-raw, -c 8192, 36 chunks (~295K tokens). Reference F16 converted by us from Qwen's official BF16 release (sha-verified shards). Full data in our results file.

Context decay — what happens when the window fills (8K → 128K)

Depth	Verified fill	Fork prefill	Fork decode	MTP acc	Prev prefill	Prev decode	Prefill Δ	391
8K	8,172	306.0	31.1	0.70	229.9	21.9	+33 %	✓
16K	16,365	281.4	28.5	0.75	211.7	22.4	+33 %	✓
32K	32,784	243.7	28.6	0.83	186.9	20.5	+30 %	✓
64K	65,505	173.1	21.6	0.77	150.1	17.5	+15 %	✓
96K	98,304	138.5	14.0	0.58	125.0	16.5	+11 %	✓
128K	131,067	113.8	14.0	0.66	102.9	10.6	+11 %	✓
192K	196,587	84.0	10.7	0.70	—	—	beyond card	✓

Every fill is the number the engine itself reported (prompt eval time = … / N tokens), not a target. Flags identical on both engines, a fresh prompt at every point. 192K is a bonus row (single run, integrity ✓); no claim is made beyond 192K.

Plain read: prefill is ahead at every depth (+11 % to +33 %), and nothing breaks at depth — every point answered the integrity question correctly. Decode is not a clean win: better from 8K–64K, worse at 96K (14.0 vs 16.5 — reproduced on a repeat run), level at 128K.

Facts

Metric	Result
File size	15.01 GB (13.98 GiB), single file
sha256	3727594804afef8140165e36c0919c70a98b43751a3153d43dd10385c1f100fd
Load	full GPU (GTT ≈ 51.6 GB at 64K ctx), no special flags needed
Context	standard serve -c 65536; measured 8K→128K; 192K reached (integrity ✓)
Served with	MTP speculative decoding (n-max 4), flash attention, KV q8_0
State of the art here	100/100/100/100 chain median (3 runs, both engines) · 6/6 depth points integrity-pass (8K–128K) · 192K bonus pass
Weights	unchanged by this update — same sha256 as the original publish

Requirements — which llama.cpp build

This model uses two things stock llama.cpp doesn't have: the qwen3_5 architecture (hybrid linear + full attention + MTP) and the Q4_0_ROCMFP4_STRIX quant type (type 105, gfx1151-native).

• Recommended (these numbers): LaurentZuijdwijk/llama.cpp @ 11bfe8a, Vulkan, gfx1151.
• Previous engine: kingjones30/ROCmFPX, branch z13-main, commit dfeacaf2f — the build the original card measured.

Build	Error you'll see
stock llama.cpp	unknown model architecture: 'qwen3_5'
llama.cpp without the ROCmFP4 types	invalid ggml type 105. should be in [0, 43)

How to run (measured, locked)

llama-server -m Qwen3.8-27B-PaoAI-ROCmFP4-STRIX-BALANCED.gguf \
  --host 0.0.0.0 --port 8091 -ngl 99 -c 65536 \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -fa on --cache-type-k q8_0 --cache-type-v q8_0 --reasoning-budget 1500

Same flags for both engines. Raise -c for long context (up to 262144). Do not add -b/-ub 2048, and do not enable adaptive drafting — both were measured and rejected (see Known limits).

Thinking budget — --reasoning-budget caps how long the model thinks before answering:

Value	Behaviour
-1 (or omit)	unrestricted thinking, best quality, slowest
1500	what every number on this card was measured with
4000	noticeably deeper reasoning on hard tasks
12000	deep work — long waits before the answer starts

Known limits (honest)

• Depth costs speed: prefill 306 → 114 t/s and decode 31 → 14 t/s from 8K → 128K of filled context. The 16 full-attention layers re-read history for every token; the 48 linear layers stay cheap. Linear, no cliff — but plan for it at 100K+.
• Decode is not a clean win on the new engine: better at 8K–64K, worse at 96K (14.0 vs 16.5), level at 128K.
• Caveat on the comparison: the "prev" numbers were measured under an unrecorded power state; the fork's are at the AC performance profile — so the deltas blend engine + profile. The clean single-variable figure is +18–27 % prefill (4-arm test, identical fills, same-session reference).
• Not adopted, and why: adaptive drafting (1 formatting failure in 3 chain runs) · wide batch -b 2048 -ub 2048 (no gain, and it lost the GPU device at depth).
• Vulkan only: every number here is from the gfx1151 Vulkan backend on a Strix Halo iGPU. CPU builds and other GPUs are not comparable.
• Long thinking: a bigger --reasoning-budget means noticeably longer waits before the answer starts.
• Don't crank context blindly: a single 15 GB file, but GTT grows with context (≈51.6 GB at 64K). Raise -c in steps on a 128 GB box; don't merge shards.

Serving features explained (plain words)

Flag	What it does	Cost
--spec-type draft-mtp --spec-draft-n-max 4	the model's own draft head guesses 4 tokens ahead, the model verifies them in one pass — roughly 2× decode	needs the Q8_0 draft head we kept
-fa on	flash attention — faster prefill and decode	none
--cache-type-k/v q8_0	half the context memory	negligible
--reasoning-budget 1500	caps thinking so answers always arrive	complex tasks may need more
prompt caching (default)	follow-up turns prefill in milliseconds	none

Draft acceptance on the depth-sweep probes measured 0.58–0.83 on the fork — lower than the 0.86–0.90 on the original card. The sweep prompts are repetitive filler, which a draft head predicts poorly, so acceptance there is not comparable across tests. On the chain (real work) the fork decoded faster than the previous engine.

Update log

Date	What changed
2026-09-18	Added quant-loss measurement (perplexity + KL vs own F16 parent): no measurable loss. Card text only — weights untouched.
2026-09-17	Numbers re-measured on the STRIX Vulkan fork (11bfe8a). Prefill +11–33 % at matching fills; quality unchanged (3 runs 100/100/100); 128K holds with integrity; 192K bonus row. Weights untouched — same sha256, same 15.01 GB.
2026-09-16	Model published: 15.01 GB single file, MTP draft head, sweep measured on the ROCmFPX engine.

Earlier engine numbers are not repeated here — they were measured on a different build; the tables above are the current measurement. If you reproduce a number, quote the engine commit with it.

Fleet recipes

Serving configs and our benchmark battery: paoai-strix-recipes (GitHub).

Credits & license

• Qwen team (Alibaba) — Qwen3.8-27B base model (Apache-2.0)
• ggml-org / llama.cpp contributors — the engine, the quant formats, the Vulkan backend
• LaurentZuijdwijk — the Strix-Halo-tuned Vulkan fork measured on this card
• nathanw1014 — ROCmFP4 types, qwen3.5/MTP work, Strix Halo gfx1151 performance
• kingjones30 / charlie12345 — ROCmFPX format & recipes
• vcruz305 — converter branch work (qwen3_5 support path)
• unsloth — imatrix methodology reference
• deepu105 — same-box Strix Halo proof and serving-config intel
• antirez — GGUF layout reference
• PaoAI — the STRIX recipe (per-tensor cut/keep table), imatrix-guided builds, sweep benchmarking, chain test

Apache-2.0, same as the base model. Not affiliated with Alibaba or the Qwen team.