Capicua25x/Qwen3.8-27B-MXFP4-Quark-RDNA4

Qwen3.8-27B — MXFP4 (AMD Quark) for RDNA4

MXFP4 weight quantisation of Qwen/Qwen3.8-27B, built with AMD Quark 0.12.post1 and targeted at RDNA4 (gfx1200/gfx1201: Radeon AI PRO R9700, RX 9070 XT) — GPUs that sit outside the official ROCm vLLM target list.

Code	github.com/Capicua25x/vllm-rocm-rdna4 — branch rdna4-port-0.26.1 · the kernel is rdna_fp8.py
Image	hub.docker.com/r/capicua25x/vllm-rocm-rdna4 — :0.26.1-rdna4-rc6 (= :latest)
Base model	Qwen/Qwen3.8-27B (Apache-2.0)
Port notes	RDNA4-PORT.md — quantisation recipes for other families, and the attribution record

What this buys you on 2×32 GB RDNA4: the full 262,144-token context window at roughly the single-stream speed of stock FP8, and markedly better throughput once prompts get long — see Throughput. On quality it is level with the bf16 reference overall, not uniformly above it: ahead on GPQA and τ²-airline, level on AIME and long-context retrieval, behind on τ²-telecom and on GSM8K flexible-extract. Every one of those is stated below with its caveats rather than summarised away.

What is and is not quantised

Two things Hugging Face's sidebar gets wrong on this repo, both from one cause. HF has no 4-bit dtype, so MXFP4 weights are stored in U8 — two 4-bit values per byte — and HF's automatic tagging reads the container rather than the contents:

• "8-bit" is wrong; the weights are 4-bit. The auto-tag comes from the U8 dtype. Every MXFP4 repo on the Hub gets it, including AMD's own amd/Qwen3.8-27B-Quark-AWQ-MXFP4. It cannot be suppressed, so this card declares 4-bit explicitly alongside it.
• "16B" is not a smaller model. HF sums safetensors elements: 12.05B packed U8 bytes + 5.11B BF16 = 17.15B. Unpacked, those bytes carry 22.7B parameters, giving ~27.8B logical — Qwen/Qwen3.8-27B, unchanged in architecture and parameter count. Only the storage precision of the MLP and linear-attention projections differs.

Only MLP and MoE-expert projections go to 4-bit. Attention (q/k/v/o and its norms), every norm, embeddings, lm_head, routers/gates and the entire vision path stay bf16.

	count
mlp.{gate,up,down}_proj	192 (64 layers × 3)
linear_attn.{in_proj_qkv, in_proj_a, in_proj_b, in_proj_z, out_proj}	240 (48 layers × 5)
total quantised modules	432
attention / norms / embeddings / lm_head / vision	0 — verified, none

Verified by tensor inspection: a module counts as quantised only if it carries a real artifact (weight_scale, weight_packed, qweight, weight_zero_point).

Keeping attention in bf16 is deliberate: the MLP stack is where the parameters are, so excluding attention costs little size and keeps those layers on the fast bf16 path.

For structural comparison, amd/Qwen3.8-27B-Quark-AWQ-MXFP4 quantises the decoder's attention as well — 496 quantised modules against 432 here, the difference being exactly the 16 full-attention layers' q/k/v/o — and is AWQ-calibrated (algo_config.name = awq) where this build is data-free RTN. Those are the two real differences. We have a single unrepeated n=50 GSM8K run against that build, which is not enough to publish a quality comparison from: strict-match moves by about ±0.06 across seeds on this hardware, which is wider than any gap it showed.

• Format: MXFP4 (E2M1 + E8M0 scale per 32 weights), pack_method: reorder, weight_format: real_quantized
• Size: 22.3 GB across 18 shards, from a 55.6 GB bf16 source — a 2.5× reduction. If that looks large for a 4-bit 27B, it is because only the MLP and linear-attention projections are 4-bit: 11.34 GB of packed U8 (22.7B parameters) + 10.22 GB of BF16 (5.11B parameters kept at full precision) + 0.70 GB of scales. A whole-decoder build is smaller — amd/Qwen3.8-27B-Quark-AWQ-MXFP4 is 19.8 GB — and pays for it in attention precision and speed (see above)
• Quark exclude list: 231 entries

The config declares W4A4, not weight-only. Quark's mxfp4 scheme enables dynamic fp4 activation quantization by default, so global_quant_config.input_tensors reads {dtype: fp4, is_dynamic: true, per_group, group_size 32, e8m0}. On the RDNA4 port that declaration is not honoured — the weight-only kernel ignores activation quant, and the FP8-WMMA kernel uses its own per-(token, 32-K-group) dynamic e4m3. If you load these weights on a runtime that does honour it, you will get a different numerical path than the one measured here. The difference from whole-decoder AMD-style builds is coverage (432 quantized modules vs 496, the delta being the 16 full-attention layers' q/k/v/o), not activation width.

Serving

Speed and window claims here need the RDNA4 port, which has the MXFP4×e4m3 FP8-WMMA kernel:

docker run --rm -it --device /dev/kfd --device /dev/dri \
  -v /path/to/weights:/model:ro -p 8011:8011 \
  -e VLLM_RDNA_MXFP4_FP8=1 \
  capicua25x/vllm-rocm-rdna4:0.26.1-rdna4-rc7 \
  serve /model --served-model-name qwen --port 8011 \
  --tensor-parallel-size 2 --max-model-len 262144 --trust-remote-code

Image: capicua25x/vllm-rocm-rdna4 on Docker Hub (public, no login needed). Source: Capicua25x/vllm-rocm-rdna4, branch rdna4-port-0.26.1. VLLM_RDNA_MXFP4_FP8=0 falls back to the weight-only bf16-unpack kernel.

Provenance of the benchmark runs: the cells below were served from the rc5 image with the two kernel files (mxfp4/rdna_fp8.py, kernels/linear/__init__.py) bind-mounted from the working tree — an iteration workflow that avoids a 37 GB rebuild per edit. Both files are byte-identical to the ones inside rc6 (sha256 2dc458c3… and a423cce9…), so pulling rc6 gives you exactly the code that produced these numbers. rc7 is rc6 plus one attention change that is inert unless you pass --kv-cache-dtype fp8, so these numbers stand on it too.

On stock vLLM these weights load and generate correctly, but slower. Without the FP8-WMMA kernel you get the weight-only dequant path — roughly 51 tok/s single-stream instead of 61 on this hardware — and on 32 GB cards you will not reach the 262k window. If you are benchmarking this against another quant, check which kernel you are actually on first.

Sampling follows the base model card: thinking temp 1.0, top_p 0.95, top_k 20, min_p 0; non-thinking temp 0.7, top_p 0.8, top_k 20, presence_penalty 1.5.

KV-cache scales (added 2026-08-18)

The weight shards are unchanged. This repo additionally carries 51 F32 KV-cache scalars in model-kvscales.safetensors (5,708 bytes) plus matching model.safetensors.index.json entries — {q,k,v}_scale for each of the 16 full-attention layers and for the MTP draft layer.

They do nothing unless you serve with --kv-cache-dtype fp8. Under the default bf16 KV cache — which is every number elsewhere on this card — they load and sit unused. Without them vLLM falls back to k_scale = v_scale = q_scale = 1.0 and logs Using KV cache scaling factor 1.0 for fp8_e4m3: correct bytes, wrong numbers. With them, that line is absent, and its absence is the only reliable proof they loaded — .q_scale/.k_scale/.v_scale are in vLLM's _ignore_unexpected_suffixes, so a mis-named scale is dropped silently instead of erroring.

Serving fp8 KV wants rc7 or newer. Upstream gates fp8-query input to CUDA, so on ROCm the unified kernel dequantizes K/V (fp8 → f32×scale → bf16) inside the KV loop — once per (query block × KV tile). Measured here on gfx1201: that in-loop path is 2.6–7× slower in decode attention than keeping K/V/Q in e4m3 through the FP8 WMMA, and prefill drops 3.2K → 1.75K tok/s. rc7 enables the fp8-query path on ROCm (kill switch VLLM_RDNA_FP8_Q=0). On rc6 or stock vLLM these scales still load and are numerically correct — you just pay the slow path, which is easy to mistake for "fp8 KV is bad on RDNA4". It isn't; at decode shapes fp8 KV attention measured 37–47% faster than bf16 KV once the query path is enabled.

How they were captured. Served at --kv-cache-dtype auto with a recorder on Attention.forward taking a running amax of q/k/v as the cache sees them — post q_norm/k_norm, post-RoPE — over 1.4M tokens / 170,905 attention calls: 354 requests spanning long documents (8k–96k), multi-subject QA, maths, code, instruction-following and a Spanish business slice, plus production traffic. Both tensor-parallel ranks recorded, per-layer maximum taken. scale = amax × 1.1 / {200, 200, 100} for q/k/v, matching vLLM's Q/K/V_SCALE_CONSTANT.

	q	k	v
range over the 16 attention layers	11.50 – 19.50	12.19 – 23.12	8.44 – 147.00
MTP draft layer	19.50	20.88	51.50

v grows monotonically with depth (12.06 at layer 3 → 147.00 at layer 63).

What this is worth — read this before assuming it is an upgrade. The largest amax anywhere is 147 against e4m3's maximum of 448, so the uncalibrated cache was never clipping. The scales shift where the distribution sits rather than rescue it, and on a float format that is a small effect — consistent with the published finding of ≤0.7 points from an uncalibrated fp8 KV cache on a model of this class. Treat this as hygiene that closes a question, not as a quality fix.

Not measured, not covered:

• No accuracy re-run of the calibrated build (GSM8K / long-context / task suites). The numbers under Quality below are the bf16-KV build and are unaffected by this addition.
• The vision tower is not calibrated — the capture corpus is text-only.
• No prob_scale is shipped. No v1 ROCm attention backend consumes one, so vLLM's …and/or prob_scale 1.0 with fp8 attention warning still fires and can be ignored.
• fp8 KV doubles the attention block on this architecture. vLLM aligns the attention page to the mamba page, so halving the bytes per token pushes the block from 800 to 1600 tokens, which coarsens prefix-cache reuse: only whole blocks are reused. If your requests share a long prefix, pad it to a multiple of the block size — otherwise the remainder is recomputed on every request. Measured here on a 7,299-token prompt: 99 tokens recomputed at block 800 vs 899 at block 1600.

What it took to run this on RDNA4

These weights are the small half of the work. RDNA4 (gfx1200/gfx1201) is outside the official ROCm vLLM target list and has no hardware microscaling datapath — nothing on the chip consumes MXFP4 directly. Making a 4-bit checkpoint fast here took ~1,230 lines across the serving stack: a new FP8-WMMA kernel, a weight-only fallback kernel, an attention-gate fix, RDNA4 dispatch in two quantisation paths, and a differential test.

The starting point was Rob Smith's (tcclaviger) gfx1201 enablement on the vLLM 0.18.1 line — gfx1201 hipBLASLt plus MXFP4/NVFP4 MoE kernels. That base is why any of this runs, and it is credited in detail in the repo's NOTICE, which marks file by file what is forward-ported from it and what is not. What this project added on top:

• Two major version lines of forward-port. 0.18.1 → 0.19.1 → 0.26.1. vLLM's kernel registration, quantisation config plumbing and attention backends all moved underneath in that span; the RDNA4 paths had to be re-landed against each. His source is no longer published, so this branch is what carries the lineage forward.
• A kernel that did not exist before. His _matmul_fp8_ogs is W8A8 — 8-bit weights, so there is nothing to unpack. RdnaMxfp4Fp8LinearKernel is W4 × A8: it unpacks each E2M1 nibble into its exact e4m3 bit pattern inside the GEMM, carries an E8M0 block scale per 32 weights, and dispatches across three shape regimes. The borrowed idea is the per-K-group scale fold on WMMA v2; the 4-bit half is new.
• The quantisation toolchain and the model portfolio — Quark adoption, per-family exclude policies derived from each architecture's tensor index, model bring-up, serving recipes, and the paired benchmark campaign these numbers come from.

Original to this port:

• RdnaMxfp4Fp8LinearKernel — MXFP4 weights × e4m3 activations on RDNA4's FP8 WMMA, which runs at roughly twice the bf16 rate. It dispatches by shape rather than using one path: at decode (M ≤ 128) a fused Triton GEMM builds each E2M1 nibble as its exact e4m3 bit pattern, does tl.dot per 32-wide K block, and applies the E8M0 block scale to the fp32 partial after the dot; mid-M falls back to weight-only; prefill (M > 512) does an exact integer dequant to bf16 into a reused scratch and hands it to hipBLASLt. Weights never materialise in high precision. VLLM_RDNA_MXFP4_FP8=0 disables it; VLLM_RDNA_MXFP4_FP8_SKIP=<prefix,…> keeps named layers on bf16 activations. 336 lines, with a differential test against a reference dequant implementation pinning correctness.
• A gate relaxation on upstream's 3D split-KV attention path so small-q speculative-decode verify shapes can use it (MAX_QLEN_3D, token-count capacity check) — 11 lines on a kernel by Burkhard Ringlein, Jan van Lunteren, Chih-Chieh Yang and Thomas Parnell. Without it, an MTP verify of q_len=4 over ~7k KV fell to the 2D grid and lost most of its throughput. Arrived at independently; equivalent relaxations were proposed upstream first in vllm-project/vllm #44652, #45450 and #46724, none merged as of 2026-08.
• RDNA4 dispatch in mxfp4_utils.py and compressed_tensors_moe_w4a4_mxfp4.py, so MXFP4 weights select the RDNA value layout and the unfused Triton experts instead of CDNA swizzles and Marlin repack ops that do not exist on gfx12xx.

Targeting AMD Quark was a deliberate change of quantisation toolchain, not just a file format. The earlier in-house pipeline was RTN; everything now goes through Quark and ships as MXFP4. Two things that took work rather than reading the docs: Quark has no template for this architecture, so the model type must be registered with LLMTemplate before direct_quantize_checkpoint or it raises; and Quark matches its exclude list with fnmatch, which does not expand braces — a pattern like *.self_attn.{q,k,v,o}_proj silently matches nothing and quantises attention anyway. The per-family exclude lists, the two build paths (file-to-file vs the stacked-3D-expert path that needs preprocess_for_quantization called explicitly), and the verification method that distinguishes real quantisation artifacts from lookalike tensors are all written up in RDNA4-PORT.md.

Throughput

Measured on this exact artifact on 2× Radeon AI PRO R9700 (TP2, gfx1201) with the rc6 FP8-WMMA kernel (VLLM_RDNA_MXFP4_FP8=1) and native MTP-3 speculative decoding. max_tokens: 256, thinking on — the shape most deployments actually run. Compared against stock Qwen/Qwen3.8-27B-FP8 at matched capacity: both hold a 262,144-token window on the same two cards.

Every cell below names its sweep log. An earlier version of this section published different 6k figures (MXFP4 199/260 aggregate, FP8+fp8-KV 79/87). Those are corrected, and the reason is worth stating precisely rather than vaguely:

• The 79/87 column was real, but mislabelled. It is the bench's "B before overlay (upstream ROCm path)" measurement — stock FP8 with fp8 KV served without the fp8-query attention path, so the kernel dequantized K/V inside the KV loop. The card presented it as plain "FP8 + fp8 KV". The same arm with that path measures 147/179 at 6k c8/c16, and that is what the table below now shows.
• The MXFP4 column was an earlier sweep, since superseded by a re-run that overwrote the log of the same name. The values below are from the surviving re-run.

No data was lost and nothing was fabricated; the card had drifted from the bench record. Each column below names the log it came from so this stays checkable.

Short prompt (~30 tokens) — per-user tok/s / aggregate tok/s. Sources: conc_q38mxq_short_thinkon.log (2026-08-18 00:52), conc_q38fp8kv8_short_thinkon.log (08-17 23:49).

concurrent	MXFP4 + bf16 KV (this build)	FP8 + fp8 KV
1	47.7 / 48	46.2 / 46
4	41.1 / 146	41.4 / 158
8	34.4 / 249	37.4 / 268
16	27.1 / 401	26.9 / 390
32	18.2 / 546	21.2 / 439
64	11.6 / 472	14.7 / 440

6k-token prompt — closer to a real application's context. Sources: conc_q38mxq_6k_thinkon.log (00:54), conc_q38fp8kv8_6k_thinkon.log (23:51).

concurrent	MXFP4 + bf16 KV (this build)	FP8 + fp8 KV
1	35.8 / 36	45.5 / 46
4	30.8 / 118	28.5 / 109
8	24.4 / 190	19.1 / 147
16	17.1 / 262	11.6 / 179
32	10.4 / 274	7.6 / 181

Reading this honestly. At short prompts the two are close, with stock FP8 slightly ahead in aggregate from 4–8 concurrent and this build ahead at 16. At 6k the picture splits by regime: stock FP8 is faster single-stream (45.5 vs 35.8 — it does not pay MXFP4's unpack cost, and one request cannot hide it), while this build scales better (190 vs 147 aggregate at 8 concurrent, 262 vs 179 at 16). If you serve one user at long context, stock FP8 is the better single-stream engine; if you serve several, this build wins on aggregate and holds a usable per-user rate further up the curve.

This is a two-variable comparison — weight format and KV dtype differ — so it does not isolate the cost of fp8 KV. Two confounds work against the fp8-KV column and are worth knowing before drawing conclusions from it:

1. fp8 KV doubles the attention block on this architecture. Halving the bytes per token pushes vLLM's mamba-aligned attention block from 800 to 1600 tokens, and prefix caching reuses whole blocks only. On this exact 6k prompt (7,299 tokens, 7,263 shared) the fp8-KV arm recomputes 899 prompt tokens per request against 99 — a 9× handicap unrelated to fp8 arithmetic.

2. The fp8-query attention path matters more than the KV dtype. Where supports_quant_query_input is CUDA-only, ROCm dequantizes K/V inside the KV loop; at decode shapes that path measures 2.6–7× slower than keeping K/V/Q in e4m3 through the FP8 WMMA. rc7 enables it — see KV-cache scales.

The single-variable measurement, same weights and same night, only the KV dtype changing (this MXFP4 build with calibrated scales, think-on, aggregate tok/s):

cell	bf16 KV (block 800)	fp8 KV (block 1600)	gap
--prompt-tokens 6000 c8	206	134	−35%
--prompt-tokens 5329 (block-aligned) c8	201	167	−17%
--prompt-tokens 5329 c16	263	201	−24%

The bf16-KV arm is flat across both lengths (99 vs 92 recomputed tokens), so the aligned cell is a fair control and not an easier workload. Roughly half the headline penalty is block alignment, not fp8 KV — and it is avoidable by padding a long shared prefix to a multiple of the block size. In exchange fp8 KV buys 2.03× the KV pool (666,343 vs 328,635 tokens at this window).

Per-user rates below ~20 tok/s fall under a usable interactive floor.

Thinking-off is measured but not tabulated here (conc_*_thinkoff.log in the bench repo); the shape of the result is the same. Figures published elsewhere for the rc6 kernel (61 tok/s single-stream, 649 aggregate) were measured on an earlier MXFP4 build of this model, before this Quark build existed — they do not describe this artifact and are omitted rather than borrowed.

Quality — measured, as of 2026-08-18

Same harness, same seed (1234), same on-spec sampling across all four columns. What each column actually is:

column	weights	KV	window / slots	where it ran
bf16 reference	qwen/qwen3.8-27b, unquantised	bf16	262k (theirs) / 8	hosted by AkashML, reached through OpenRouter — provider pinned, fallbacks off, so every reference number came from that one endpoint
FP8 + bf16 KV	Qwen/Qwen3.8-27B-FP8 (vendor)	bf16	131k / 16	this box
FP8 + fp8 KV	Qwen/Qwen3.8-27B-FP8 (vendor)	fp8	262k / 16	this box
MXFP4 (this)	this repo	bf16	262k / 32	this box

"this box" is 2× Radeon AI PRO R9700 (32 GB each), TP2, on the RDNA4 port with MTP-3 speculative decoding. The reference is a hosted third-party endpoint, not a local bf16 run — bf16 weights for a 27B leave no room for a usable KV cache on a 64 GB pair, which is the whole reason this quantisation exists. Two consequences worth carrying into every row: the reference's behaviour is AkashML's serving of the model as much as the model's own, and we cannot separate the two from the outside. That endpoint returns empty content on some items — 1 of 30 on AIME, 3 of 60 on GPQA, 33 of 120 on HLE — and each scores zero, which depresses the reference and flatters everything compared against it. On GPQA those failures are measured as finish_reason=error, i.e. the endpoint dropping a very long generation. The HLE empties are now measured to be the same thing (2026-08-18): that run predated the instrumentation, so two of its 33 empty items were re-run with the identical request shape — same model, provider pinned to AkashML with no fallbacks, same sampling and seed — and both reproduced it, ending finish_reason=error with zero content after 11,767 and 74,076 characters of reasoning (3,172 and 19,073 completion tokens; the second ran 32 minutes). With the two GPQA documents that failed the same way at 128,242 and 254,497 reasoning characters, that is four for four. Two things rule out the alternatives without spending anything: none of the 33 empties has completion_tokens == 0, so they are not dropped connections, and their token counts span 19,022–81,884 with no clustering at any cap, so they are not truncation. Each one generated tens of thousands of tokens and emitted no answer. This is the endpoint terminating a long generation, not the model declining to answer — the scored zeros are an artifact of the serving path, and the reference's HLE cell is a lower bound. That is a property of this measurement, not a claim about AkashML's service generally: a different provider, or bf16 run locally, could land elsewhere. Where it changes a margin, the footnotes say so.

The FP8 columns differ from each other only in KV cache dtype, so the gap between them is the cost of fp8 KV, not of anything else.

	bf16 reference AkashML (hosted)	FP8 + bf16 KV local, 131k	FP8 + fp8 KV local, 262k	MXFP4 (this) local, 262k
Math & reasoning
Grade-school math, thinking GSM8K · n=50 · flex / strict	0.96 / 0.82	0.96 / 0.90	0.94 / 0.70	0.94 / 0.92 c
Grade-school math, no thinking GSM8K · n=50 · flex / strict	0.98 / 0.98	0.98 / 0.98	0.98 / 0.98	0.98 / 0.98
Graduate-level science GPQA-Diamond · n=60 · flexible	0.8000 d	0.8333	0.8333	0.9167
Competition math AIME 2025 · n=30	0.9333	1.0000	0.9667	0.9333
Frontier exam questions HLE · n=120 · judge-scored	0.3000 e	—	—	0.2417
Instruction following
Verifiable instructions IFEval · n=80 · inst / prompt strict	.9688 / .9500	.9688 / .9500	.9688 / .9500	.9688 / .9500
Long context
Long-context retrieval AA-LCR · n=100 · ~107k-token prompts · judge-scored	0.780	0.800 a	0.800	0.780
Agent & tool use
Multi-turn tool use — telecom τ²-bench · n=114 · Pass^1	0.939	0.904	0.895	0.868
Multi-turn tool use — airline τ²-bench · n=50 · Pass^1	0.760	—	—	0.840
Multi-turn tool use — retail τ²-bench · n=60 · Pass^1	0.850	—	—	0.767
Coding
Agentic software engineering SWE-bench Verified · n=100	0.83	—	—	running
Agentic terminal coding Terminal-Bench Hard · n=44 · time-limited f	0.273	—	—	0.182

^a Scored on the 90 items it served; 10 were refused because the prompt exceeded that configuration's 131k window. Blended over the full 100 it reads 0.720.

^e The reference's HLE run was re-run and is still not a clean cell. Its first pass scored 37/120 = 0.3083 with 35 of 120 generations empty and the runner aborting on its own threshold ("29.2% failed generations — the score would be noise, not capacity"). A targeted re-run of the empty items recovered two, leaving 33/120 still empty (27.5%) and scoring 36/120 = 0.3000, which is what this table shows — the later, lower number, because it is the more complete run, not because it is more flattering. Both passes tripped the same abort. Read the reference's HLE cell as a lower bound on the model and a statement about that endpoint's behaviour on very long generations; the gap to this build (0.2417) is real but its size is not well determined.

^f Terminal-Bench here is a time-limited measurement, and is not comparable to the published 73.0. Both columns ran the terminus-2 agent with a 30-minute per-task agent timeout (TB_TIMEOUT=1800) at stock clocks (--global-timeout-multiplier 1.0), with no ×3 rerun of timed-out tasks. On this build 35 of 44 tasks (80%) ended in agent_timeout; of the 9 that ran to completion 5 were solved, and 3 of the timed-out tasks still passed their tests — 8/44 total. The reference behaved the same way (38/44 timeouts, and its run never closed: end_time: null). So both cells largely measure how much work fits inside 30 minutes on this hardware, not capability, and the comparison between them is only fair because the limit is identical.

Qwen publishes 73.0 for this checkpoint on Terminal-Bench 2.1 (Terminus). Do not read 0.182 against it: different harness version, different agent, and — decisively — not this time limit. No figure on this row should be compared to a vendor number.

^d The reference's three unanswered documents were re-run; one recovered. Its original 47/60 = 0.7833 included three documents that returned empty. Re-running exactly those three (and committing in advance to accept the result either way) recovered one — correct — giving 48/60 = 0.8000, which is what this table shows. The other two failed again with finish_reason=error after 128K and 254K characters of reasoning: the endpoint failed mid-generation, it is not the model declining to answer. Re-running an errored request is repair; re-running a completed-but-wrong answer would be selection, and none were re-run on that basis. Against the repaired reference this build's margin is +7 items, not the +8 first published.

^c The first run is published, by rule. Four runs of this build exist — a promotion gate ran three seeds, then the benchmark campaign ran its own cell — while the comparison arms have one run each:

run	seed	flex / strict
gate	1234	0.94 / 0.92	← published: the first, before any result existed to prefer
gate	99	0.94 / 0.88
gate	7	0.94 / 0.94
campaign	1234	0.96 / 0.94

Publishing the first run removes the selection question entirely; picking any later one means choosing with hindsight, and the highest of four is not a measurement. Two things the spread shows: generation is not deterministic at a fixed seed (the seed fixes which 50 questions are drawn, not the sampling — the two seed-1234 runs differ by one item), and n=50 is small (strict-match spans 0.88–0.94 across seeds, ±3 items). The other three columns are single runs carrying the same unmeasured uncertainty. Differences of a few items in this row are noise in either direction, including this build's.

τ² is this build's weakest area, and all three domains are now in. Against the reference:

domain	n	reference	this build
telecom	114	107/114 = 0.939	99/114 = 0.868	−8 items
airline	50	38/50 = 0.760	42/50 = 0.840	+4 items
retail	60	51/60 = 0.850	46/60 = 0.767	−5 items
total	224	196/224 = 0.875	187/224 = 0.835	−9 items

Ahead on airline, behind on telecom and retail, −9 items net. An earlier revision of this card called τ² "domain-split" when only telecom and airline had landed; retail settles it — multi-turn tool use is where 4-bit weights cost this model the most. If you are choosing a quant specifically for agentic tool-calling, weigh this section above the rest of the table.

On telecom, 113 of 114 simulations ended normally and one hit the harness's error ceiling (too_many_errors, scored 0); airline and retail ran 50/50 and 60/60 clean. So the shortfall is a capability difference, not harness noise.

Cells marked running / pending are genuinely unfinished, not withheld. This card is dated and will be revised as they land; the commit history is the record of what was known when.

Reproducing the quantisation

Data-free, CPU-only, file-to-file — no calibration set, no GPU, ~3 minutes for this model. (This describes the weights. The KV-cache scalars above are the one calibrated artefact in the repo and came from a separate capture pass on the served model; they do not touch the shards.)

from quark.torch.export.api import direct_quantize_checkpoint

EXCLUDE = [
    "lm_head", "*embed_tokens*",
    "*.self_attn.q_proj", "*.self_attn.k_proj", "*.self_attn.v_proj", "*.self_attn.o_proj",
    "*.self_attn.q_norm", "*.self_attn.k_norm", "*norm*",
    "*.linear_attn.conv1d", "*.linear_attn.norm",
    "*.mlp.gate", "*.mlp.shared_expert_gate",
    "mtp*", "*visual*", "*vision*",
]

Two things that are easy to get wrong:

• *.mlp.gate and *.mlp.gate_proj are different modules. The first is the MoE router and must stay bf16; the second is the SwiGLU gate projection and should be 4-bit. A glob that catches both silently quantises the router.
• When verifying, key on real artifacts, not on a _scale suffix. Several bf16 checkpoints in this family ship tensors like vision_tower.std_scale or per-layer layer_scalar in the original weights, and a naive check reports leaks on a perfectly correct build.

Check both directions — leakage (something quantised that should not be) and over-exclusion (projections that were meant to be 4-bit but stayed bf16) — and make a mismatch raise.

Per-family recipes for other architectures (Gemma-4 dense and MoE, Mistral-Small, and a hybrid sliding-attention model) are in the port repo; none of the exclude lists transfer between families.

Licence and attribution

Apache-2.0, inherited from Qwen/Qwen3.8-27B. The LICENSE file here is byte-identical to upstream's.

Modification made: weights of the MLP and linear-attention projections converted from bf16 to MXFP4 via AMD Quark 0.12.post1, as described above. No fine-tuning, no distillation, no change to architecture, tokenizer or chat template. All other tensors are the upstream values.

The gfx1201 enablement this port descends from was first done by Rob Smith (tcclaviger) on the vLLM 0.18.1 line; the FP8-WMMA kernel takes its per-K-group scale-fold design from his _matmul_fp8_ogs. See the NOTICE in the port repo for the full lineage.