bw24-bench — benchmark artifacts for bw24

Reproduction artifacts for the performance claims in github.com/avifenesh/bw24 — a from-scratch LLM inference engine for RTX 5090 Laptop (sm_120a). The README there compares bw24 against llama.cpp with specific speculative-decoding configs and vocab-trimmed draft heads. Those trims and prompts are not standard files you can download elsewhere, so they are published here: if the claim depends on a specific artifact, the artifact must be public.

What is in this repo

Trimmed MTP draft-head GGUFs (Qwen3.6-27B)

Three FR-Spec vocab-trimmed variants of the Qwen3.6-27B MTP draft head (Q4_K_M). All three are the same model-trained head (original Qwen3.6 release) with a pre-defined limited draft vocabulary: a d2t ranked-token subset (32768 tokens) over the full Qwen vocab, built from token-frequency analysis. Trimming shrinks the draft lm_head reads ~5x; it does not retrain anything.

File	Ranking	Use
mtp-Qwen3.6-27B-Q4_K_M-frspec32768.gguf	generic frequency-ranked cover	best for medium-code / long-agentic prompts (p2/p3)
mtp-Qwen3.6-27B-Q4_K_M-frspec-code75-32768.gguf	code-skewed (75% code corpus)	best for short-code prompts (p1) only
mtp-Qwen3.6-27B-Q4_K_M-frspec-balanced32768.gguf	balanced code/prose mix	superseded by the two above; kept for the experiment record

Measured lesson (2026-07-07, in the tune-data corpus): trim rankings are class-dependent and head-specific. code75 wins only on short-code content; the generic ranking wins everywhere else; none of these rankings transfer to a differently-trained head (e.g. the NVIDIA-finetuned checkpoint's own head) — re-derive the ranking from the target head's draft distribution before trimming.

Each trim pairs with bw24 via BW24_FRSPEC_TRIM=<path-to-trim.gguf>.

9B-native trim (trims/frspec-9b-32768.gguf)

The Qwen3.5-9B has a different vocabulary (248320 tokens vs the 27B/35B's 151936), so the trims above cannot transfer to it. This d2t was built with bw24's frspec_rank tool (in-repo): tokenize a local ~2.6M-token code/docs corpus with the 9B's own GGUF tokenizer, rank ids by frequency, keep the top 32768. Measured effect on the 9B raw-prompt spec board (2026-07-08): p1 199→243, p2 166→195, p3 142→162 tok/s (+22/+17/+14%), self-consistency PASS. The trim mechanism byte-gathers rows from the running model's own lm_head — the file carries only the ranked id list (131 KB).

A second cross-model lesson (2026-07-08): between models that DO share a tokenizer (27B/35B), the generic ranking transfers unmodified — the 35B uses the 27B's frspec32768 file directly (spec p2 174.5→187.1 from the trim alone).

Bench prompts (prompts/)

The exact prompts behind the README numbers, plus the long-context set:

File	Size	Class
p1-code-short.txt	146 B (28 tok)	short code
p2-code-medium.txt	6.3 KB (~1.8k tok)	medium code
p3-agentic-long.txt	23 KB (~6.3k tok)	long agentic
p4-16k.txt / p5-32k.txt / p6-64k.txt	79/135/247 KB	long-context prefill set

Exact bench protocol

• 256 generated tokens, temperature 0, N=3 runs, report the median.
• bw24 env law — every bw24 run sets all four fast-path flags (a missing flag silently measures the slow path): BW24_FAST=1 BW24_GEMM=1 BW24_MMVQ=1 BW24_FA_VEC=1
• Speculative runs add: BW24_SPEC_K=<K> BW24_SPEC_PMIN=<pmin> BW24_SPEC_HPOST=1 BW24_FRSPEC_TRIM=<trim.gguf>
• Runner: research/e2e/run-e2e.sh in the bw24 repo (same prompts fed to both engines).
• Speculative output is gated bit-exact: K=1..8 self-consistency check pins it token-identical to plain greedy decode.

Per-class configs behind the README spec board (2026-07-08, raw-prompt protocol, both engines full power)

Qwen3.6-27B (108 / 91 / 79.5 vs llama 86.4 / 89.9 / 73.2 = 1.25x / 1.01x / 1.09x):

Prompt	K	pmin	Trim	bw24 tok/s	llama.cpp tok/s
p1 short code	3	0.15	generic frspec32768	108.4	86.4
p2 medium code	3	0.3	generic frspec32768	90.9	89.9
p3 agentic long	3	0.3	generic frspec32768	79.5	73.2

Qwen3.5-9B (243 / 195 / 162 vs llama 186 / 158 / 155 = 1.31x / 1.23x / 1.04x): K=3, pmin 0.15, BW24_SPEC_HPOST=1, trim trims/frspec-9b-32768.gguf (the 9B-native ranking — the 27B trims cannot transfer, different vocab).

Qwen3.6-35B-A3B (197 / 194 / 177 vs llama self-MTP 215 / 208 / 202): K=2, pmin 0.4, BW24_SPEC_PMIN0=1 (zero-draft rounds), trim = the 27B frspec32768 file (transfers unmodified — same tokenizer; the d2t gather reads the 35B's own lm_head). Plus BW24_MOE_CACHE=1.

Config laws from the corpus: K optimum is (content-class, protocol) dependent — chat-templated prompts shift both K and the trim choice (chat short-code runs K=7 + code75 at 122 tok/s on the 27B; raw continuation runs K=3 + generic). Zero-draft rounds pay below ~75% base acceptance and hurt above ~90%. Trim before gating — ungated pmin sweeps measure flat without a cheap probe.

llama.cpp comparison config

llama.cpp built on the same machine at commit 047bfa508 (includes the FR-Spec d2t draft-vocab-trim patch for MTP), run at its best serve config:

llama-server -m Qwen3.6-27B-NVFP4-Q4_K_M-mtp.gguf \
  --model-draft mtp-Qwen3.6-27B-Q4_K_M.gguf --spec-type draft-mtp \
  --spec-draft-n-max 3 --spec-draft-p-min 0.1 -ngld 999 \
  --ctx-size 16384 -ngl 999 -fa on --cache-type-k q8_0 --cache-type-v q5_1

Protocol note from the corpus: sequential cross-session A/B numbers lie by up to 10% (clock behavior differs when an engine runs alone/cold) — interleave both engines within the same minute, N>=3 pairs, both orders.

Full experiment corpus

Every tuning experiment (positive and negative) is recorded as JSONL at github.com/avifenesh/bw24 — research/tune-data/. The rows named qwen36-27b-nvfp4-q4km and qwen35-9b-nvfp4 / qwen36-35b-a3b-iq4xs dated 2026-07-07/08 are the ones behind the table above.