alimpfard/qwen3.8-27b-kv-approximation

Qwen3.8-27B-KVA: late-layer KV approximation

A (currently) 931M-parameter projector that approximates half of Qwen3.8-27B's prompt processing layers. As measures, this provides a 30-60% prefill performance improvement in exchange for ~4-5% perplexity (measurements below).

This repo contains only the projector (kva.* tensors, ~0.5 GB); which can be merged into any Qwen3.8-27B GGUF using the provided merge script.

A reference llama.cpp implementation that can read and use the kva tensors is provided on the kva branch in my fork of llama.cpp.

What the hell?

Qwen 3.8-27B has 64 layers, 48 gated-DeltaNet layers and 16 full-attention layers; during prefill the engine runs layers 0..31 as is, and uses this approximator to map the layer-31 residual of every token to the token-mixer inputs of each late layer:

late layer type	predicted per token	dims
gated deltanet (24 layers)	in_proj_qkv output (q, k, v before conv/silu), in_proj_a, in_proj_b	10240 + 48 + 48
full attention (8 layers)	k_proj, v_proj outputs (before k_norm / rope)	1024 + 1024

Afterwards, the engine runs some cheap state updates for the tokens (conv + silu, l2-norm, and deltanet scan, plus k_norm + rope + KV cache writes for the attention layers), and runs the last token of each ubatch through the late layers to produce exact logits on top of the approximated context.

The projector's own architecture is a simple RMSNorm -> Linear{5120 -> 2560} -> 3x[Causal transformer blocks] -> RMSNorm -> Linear head; where the causal transformer blocks are each 20 heads @ 128, with SwiGLU FFN 7080, for a total of 931.2M parameters.

This work was inspired by nowokay's blog post and the causal encoder-decoder split of DeepSeek-V4.1-Flash.

v1 Eval Results

Measurements are done on held-out text cut into windows of 1536 prompt tokens + 512 continuation tokens; the prompt is prefilled (exact vs. KVA) and the continuation is forced through the exact model on top of that cache. This report presents perplexity, and top-1% greedy matches per token. "wiki" is Wikitext-103, and "code" is a python code file.

llama.cpp engine, Q4_K_M base, (24 windows, q8_0 projector)

	exact	KVA	delta
wiki continuation ppl	5.949	6.214	+4.5%
code continuation ppl	2.109	2.207	+4.6%
greedy top-1 agreement with the exact model, wiki		91.3%
greedy top-1 agreement with the exact model, code		93.8%
prefill, ubatch 1024, RTX Pro 4500	1753 tok/s	2684 tok/s	1.53x
prefill, ubatch 1024, 2x Titan Xp (layer-split)	256 tok/s	404 tok/s	1.58x

Roughly, the quality loss is equivalent to going from Q4_K_M to unsloth's UD-Q3_K_XL. For scale, llama-perplexity reports ppl ratios of 1.007 (Q4_K_M), 1.003 (UD-IQ4_XS), 1.034 (UD-IQ3_XXS) and 1.055 (Q3_K_M) on wikitext (2k token chunks, 40 chunks) -- note that this is a different protocol from the report above.

Projector quantization and flash-attn (Q4_K_M base, 4500, 12 windows, -c 4096)

projector	file size	wiki ppl	code ppl	prefill ub 1024
q8_0	990 MB	6.0253	2.3087	2659 tok/s
q4_0	525 MB	6.0304	2.3113	2638 tok/s

From our measurements, q4_0 of the projector is more-or-less equivalent in this version; even though it has no meaningful impact on performance, it frees up ~400 MB of VRAM that can be used elsewhere.

Training data

Round 2 (this release) was trained on 34M tokens, one sequence of at most 3072 tokens per step for 36k steps, sampled by sequence count from:

source	domain	sequences	tokens
Manusagents/GPT-5.5-…-Distillation-Dataset (coding + applied splits)	code, chat-templated	24.7k	16M
r0b0tlab/qwen3.8-max-glm5.2-kimi-k3-distillation (canonical split, with tools)	agentic, chat-templated	11.5k	12M
Salesforce/wikitext (wikitext-103-raw-v1, train)	prose, plain text	2.0k	6M
mondk/GptModel-CoT	chain-of-thought, chat-templated	47	0.07M

The CoT source contributed almost nothing in this round (47 sequences); the eval prompts (wikitext test, a held-out Python file) are not part of the training data.

Limitations

• Quality was measured on 1536-token prompts of English prose and Python. Other domains, other languages and much longer prompts remain untested.
• The projector attends over the entire ubatch, so its behaviour at ubatch sizes above 1024 is also untested.
• This projector only applies to the "base" Qwen3.8-27B architecture (quantized and abliterated variants are fine).
• Tested on CUDA only (Blackwell sm120 and Pascal sm61).