pflash-scorer-head

The relevance head of PFlash: it scores how relevant each unit of a long prompt is to the user's request.

PFlash is Lucebox's prefill compression for long prompts. A small drafter, Qwen3.5-0.8B, reads the whole prompt once, and three small components on its activations decide what the large reader model prefills:

1. Lucebox/pflash-segment-probe cuts the prompt into whole units (statements and definitions, sentences, records, chat turns, tool calls);
2. Lucebox/pflash-request-probe finds the request, wherever it sits in the prompt;
3. Lucebox/pflash-scorer-head ranks the units against that request.

The highest-ranked units that fit the budget go to the reader in their original order, and the reader prefills only those. All three read the same tap, the drafter's block-14 residual (post_block14_residual_before_block15), from the same single pass.

How it works

• Architecture: two projection matrices on the drafter's block-15 attention geometry, scoringhead.attn_q.weight F32 [1024, 2048] and scoringhead.attn_k.weight F32 [1024, 512], 2.62M parameters. GGUF architecture pflash_scoring_head, schema qwen3_5_0_8b_nope_qk_mass_v1.
• Scoring: the request tokens (from the request probe) form queries, every context token forms a key, and the head computes query→context attention without rotary positions. Each context token gets its attention mass; a unit's score is the mean token mass over the unit (density), so long units are not favoured for their length.
• Selection: the server keeps the highest-density units up to the token budget, in their original order, plus the request itself.

Files

file	sha256	size
seg-scoring-head.gguf	95e13076bff5c9b204c615366dfe08dbfa56aaa0bb417ff3dbf7d09c5906c1d6	10,486,400 bytes

Using it

PFlash runs inside the Lucebox server. Download the three components and the drafter, then set:

export PFLASH_SCORING_HEAD_GGUF=/path/to/seg-scoring-head.gguf
export PFLASH_SEGMENT_PROBE_GGUF=/path/to/segment-probe-s3-line512.gguf
export PFLASH_REQUEST_PROBE_GGUF=/path/to/request-probe.gguf
export PFLASH_SELECT_SEGMENTS=probe PFLASH_SELECT_SCORE=density PFLASH_SELECT_MODE=budget_only
export PFLASH_SELECT_QUERY_PROBE=1 PFLASH_QUERY_CANDIDATES=1 PFLASH_QUERY_PIN=512
export PFLASH_SELECT_QUERY_TOKENS=128 PFLASH_SELECT_QUERY_HEAD=1
export PFLASH_SELECT_PARAGRAPH_JOIN=0 PFLASH_SELECT_CHUNK_SIZE=256
export PFLASH_ADMISSION=cache PFLASH_ARRIVE=1 PFLASH_CHAT_RECALL=1
export PFLASH_CHAT_REBUILD_RATIO=2 PFLASH_CHAT_REBUILD_MIN_TOKENS=65536
export PFLASH_FALLBACK_SPREAD=0.11 PFLASH_COLD_HISTORY=raw
# server: --prefill-compression auto --prefill-threshold 16384
#         --prefill-curve 16384:0.315 32768:0.270 65536:0.270 120000:0.225
#         --prefill-skip-park --cache-type-k q4_0 --cache-type-v q4_0 --max-ctx 139264

• Drafter (required, checked by sha256 at load): unsloth/Qwen3.5-0.8B-GGUF Qwen3.5-0.8B-BF16.gguf, sha256 cedf89af31c9041b601fa58303285bc46d99c51baee1b13f5e919626ca526ee5. Each component refuses to load against any other drafter file.
• PFlash acts when at least 16,384 prompt tokens are new to the server and less than that is already cached. The reader then prefills 31.5 % of the prompt at 16K tokens, 27 % at 32K–64K and 22.5 % at 120K.
• When the relevance scores are too flat for the budget, the request is served with full prefill (PFLASH_FALLBACK_SPREAD). An agent's uncached earlier conversation is never compressed (PFLASH_COLD_HISTORY=raw).
• Runtime support: Luce-Org/lucebox PR #746.

Results

Measured with all three components together, reader Qwen3.8-27B IQ4_XS, the configuration above, NVIDIA L40S, greedy, each benchmark's official metric:

set	n	full prefill	PFlash
Document QA	30	27	27
LongBench v2	12	6	7
NoLiMa	24	14	14
InfiniteBench CodeDebug	30	24	24
BRIGHT	28	23	23
needle in a haystack, 16K–120K	100	100	100
public sets, unique questions (RULER QA's 15 are inside Document QA)	224	194	195
LongCodeQA, questions that need the code	114	57	62

None of these differences is outside its 95 % interval. The reader prefills a median 29 % of the prompt, and the first token arrives 2.7× sooner on average (mean 30.5 s → 11.2 s over the public requests; 60.4 s → 14.8 s median on 82K–98K-token CodeDebug prompts).

Limits

• Not lossless: a request can still lose something the answer needed.
• Cold agent histories are served with full prefill, so a resumed agent session gets no speed-up.
• "List everything matching X" over many similar units (emails, wiki pages) is the hardest case; the flat-score fallback catches most of it.
• Measured serving one request at a time.