pflash-segment-probe

The segment probe of PFlash: it cuts a long prompt into whole structural units before the relevance head ranks them.

PFlash is Lucebox's prefill compression for long prompts. A small drafter, Qwen3.5-0.8B, reads the whole prompt once, and three small components on its activations decide what the large reader model prefills:

1. Lucebox/pflash-segment-probe cuts the prompt into whole units (statements and definitions, sentences, records, chat turns, tool calls);
2. Lucebox/pflash-request-probe finds the request, wherever it sits in the prompt;
3. Lucebox/pflash-scorer-head ranks the units against that request.

The highest-ranked units that fit the budget go to the reader in their original order, and the reader prefills only those. All three read the same tap, the drafter's block-14 residual (post_block14_residual_before_block15), from the same single pass.

How it works

• Architecture: LayerNorm(1024) → Linear(1024, 256) → GELU → three linear rows, each with a 5-tap convolution over its logit sequence plus the residual logit; about 263K parameters, 16 tensors. GGUF architecture segmentprobe, schema qwen3_5_0_8b_segment_probe_v2.
• Three outputs per token:
  • unit: a structural unit starts here. Units are the finest whole pieces: every statement, declaration, definition, comment or decorator block at file or container scope in code; sentences in prose; records, chat turns, diff hunks, tool calls and observations. Nothing inside a function body is a unit cut, so a kept unit is never a truncated function.
  • container: the enclosing region (class, file section, paragraph, session, assistant turn). Exported, not used by the served configuration.
  • sub-unit: a nested definition inside a body, used only to split units that are too long.
• Segmentation (settings in the GGUF): a cut where the unit score passes the threshold 0.8; cuts closer than min_segment 32 tokens to the previous one merge forward, so small leaves such as doc comments group with what follows; a unit longer than max_segment 512 tokens is split at line boundaries (oversize_split = line_aligned), preferring sub-unit starts. No parser runs at inference.
• Why the threshold is asymmetric: a missed boundary merges two whole units, which the head can still rank; a cut inside a unit truncates it, which it cannot repair. 0.8 keeps harmful cuts (inside a function body or mid-line) at or below 1 % of cuts on code, tool output, dialogue, records and prose, with boundary precision of 0.97 or more.

Files

file	sha256	note
segment-probe-s3-line512.gguf	470b7b0efe3736518dcb4c1ba8eb5b806107007e378a74668c17e26434602e68	the served file: max_segment 512, line-aligned splitting
segment-probe.gguf	bd3f0bff1057ad0c953deb79d4b82415076906494c7bbb47a8a0f8e85433b667	the same weights and thresholds with max_segment 2048; not used in the configuration below

Using it

PFlash runs inside the Lucebox server. Download the three components and the drafter, then set:

export PFLASH_SCORING_HEAD_GGUF=/path/to/seg-scoring-head.gguf
export PFLASH_SEGMENT_PROBE_GGUF=/path/to/segment-probe-s3-line512.gguf
export PFLASH_REQUEST_PROBE_GGUF=/path/to/request-probe.gguf
export PFLASH_SELECT_SEGMENTS=probe PFLASH_SELECT_SCORE=density PFLASH_SELECT_MODE=budget_only
export PFLASH_SELECT_QUERY_PROBE=1 PFLASH_QUERY_CANDIDATES=1 PFLASH_QUERY_PIN=512
export PFLASH_SELECT_QUERY_TOKENS=128 PFLASH_SELECT_QUERY_HEAD=1
export PFLASH_SELECT_PARAGRAPH_JOIN=0 PFLASH_SELECT_CHUNK_SIZE=256
export PFLASH_ADMISSION=cache PFLASH_ARRIVE=1 PFLASH_CHAT_RECALL=1
export PFLASH_CHAT_REBUILD_RATIO=2 PFLASH_CHAT_REBUILD_MIN_TOKENS=65536
export PFLASH_FALLBACK_SPREAD=0.11 PFLASH_COLD_HISTORY=raw
# server: --prefill-compression auto --prefill-threshold 16384
#         --prefill-curve 16384:0.315 32768:0.270 65536:0.270 120000:0.225
#         --prefill-skip-park --cache-type-k q4_0 --cache-type-v q4_0 --max-ctx 139264

• Drafter (required, checked by sha256 at load): unsloth/Qwen3.5-0.8B-GGUF Qwen3.5-0.8B-BF16.gguf, sha256 cedf89af31c9041b601fa58303285bc46d99c51baee1b13f5e919626ca526ee5. Each component refuses to load against any other drafter file.
• PFlash acts when at least 16,384 prompt tokens are new to the server and less than that is already cached. The reader then prefills 31.5 % of the prompt at 16K tokens, 27 % at 32K–64K and 22.5 % at 120K.
• When the relevance scores are too flat for the budget, the request is served with full prefill (PFLASH_FALLBACK_SPREAD). An agent's uncached earlier conversation is never compressed (PFLASH_COLD_HISTORY=raw).
• Runtime support: Luce-Org/lucebox PR #746.

Results

Measured with all three components together, reader Qwen3.8-27B IQ4_XS, the configuration above, NVIDIA L40S, greedy, each benchmark's official metric:

set	n	full prefill	PFlash
Document QA	30	27	27
LongBench v2	12	6	7
NoLiMa	24	14	14
InfiniteBench CodeDebug	30	24	24
BRIGHT	28	23	23
needle in a haystack, 16K–120K	100	100	100
public sets, unique questions (RULER QA's 15 are inside Document QA)	224	194	195
LongCodeQA, questions that need the code	114	57	62

None of these differences is outside its 95 % interval. The reader prefills a median 29 % of the prompt, and the first token arrives 2.7× sooner on average (mean 30.5 s → 11.2 s over the public requests; 60.4 s → 14.8 s median on 82K–98K-token CodeDebug prompts).

Limits

• Not lossless: a request can still lose something the answer needed.
• Cold agent histories are served with full prefill, so a resumed agent session gets no speed-up.
• "List everything matching X" over many similar units (emails, wiki pages) is the hardest case; the flat-score fallback catches most of it.
• Measured serving one request at a time.