light-sufficiency-predictor (GGUF, 8 blocks) (DRAFT, not uploaded)

Draft model card. Nothing has been uploaded. The apache-2.0 header is the licence of the Qwen3-0.6B base model; it is not confirmed for the whole package (see the licence notes).

What it predicts

From the text alone, the probability that a summary written by one specific small light model (Qwen3-0.6B, BF16 GGUF through the llama backend, English prompt of app commit 97272ef, target 100 words) will be sufficient. English texts only. The prediction is made before the summary exists, so the app can route risky texts to a heavier model.

Label v2: sufficient (1) when the light summary passes the technical checks without too_long and the Gemma 3 12B judge scores faithfulness >= 4. The judge labels come from Gemma 3 12B (4-bit, mlx).

How it works

The 8-block GGUF is run on the app prompt that contains the text. The per-token outputs of the document tokens are averaged (1024 values). One linear map z = w.x + b (scaler, PCA 8 and logistic regression folded together, C 0.1) gives a score, and p_sufficient = sigmoid(A * z + B) is the Platt calibrated probability. p_insufficient = 1 - p_sufficient. probe.json holds w, b, A, B and export.json the exact recipe:

• Tokenize the prompt with add_bos false and parse_special true.
• Document tokens: Byte offsets of every token piece in the prompt (UTF-8, concatenated pieces equal the prompt bytes). The document text occupies bytes [len(prefix), len(prefix) + len(document bytes)) with prefix = '<|im_start|>user\n\n'. A token is a document token when its byte range overlaps that range, so the token that merges the last document character with the following newline is included. The document text is used as given, not stripped.
• Pooling: Mean over the document tokens of the per-token embeddings (llama.cpp pooling NONE, output after output_norm of block 8), one float32 vector of 1024 values.

Routing

export.json has a routing block: the app routes a text to the light model when p_sufficient >= threshold(quality_preference), with the threshold linearly interpolated over the points (preference, threshold) [[0.0, 0.5], [0.5, 0.8], [1.0, 0.95]]. The points were chosen on the 206 validation texts, where Platt is also fitted, so the calibration is optimistic. See results/<run>/threshold_analysis.md.

Advice

export.json also has an advice block, mode none_with_prediction: with a prediction, do not suggest the heavy model; without one, the app's length advisor decides. Why: on validation the probe does not separate failures among light-routed texts (see threshold_analysis.md).

Results

Fitted on 616 train texts, Platt calibration on 206 validation texts. AUROC and AP use the raw score. 95% bootstrap intervals (2000 resamples, seed 42).

split	n	insufficient	AUROC	AP insufficient	Brier	ECE
validation	206	30	0.710 [0.597, 0.816]	0.410 [0.250, 0.581]	n/a	n/a
test	151	33	0.714 [0.611, 0.814]	0.490 [0.332, 0.648]	0.155 [0.118, 0.197]	0.057 [0.035, 0.134]

The test set is the 151 labelled English texts of the multilingual run, used a second time here to confirm this deployable variant (the HF layer 8 probe used it first). Paired AUROC differences:

• option 1 (GGUF 8 blocks) - (d): +0.053 [-0.041, +0.148]
• option 1 (GGUF 8 blocks) - HF probe: -0.022 [-0.040, -0.004]

Cost

The embedding run of the 8-block GGUF on Metal (llama.cpp):

• 275 input tokens: 0.072 s (the app's full summary: 0.97 s)
• 941 input tokens: 0.231 s (the app's full summary: 1.97 s)
• 2494 input tokens: 0.742 s (the app's full summary: 2.84 s)

The probe itself is one dot product of 1024 values.

Intended use

Research on routing between a light and a heavy summarization model, as a risk signal next to other signals, on English texts like the training domains.

Limitations

• One light model (Qwen3-0.6B BF16) and one prompt version (97272ef, target 100 words). Other models, prompts or lengths are not covered.
• English only. Machine-translated texts are not covered.
• The test set is small (151 texts, 33 insufficient), so the intervals are wide.
• The label depends on one judge model (Gemma 3 12B) and on the technical checks.
• It does not read the summary, so it cannot decide alone whether a summary is trusted.
• The llama.cpp output includes output_norm, so the features differ from the HF hidden states and need this exact GGUF recipe.

Licence notes

• Base model Qwen3-0.6B: Apache-2.0. The GGUF is derived from unsloth/Qwen3-0.6B-GGUF/Qwen3-0.6B-BF16.gguf.
• Judge labels: Gemma 3 12B output is subject to the Gemma Terms of Use, not an open-source licence: check them before publishing the labels or a model trained on them.
• Source texts come from public datasets (CNN/DailyMail, DialogSum, PubMed summarization, GovReport). The repo records no dataset licences: check each before publishing or reusing the model.

Parity check

From the GGUF and probe.json only (llama-cpp-python, no HF tokenizer, no sklearn), 20 GovReport validation texts against the Python probe (sklearn pipeline and Platt on the cached features): max abs difference of p_sufficient 4.19e-07, mean 9.88e-08 (tolerance 0.0001). The byte-offset document span equals the HF offset span (and the token ids are equal) on 822 of 822 train and validation texts; 0 differ.