llm-d-sc-sensitivity-v2

sensitivity classifier for llm-d semantic classification. Labels: PUBLIC, INTERNAL, CONFIDENTIAL, REGULATED, NEVER_EGRESS.

Architecture: sequence-classification head (requires a runtime that reads logits), base sentence-transformers/all-MiniLM-L6-v2.

Accuracy

Read the real-traffic row first.

eval set	n	accuracy	95% CI	macro F1
real traffic (WildChat, unanimous 3-model jury)	284	0.8873	0.845 – 0.919	0.4946
enterprise, secrets-handling situations	707	0.7793	0.747 – 0.808	0.7186
enterprise (unconditioned generation, unanimous jury)	744	0.7043	0.671 – 0.736	0.6019
legacy hand-authored held-out	75	0.9200	0.836 – 0.963	0.9189

Hand-authored minus real traffic: +0.033. Hand-authored held-out prompts are written in the same clean register as the anchors; real users send truncated pastes, fragments and roleplay preambles. The real-traffic row is the one that predicts production behaviour.

Per-tier recall — read this before the accuracy

For an egress or capacity gate the errors are not interchangeable: missing a NEVER_EGRESS prompt leaks a live credential, while over-flagging a PUBLIC one costs a cheap round trip. Aggregate accuracy is carried by the largest class and hides this.

tier	recall
PUBLIC	0.69
INTERNAL	0.77
CONFIDENTIAL	0.88
REGULATED	0.71
NEVER_EGRESS	0.90

Run-to-run variance. Two seeds give 0.7793 / 0.7822 on the enterprise-secrets eval; sensitivity's seed spread is 0.0014, so this is a stable measurement. Trained on 46,930 rows including a 31,168-row enterprise corpus - scaling that corpus 2.3x is worth +2.27 points, 16x the noise floor and the largest verified single intervention in this project.

The eval has a measured ceiling

Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong, so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0.

Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.

How the eval was built

Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models (claude-opus-5, claude-sonnet-5, claude-fable-5-1) from the task rubric alone -- no labeller ever saw a proposed label, so agreement is evidence rather than assent. Only unanimous rows are scored.

Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.

Training data

42237 rows from sensitivity-v2:6783+sensitivity-real:8361+sensitivity-enterprise:31168+sensitivity-real-contested:618, mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: {'INTERNAL': 14842, 'REGULATED': 6948, 'PUBLIC': 9831, 'NEVER_EGRESS': 4561, 'CONFIDENTIAL': 6055}. Held-out eval prompts are excluded by content hash.

Latency

CPU single-request: p50 2.92 ms, p99 3.15 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.

Limitations

• WildChat is consumer traffic. For sensitivity it is ~93% PUBLIC and cannot measure the tiers that gate egress; the enterprise row above covers those.
• Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
• Not independently reproduced.