llm-d-sc-egress-gate

egress classifier for llm-d semantic classification. Labels: ALLOW, BLOCK.

Architecture: sequence-classification head (requires a runtime that reads logits), base sentence-transformers/all-MiniLM-L6-v2.

Accuracy

Read the real-traffic row first.

eval set	n	accuracy	95% CI	macro F1
real traffic (WildChat, unanimous 3-model jury)	284	0.9965	0.980 – 0.999	0.8324
enterprise, secrets-handling situations	707	0.9533	0.935 – 0.967	0.9169
enterprise (unconditioned generation, unanimous jury)	744	0.9946	0.986 – 0.998	0.6653

Per-tier recall — read this before the accuracy

For an egress or capacity gate the errors are not interchangeable: missing a NEVER_EGRESS prompt leaks a live credential, while over-flagging a PUBLIC one costs a cheap round trip. Aggregate accuracy is carried by the largest class and hides this.

tier	recall
ALLOW	0.97
BLOCK	0.85

Run-to-run variance. PUBLISHED SEED IS THE FIRST ONE RUN (11), not the best. Three arms: seed 11, class-weighted entsec 0.9533 seed 22, class-weighted entsec 0.9434 seed 11, no class weight entsec 0.9505 median 0.9505

WHAT THIS MODEL IS. The binary egress gate: does this request contain live credentials or legally privileged material -- BLOCK -- or not -- ALLOW? It is the NEVER_EGRESS threshold of the five-tier sensitivity ladder, trained directly. Of every taxonomy fold measured in this project, this is the only one whose three-model jury agreement reaches the high 90s (96.4%), which is why a high-90s model score is a real target here rather than an artifact of grading against noisy labels.

ACCURACY IS THE WRONG HEADLINE AND THIS MODEL SHOWS WHY. Majority baseline is 82.89%. BLOCK recall -- what fraction of secret or privileged content is actually stopped -- is 85.12% at argmax, with 2.56% over-block. The five-tier model folded onto the same decision has HIGHER containment (89-90%) at LOWER accuracy. At matched containment the two cross: below 90% containment the folded model is cheaper, at 95% and above this one is, and at 99% containment it over-blocks 49% against 79%. A security gate runs at high containment, so this is the right model for that use -- for the opposite of the reason its headline number suggests.

DO NOT QUOTE ITS OTHER EVALS. This model scores 99.65% on real-gold and 99.46% on enterprise-gold. Both are worthless: those sets contain 2 and 5 BLOCK rows, majority baselines are 99.30% and 99.33%, and BLOCK recall is 1/2 and 1/5. entsec is the only eval here with enough BLOCK mass (121 rows) to say anything.

SCOPE LIMIT. entsec is a SYNTHETIC enterprise eval, 95.8% distinguishable from real assistant traffic. It exists because real consumer traffic is ~93% PUBLIC and cannot exercise these tiers at all.

The eval has a measured ceiling

Gold labels were audited by blind paired adjudication in two strata — the rows this model got wrong, and a sample of the rows it got right — with the judge shown two candidate labels in random order and no indication of provenance. Roughly 4.9% of the gold labels are themselves wrong, so a PERFECT classifier scored against this eval would reach about 0.95, not 1.0.

Read the real-traffic accuracy against that ceiling, not against 100%. Auditing only a model's mistakes would move the number up artificially; sampling the correct rows too is what makes the estimate honest, and it revealed that on ~3.3% of "correct" rows the model agreed with a bad label — meaning measured accuracy is very slightly overstated.

How the eval was built

Real-traffic rows come from WildChat-1M (ungated real assistant traffic). Each prompt was labelled independently by three models (claude-opus-5, claude-sonnet-5, claude-fable-5-1) from the task rubric alone -- no labeller ever saw a proposed label, so agreement is evidence rather than assent. Only unanimous rows are scored.

Those three agree unanimously on roughly 70-74% of real prompts. The remaining prompts are published as a contested split rather than discarded: they measure how much real traffic this taxonomy does not resolve, which no single accuracy figure can express.

Training data

744 rows from egress-v2+egress-real+egress-enterprise+egress-real-contested, mixing jury-labelled real traffic (register and class prior) with rubric-grounded synthetic data (coverage of tiers that are rare in real traffic). Training prior: None. Held-out eval prompts are excluded by content hash.

Latency

CPU single-request: p50 7.7 ms, p99 23.32 ms (Apple M-series, single thread). llm-d-sc serves the classifier on CPU, so model size trades directly against per-replica throughput.

Limitations

• WildChat is consumer traffic. For sensitivity it is ~93% PUBLIC and cannot measure the tiers that gate egress; the enterprise row above covers those.
• Labels come from LLM jurors, not human annotators. The rubric was validated by reproducing the project's hand-authored gold labels (complexity 0.9875, cost 1.000, sensitivity 1.000) before use.
• Not independently reproduced.