verdict-laya

One small model that runs four guardrail checks for LLM apps and returns a calibrated probability for each:

check	question it answers
pii	Does the text contain personal data about an individual?
toxicity	Is the text toxic, insulting or threatening?
prompt_injection	Is this user input trying to override the system's instructions?
grounded	Is the response supported by the provided context?

It is Laya (ModernBERT-large encoder plus a typed-decision head, 421M parameters) fine-tuned on all four checks at once, with per-check temperature scaling and thresholds fitted on a validation split. Code, data preparation and the full benchmark: github.com/ashiksharonm/verdict.

Results

Held-out test sets, macro-F1 with 95% bootstrap intervals (1,000 resamples). Full tables, calibration, robustness and latency are in evals/results/2026-10-17/report.md.

check (test n)	verdict-laya, ONNX int8 (Apple M5)	best other system tested
pii (2,000)	0.988 [0.983, 0.992]	Presidio (spaCy lg) 0.820
toxicity (2,000)	0.899 [0.885, 0.911]	DeBERTa-v3 NLI zero-shot 0.756
prompt_injection (1,249)	0.973 [0.963, 0.982]	ProtectAI DeBERTa-v3 injection v2 0.916
grounded (1,200)	0.693 [0.668, 0.721]	Laya zero-shot 0.611

• Expected calibration error 0.006–0.034 across checks.
• Prompt-injection recall on obfuscated attacks (leetspeak, homoglyphs, zero-width characters, random case, base64): 0.963–1.000.
• On Apple M5 the int8 ONNX export matches the fp32 model's decisions on 99.2–100% of validation items (onnx/parity.json); on x86 use the fp32 file (see below).
• A Qwen3-1.7B judge was also benchmarked, on a 200-item subsample per check; it scored below every specialised system here.

Which file to use

CPU	file	why
x86 (Intel / AMD) — servers, cloud	onnx/model.onnx (fp32, 1.7 GB)	matches the benchmark exactly
Apple silicon (tested on M5); other ARM untested	onnx/model.int8.onnx (int8, 443 MB)	matches fp32 on 99.2–100% of validation decisions per check

Do not serve the int8 file on x86. Measured on Modal's x86 CPUs over all 6,449 held-out test items (report):

check	fp32 on x86	decisions = fp32 benchmark
pii	0.988	100.0%
toxicity	0.898	100.0%
prompt_injection	0.972	100.0%
grounded	0.687	100.0%

On x86 CPUs without VNNI (AVX2-only, common on AMD cloud hosts) the int8 file's u8s8 products saturate and every score collapses to about 0.5. Re-quantising with reduce_range avoids the collapse but still loses 3–7 macro-F1 points on x86 (pii 0.945, toxicity 0.873, prompt_injection 0.940), so fp32 is the supported x86 path.

Speed

Measured on an Apple M5 (16 GB). Batch-1 latency, p50 / p95:

variant	device	p50	p95
fp32 (PyTorch)	Apple GPU (MPS)	87 ms	250 ms
int8 (ONNX Runtime)	CPU	102 ms	545 ms

Long grounded inputs dominate p95. Expect a 2-vCPU cloud CPU to be slower.

Files

• model.safetensors, encoder/, tokenizer/, questions.json, rl_agent_config.json: the fine-tuned Laya checkpoint (fp32), loadable with laya.
• onnx/model.onnx: fp32 ONNX export; use this on x86 CPUs.
• onnx/model.int8.onnx: dynamic int8 export; ARM CPUs only (see above).
• verdict_calibration.json: per-check temperature and decision threshold.
• train_report.json: Laya's training report.

Limitations

• English only, trained on public datasets; production text will differ.
• grounded is the weakest check: RAGTruth contexts are often longer than the 512-token input limit, so the evidence can be cut off.
• The toxicity test set is class-balanced; real traffic is far less toxic, so production precision will be lower.
• Decision thresholds come from validation data; retune them on your own traffic.
• A guardrail lowers risk; it does not remove it. Keep human review for high-stakes decisions.

Training data

All sources are permissively licensed (Apache-2.0, MIT, CC0-1.0; RAGTruth upstream MIT). Splits are seeded and deduplicated across train/val/test; see evals/DATASETS.md. Fine-tuned for one epoch on 9,600 examples (about 2,400 per check) on an Apple M5 in 3.9 hours.