First-run evaluation: K-12 school email triage (what worked, what didn't)

by nccyber - opened 3 days ago

First-run evaluation: Laya for K-12 school email triage (what worked, what didn't)

I work in data for a K-12 public virtual school, and a lot of what our front office does all day is
read family email and decide who should handle it, how fast, and whether anything in it needs
escalating. That is a routing and scoring problem, not a generation problem, so Laya's one-pass
typed-question design looked like a good fit. Here is an honest first pass.

Everything below uses six synthetic emails I wrote for the test. No real student or family data.
Script: eval_school_email.py. Hardware: Apple M4, 16 GB, MPS.

Setup

Five questions per email, all in one predict call:

department (choice, 5 options), sentiment (choice, 3), urgency (score, 0 to 2)
leaving? asked twice: once as noul (yes/no) and once as a two-option choice,
to see whether the question type matters. It does.

Results, English checkpoint (repo root)

email	department	sentiment	urgency (0-2)	leaving? noul P(true)	leaving? choice P(yes)
login problem + "we will look at other schools"	tech_support (0.88)	negative (0.96)	1.32	0.17	0.60
grade dropped, asks to see gradebook	teacher (0.82)	negative (0.85)	0.92	0.07	0.16
thank-you note to a teacher	teacher (0.74)	positive (0.87)	0.57	0.00	0.10
moving out of state, needs withdrawal forms	front_office (0.27)	neutral (0.90)	1.06	0.67	0.65
school laptop screen cracked	teacher (0.35)	negative (0.87)	0.97	0.01	0.22
"nobody calls back", enrollment hold	front_office (0.58)	negative (0.93)	1.49	0.14	0.14

268 ms per email for all five questions.

Results, `typed-decisions` checkpoint

email	department	sentiment	urgency (0-2)	leaving? noul P(true)	leaving? choice P(yes)
login problem + "other schools"	tech_support (0.62)	negative (0.91)	1.72	0.64	0.84
grade question	teacher (0.63)	negative (0.69)	1.03	0.13	0.23
thank-you	teacher (0.50)	positive (0.82)	0.70	0.03	0.10
withdrawal forms	enrollment (0.27)	neutral (0.75)	1.00	0.55	0.70
cracked laptop	tech_support (0.32)	negative (0.72)	1.31	0.06	0.28
"nobody calls back"	front_office (0.45)	negative (0.82)	1.60	0.38	0.33

175 ms per email.

What I took from it

Sentiment and urgency are good out of the box. Six for six on sentiment, with sharp
distributions, and the urgency ordering matches what a human would say (the "test tomorrow"
and "nobody calls back" emails on top, the thank-you at the bottom).
The noul type under-reads hedged language on the English checkpoint. "We are going to
look at other schools" is a threat to leave to any front-office reader. noul gave it 0.17.
The exact same instruction as a two-option choice gave 0.60, and 0.84 on typed-decisions.
If you need yes/no, phrase it as a choice with the yes option described, or use typed-decisions.
Routing is decent, not finished. Four of six on the English checkpoint. The two misses
(cracked laptop to "teacher", withdrawal to "front office") both came with low top
probabilities (0.35, 0.27), which is the model being honest that it was unsure. A confidence
gate would have sent both to a person, which is the right behaviour.
typed-decisions is the better yes/no model and the worse everything-else model.
It fixes direction on the leaving question but flattens sentiment and routing probabilities.
The calibration warning on load is real. The shipped choice:11+ temperature is 0.10 and the
library clamps it to 0.5, so treat confidence on long option lists as uncalibrated.

For a first release I think this is a genuinely useful shape of model: cheap, fast, calibrated, and
it fails loudly with low probabilities rather than confidently wrong. The gaps are in the yes/no
type and in soft, indirect phrasing, which is exactly what parent email is full of.

Demo Space with the same questions, editable: https://huggingface.co/spaces/nccyber/laya-school-email-triage

Happy to share more emails or rerun with different question sets if useful to the authors.

nccyber

3 days ago

Follow-up: a second task, and a comparison with Jev and openjev

Same setup as above (synthetic data, Apple M4, one choice question), different task: place a K-12 assignment on the NC DPI AI usage scale, A0 (no AI) to A4 (full AI), with the five level definitions as the choice criteria. Ten assignments, two per level. Two versions of the test:

Easy set: the teacher's instructions state the AI policy in words (never the level name).
Hard set: no AI wording anywhere; the level has to be inferred from the shape of the task. The "truth" here is what a reasonable teacher would set, so "within one level" is the fairer score.

Models: Laya (both checkpoints, local), AlexWortega/openjev 0.8B and 4B (NLI cross-encoder, local, one hypothesis per level), and TypeSafe Jev 1.13 via OpenRouter's Decisions API (same state/questions request shape as Laya).

Easy set (policy stated)

model	exact	within one level	typical top probability
Laya English	8/10	10/10	~0.43
Laya typed-decisions	9/10	10/10	~0.30
openjev 0.8B	10/10	10/10	~0.45
openjev 4B	10/10	10/10	~0.80
Jev 1.13	10/10	10/10	1.00 on every item

Hard set (no AI wording)

model	exact	within one level	failure pattern
Laya English	3/10	6/10	defaults to A2 (7 of 10)
Laya typed-decisions	4/10	6/10	defaults to A2, flatter
openjev 0.8B	3/10	4/10	defaults to A0 (8 of 10)
openjev 4B	5/10	8/10	drifts one level high in the middle
Jev 1.13	6/10	6/10	A0–A2 perfect; every A3/A4 called A0

What I take from it

Reading a stated policy is solved by everyone. Laya's two misses were the two most similar definitions (A1 vs A2, A3 vs A2), both near-ties in the probabilities.
Inferring a policy is solved by no one. Each model has a default it falls back to when the text gives it nothing: A2 for Laya, A0 for the small openjev and for Jev. Jev's version is the most interesting: it reads "nothing here permits AI" as A0, so the biggest, most open-ended projects (research paper, client product build) all became "no AI," with the true level under 0.03. A defensible instinct for a guardrail, the wrong one for this scale.
Calibration is where Laya and the 4B cross-encoder look best. Jev returned exactly 1.00 on all ten easy items and 0.5–0.65 on its hard-set misses. openjev 4B went from ~0.8 on the easy set to ~0.3 on the hard set, which is the right thing to do. Laya sat near 0.3–0.45 on both, which is honest but not very informative.
For Laya specifically, I checked whether the 192-token option budget was cutting my definitions. It was not: the five options total 167 tokens and the model saw all of them plus the full instruction. So the gap on the hard set is not truncation. Given that the 0.8B decoder did worse than Laya here, I don't think it is the ModernBERT backbone either. My read is that it is the head's training distribution plus the one-pass design, and the fix for a domain like this is the fine-tuning notebook with a few hundred teacher-labeled assignments.

Scripts and both assignment sets: ai_levels/ in the Space repo. The openjev runner expects modeling_openjev.py from the AlexWortega repo next to it. Jev cost about $0.0002 per ten-assignment run.