Independent Laya vs Jev head-to-head on identical inputs
Ran Laya (this checkpoint, local CPU) and Jev (pinned jev-1.13.0, API) on the same 100 states, same question dicts, same seed. Question hashes verified identical before comparing, so the inputs match byte for byte.
Results (310 decisions):
| Suite | Laya | Jev |
|---|---|---|
| triage (n=160) | 0.800 | 0.894 |
| guardrails (n=60) | 0.883 | 0.950 |
| moderation (n=90) | 0.833 | 0.989 |
The gap sits in multi-class questions: 6-way intent (Laya 0.725, Jev 0.975) and toxic (0.767 vs 1.000). The one Laya win is churn_risk (0.800 vs 0.750). Binary flags are close throughout. ECE 0.065-0.082 (Laya) vs 0.034-0.055 (Jev). Gating at 0.85: Laya 69% coverage at 0.953, Jev 83% at 0.988.
Latency per 5-question call: Laya 375-476 ms local CPU, Jev 885-1017 ms over the API.
Limits: hand-labeled cases, one author, wide CIs (guardrails gap is inside the noise), English only, preset subsets. Full harness, raw JSONs, and writeup here: https://github.com/instax-dutta/sysone-bench
thanks for posting this!
Do I understand correctly that your eval does not use the router between the three released checkpoints, as is recommended?
Thanks!
Correct. No Router. Every Laya number is the English checkpoint at repo root, defaults, single forward pass.
Two reasons. First, the comparison is single model vs single model: one checkpoint against one pinned Jev version. Routing is a system on top, and scoring it would mix checkpoint quality with routing quality. Second, your own BENCHMARKS.md routes the same way for analysis, so base-checkpoint numbers stay comparable to yours.
The cost of skipping it is visible in my data: multilingual intent scores 0.360 on the English checkpoint against Jev at 1.000. That gap is the Router use case, stated in numbers.
Planned v3: a Router runner in the same harness, same states, so routed-Laya vs Jev becomes its own column instead of an argument. Harness is open if you want to point at anything: https://github.com/instax-dutta/sysone-bench
v3 is done, same harness: routed-Laya reproduces every English number exactly and lifts multilingual intent 0.360 to 0.840 (Jev 1.000 on the same states). Detail worth knowing: the router keys on script, so Spanish/French/German stayed on the English checkpoint and only 15 of 590 calls went multilingual. Run files: https://github.com/instax-dutta/sysone-bench/tree/master/results
v4: fourth column added, same 751 states. Qwen2.5-1.5B-Instruct with parallel constrained decoding (the harshatheg RLCD engine, which ships no fine-tuned weights, so stock 4bit Instruct under it).
It lands between the two of you on triage (0.825) and guardrails (0.900), ties Jev on sst5 (0.617), hits 0.880 multilingual, but falls to 0.500 on 12-way banking and 0.700 moderation. Emotion stays a three-model wall at 0.54-0.55.
Full table and run files: https://github.com/instax-dutta/sysone-bench
bravo