Perfica-Dev/laya-multilingual-onnx

Laya multilingual, fine-tuned for Perfica's Quick judge, ONNX int8

A fine-tune of the multilingual checkpoint of Laya by Convai Innovations (github.com/NandhaKishorM/laya), exported to ONNX and quantised to int8. The model, its architecture, its training recipe and the weights this fine-tune starts from are Convai Innovations' work. Laya is licensed under Apache-2.0, and so is this derivative. See LICENSE and NOTICE.

Laya is a non-generative decision model: given a state (text) and a typed question (a choice, a score or a yes/no noul), it returns a probability for each option in one forward pass. Its multilingual checkpoint is mmBERT-base (22 layers, 256k vocabulary) with a two-layer decision head.

This is what Perfica's Quick judge downloads, on a person's consent, to run on their own CPU through onnxruntime-node. It is published so that download is a pinned, public file anyone can inspect.

Version

Fine-tune round r27, export "m" (int8 with the MLP down-projections kept fp32), published 2026-10-05. Perfica pins this revision by commit hash, file sizes and SHA-256, so a later upload never changes what an installed app runs. Each revision carries its own thresholds in the app; numbers measured on one revision are never carried to another.

It replaces revision 2083982bdba7a1651e96ecb090402d39244ea868 (round r25). Before that, e82c461c768b016aef2e2e41744d1a8a69ae8aa4 (round r21) had supervisor labels from a free, trial-only endpoint (below), and the first revision, b8c076a6ef163461ab1a207496a8a9344a780e85, is Laya's own weights exported unchanged (no fine-tune, 323 MB). All stay in this repository's history.

What Quick judge asks it

Eight questions from a coding-agent app (six of which the app still asks), each one asked as Laya's own typed question over a state the app writes:

Job	The question	In Perfica	Threshold (on the answer's probability)
supervisor	Is this agent's pass on track for its role, drifting, looping, or stuck?	decides: ON_TRACK without calling a model	p(ON_TRACK) ≥ 0.807
wait	Which event should a waiting loop wait for?	not asked by the app since 2026-10-05	(missed its bar on set A; it has no test set C)
needsPerson	Does a loop that stopped on an error need the person?	decides: resume once on its own	p(alone) ≥ 0.713
goalMet	Does a finished run's report meet the project's goal?	decides: may mark a finish unfinished, never the reverse	p(unmet) ≥ 0.713
command	Which of six kinds is a shell command a person was asked about?	records only	(met its bar on set A; the app has no path that acts on it yet)
mode	Which of the app's modes does a request need?	decides: the mode a request is sent in	p(top mode) ≥ 0.458
taskChoice	Which card on a task list fits this loop?	not asked by the app since 2026-10-05	(no threshold met its bar)
honesty	Is a PASS backed by what the pass ran?	decides, under its bar (below): asks the pass once more whether it did the work	p(unbacked) ≥ 0.908

Below its threshold, and for every job that only records, the app does what it did without Quick judge: it asks its model, or the person. wait and taskChoice stay in the model and are measured, but the app no longer asks them: its waiting loop was removed, and choosing a card is left to a larger model. The fifth of the later questions, screen, was not trained.

How it was fine-tuned

• Teacher-labelled, synthetic cases. Every case is composed by Perfica's own code from synthetic material: fictional tasks, projects, team prompts and shell commands written in bulk by Gemini 3.5 Flash-Lite, then rendered into the exact states and questions the app asks. No person's data is in any case. Each case was labelled by a teacher model asked the app's own prompt for that decision:
  • training: Gemma 4 31B and Gemma 4 26B-A4B, with Gemini 3.5 Flash-Lite when they were unavailable;
  • calibration and held-out sets: Gemini Flash models (3 preview, 3.5 to 3.8);
  • the supervisor, every split: NVIDIA Nemotron 3 Ultra (nvidia/nemotron-3-ultra-550b-a55b), bought through OpenRouter from a paid host, DeepInfra, with the route pinned to that host and no fallback. The model's licence (OpenMDW-1.1) puts no restriction on its outputs, and the host's terms put none on production use or on training with them. The previous revision's supervisor labels came from the model's free endpoint, whose trial terms allow evaluation only; all of them were bought again from the paid host, and none was used for this revision;
  • the four later questions' calibration and test sets are labelled by written rules.
• Recipe. Laya's own fine-tune recipe (RLCD over noisy logits plus soft cross-entropy, AdamW, cosine schedule) on one RTX 3060, with the 256k-token embedding frozen. One multi-task model serves every job. Round r10 fine-tuned Laya's checkpoint on 30,783 items (three epochs, the second kept); r27 continued from r10 for one epoch on 37,360 items, at half r10's learning rates, with the supervisor labelled by Nemotron on the paid host, seed 14: the previous revision's items plus 838 hard supervisor cases (drift, neighbouring roles, borderline passes), each seen twice. The choice temperature (1.436) was fitted on calibration.
• Export. ONNX opset 17 with dynamic batch, sequence and option axes; the decision head's nn.TransformerEncoder rewritten as the same arithmetic with shape-agnostic reshapes. Quantised with ONNX Runtime's dynamic int8 quantisation (MatMul and Gather weights), except the 22 MatMuls named /encoder/layers.N/mlp/Wo/MatMul, which stay fp32: quantising those cost the supervisor about eight points of agreement with its teacher. 1.29 GB in fp32, 364 MB here.

The scripts are in Perfica's source: scripts/quickJudge/train_laya.py and scripts/quickJudge/export_laya_onnx.py (with --fp32-matmuls 'mlp/Wo/').

How it was measured

Each job has up to four sets, never trained on, split by source material so no test case shares a scenario with a training case:

1. Calibration chooses the threshold: the lowest at which the cases the job would act on meet its bar.
2. Sets A and B are validation: every candidate round was read on A, and B was the test for the two revisions before this one, so it chooses nothing now.
3. Set C is the test: new material (56 supervisor scenarios sharing no task or main file with any other set, and new rule-built cases for the four later questions), its case ids frozen and published before it was labelled or read, read once for this revision at thresholds fixed before it was read. The jobs that passed on C decide in the app, and honesty, which did not, decides too (below). wait, command and taskChoice have no set C.

A bar is met only with a margin. The one-sided 90% Clopper–Pearson lower bound of the accuracy on the cases acted on must clear the bar, not the accuracy itself: 95% for the supervisor and goalMet, 90% for the others. For command and needsPerson the bar is "none wrong", read as no wrong case and a lower bound of at least 90%. Each job also has a minimum number of acted cases.

Set C at the shipped thresholds (acted of the set, wrong, accuracy, the lower bound):

Job	Acted	Wrong	Accuracy	Lower bound	Bar
supervisor	328 of 672	9	97.3%	95.7%	95%: met
mode	138 of 138	2	98.6%	96.2%	90%: met
needsPerson	52 of 128	0	100%	95.7%	none wrong: met
goalMet	91 of 144	0	100%	97.5%	95%: met
honesty	49 of 192	4	91.8%	84.3%	90%: not met; decides anyway (below)
wait, command, taskChoice	—	—	—	—	no set C (on set A: wait 192 of 230, 18 wrong, bound 87.3%, not met; command 41 of 105, none wrong, bound 94.5%, met; taskChoice no threshold)

Read these with care. "Right" means agreeing with the teacher's label (or the rule's), on synthetic cases, not with the truth of a real run. The sets are small: a few hundred cases at most per job. On set A the supervisor passed on both seeds of this recipe (bounds 96.2% and 95.7%), and this one acted on 101 cases there with one wrong. wait missed its bar on set A (bound 87.3%); the previous revision missed there too and passed on set B, a gap traced to one board shape the teacher labelled differently across the sets. The app no longer asks wait at all.

honesty decides under its bar, by the app owner's choice. On set C its lower bound is 84.3% against a 90% bar (49 acted, 4 wrong). It is asked only about a PASS that every exact check in the app already let through, and acting on it only asks that pass once more whether it did the work. A wrong "unbacked" costs one more turn; a wrong "backed" is what the app did without Quick judge.

Files

File	What it is
laya-multilingual.int8.onnx	The model. Inputs: input_ids, attention_mask (int64, [batch, seq]), marker_pos (int64, [batch, options]), marker_mask (bool, [batch, options]), qtype (int64, [batch]: 0 choice, 1 score, 2 noul). Outputs: logits ([batch, options]) and act_logits.
tokenizer.json, tokenizer_config.json	Laya's multilingual tokenizer, unchanged.
laya_config.json	maxLen 1024, headMaxLen 256, the per-type temperatures, the source revision and the fine-tune's round.

Inputs are laid out as in Laya's rl_common.build_sequence: [CLS] <type> question: <instructions> [SEP] [MASK] option… [SEP] <state> [SEP], and each option is scored at its [MASK].

Measured on a CPU

Measured on the previous revision, whose file has the same graph and size: on an AMD Ryzen 5 3600 (6 cores), onnxruntime-node 1.30, four threads, one question at a time on Perfica's own prompts: loading takes 2.7–3.5 s, the process holds 0.8–0.9 GB, and a warm decision takes 0.1–0.2 s for the short questions and about 0.4 s for the supervisor (median). The fp32 MLP down-projections cost no measurable time against a fully int8 file.

Licence

Apache-2.0. Laya is © Convai Innovations; this fine-tune, its export and its documentation are by Perfica (Uezar Labs). The mmBERT-base encoder Laya is built on is by JHU CLSP (jhu-clsp/mmBERT-base, MIT).