alfred361/laya-multilingual-typed-decisions

Laya multilingual, fine-tuned on typed decisions

convaiinnovations/laya-multilingual (mmBERT-base, 322M) fine-tuned on LocalLLaMA/typed-decisions with the authors' own recipe, unchanged except for the base model.

The English base already had a fine-tune of this kind; the multilingual one did not.

Measured

400 test cases, 2000 decisions, official test split.

model	accuracy	Brier	ECE	score MAE
this checkpoint	0.749	0.063	0.112	0.247
laya-multilingual before this fine-tune	0.352
teacher self-agreement ceiling	0.735
TypeSafe Jev 1.13.0 (published)	0.727	0.148	0.144	0.391
per-question majority class	0.461
laya-typed-decisions, English (published)	0.766	0.062	0.213	0.242

0.352 is measured here, not quoted: the base model was run over the same test split through the same code before training, which is also what makes the before/after a single comparison rather than two harnesses. The published figure for that base is 0.342, so the harness agrees with upstream to within 0.01.

Accuracy lands 1.8 points below the English fine-tune and above both the published Jev number and the teacher ceiling. Calibration is the larger difference: ECE 0.112 against the English fine-tune's 0.213. That is not a better training run -- the English checkpoint ships temperature_by_options inherited from its own base, which overrides the temperatures fitted for it, and its card says so. This one has temperature_by_options: {} and temperatures fitted at the end of training.

By workflow

workflow	accuracy
agent trace observability	0.720
customer service	0.736
invoice processing	0.792
security incidents	0.746

What is not measured

The training split is English only. This is a multilingual encoder taught English decisions, and the benchmark has no non-English cases, so nothing here says whether the multilingual property survived. Treat cross-lingual use as untested rather than supported until somebody measures it -- the right metric is agreement between languages on the same case, which needs no new labels, only translated questions.

Inherited from the base and the data: this is a specialist in four synthetic workflows (agent traces, customer service, invoice processing, security incidents). Outside them, expect the base checkpoint's behaviour.

Training

71.4 ms per case at p50 on a T4. 4 epochs, 2xT4 DDP, RLCD with a strictly proper scoring rule, temperatures refitted afterwards -- the recipe from laya_finetune_typed_decisions_2xT4_kaggle.ipynb, with MODEL_ID pointed at the multilingual base.

Apache-2.0, as upstream. Base model and recipe by Convai Innovations; benchmark by LocalLLaMA.