Ru-Jev v1.3: release notes, known limitations, how to report results
u-Jev v1.3 is an open typed-decision model: a pointwise cross-encoder on mmBERT-base that scores caller-supplied options and runs on CPU. It is trained and tested on Russian-language data.
What changed in v1.3. Each training label appears in several surface forms. On held-out label vocabularies made of words (English-derived jargon, office slang, department names) v1.3 is more accurate than v1.1. On three-letter abbreviations it is not better, and opaque codes defeat both versions.
Known limitations.
One training run per release.
With 18 candidates, confident errors rise on jargon and slang, so refit the temperature on your own labels before thresholding on confidence.
On two datasets neither release saw, accuracy moved by −8.4 and +5.6 points.
Where to look.
Preprint: arXiv link to follow.
Code, per-row results and evaluation scripts: https://github.com/beskvladimir-create/ru-jev
Archived release: https://doi.org/10.5281/zenodo.23986847
v1.1 is on the v1.1 branch. Pin the revision if you need it.
Reporting results. If you test Ru-Jev on your own label set, please post the number of labels, the number of candidates per question, accuracy, and whether your labels are words, abbreviations or codes. Negative results are as useful as positive ones.