Ру-Жев (Ru-Jev) v1.1

Open, Russian-first typed-decision model. It does not generate text. Given a context and a set of candidate options, it returns a calibrated probability over the options and picks one. It covers three decision shapes: yes/no, choice from options, and ordinal scale. It is a compact encoder (mmBERT-base, 307M), so it runs on CPU and can be deployed inside a closed perimeter.

Project page: https://ru-jev.besk.tech Training and evaluation code: https://github.com/beskvladimir-create/ru-jev

The previous release is kept on the v1 branch of this repository. Load it with revision="v1" if you depend on its exact behaviour.

The inference widget is switched off on purpose. This is a backbone plus a separate scoring head, not a standard sequence-classification checkpoint, so the stock widget cannot load it. Use the snippet below.

Not affiliated with TypeSafe AI. Not trained on Jev outputs. Ru-Jev is trained from scratch on open, human-labelled datasets. The name is a play on the category, not a claim of compatibility or endorsement.

What changed since v1

v1 could not route. It was trained on natural language inference, sentiment and toxicity, and no task in its mix ever presented category names as the options, so asking it to pick a department from a list left it at roughly chance.

Task	v1	v1.1
Routing, 60 intents	0.344	0.965
Routing, 18 scenarios	0.305	0.954
Reviews, 5 levels	0.391	0.629
Reasoning questions	0.327	0.522
All 13 tasks	0.663	0.825

On the nine tasks v1 was trained for, scored on identical rows, v1.1 is ahead overall as well, 0.843 against 0.824, with the gains on acceptability and sentiment and losses on rcb and russe.

Calibration moved the other way on those nine tasks: 0.009 for v1 against 0.020 for v1.1, both measured held out. On the full thirteen-task mix the order reverses, 0.025 for v1 against 0.020 for v1.1, because a single temperature cannot serve a model that is at chance on half the work. v1.1 is the better trade: two points of calibration on a narrow slice bought routing that works.

Results

Measured on half of the validation split, 13,373 decisions, with the temperature fitted on the other half.

Metric	Value
Accuracy (micro)	0.825
ECE, temperature fitted on a disjoint half	0.0200
ECE, temperature fitted on this same data	0.0159
Shipped temperature	2.2754

Two ECE figures, because the number depends on where the temperature came from. Fitting and scoring on the same rows is in-sample and optimistic. The first figure is what to expect on new data of the same kind, and it is the one to quote.

Per task, full validation split:

Task	Accuracy	n
massive_intent (60 labels)	0.965	3000
toxicity	0.961	3000
massive_scenario (18 labels)	0.954	3000
russe (word sense)	0.822	8505
terra (entailment)	0.775	307
rureviews (sentiment)	0.772	3000
danetqa (yes/no)	0.742	821
parus (choice)	0.680	100
rucola (acceptability)	0.650	1344
georeview (5 levels)	0.629	3000
rwsd (coreference)	0.564	204
mera_qa (reasoning)	0.522	245
rcb (3-way NLI)	0.450	220

Name your options carefully

The model reads the option text. How you word your categories changes the answer, and the effect is large. Three hundred routing questions, the same eighteen categories every time, only the wording differs:

Wording	Example	Accuracy	Confident errors
Russian word	погода	0.973	3
English key	weather	0.963	4
Descriptive sentence	человек спрашивает про погоду	0.953	2
Russian phrase	прогноз погоды	0.890	9
Department name	метеослужба	0.870	10
Jargon, transliterated	вэзер	0.630	26

Write options as ordinary words that describe the category. Avoid borrowed jargon: биллинг costs far more accuracy than оплата, счета и возвраты. The last column counts answers that were wrong while above 0.8 confidence, so on unfamiliar wording the probability stops being a reliable guide as well. If you must use in-house jargon, measure it on your own data first with scripts/robustness.py from the code repository.

Withholding an answer

A calibrated probability is worth something only if the caller acts on it. The serving code in the repository takes a threshold and withholds anything below it for a person to handle.

Share answered automatically	Threshold	Accuracy on those	Errors per 1000
30%	0.98	99.3%	7
50%	0.85	97.1%	29
60%	0.77	95.1%	49
80%	0.60	89.5%	105
100%	none	82.5%	175

Per decision shape at 95% accuracy or better, routing over many options clears the entire stream, while yes/no questions and small choices clear about half.

What a cheap baseline gets

Task	Majority class	TF-IDF + logreg	Ru-Jev v1.1
massive_scenario	0.066	0.435	0.954
massive_intent	0.063	0.820	0.965
terra	0.498	0.394	0.775
danetqa	0.502	0.568	0.742
russe	0.631	0.701	0.822
georeview	0.199	0.544	0.629
rureviews	0.324	0.720	0.772
rucola	0.587	0.609	0.650
toxicity	0.766	0.943	0.961
rwsd	0.554	0.554	0.564

The model earns its place on routing, entailment and yes/no questions. On toxicity it is 1.8 points over character n-grams and on sentiment 5.2 points, so for those tasks alone a linear model is cheaper to run and easier to debug.

Limitations

• Coreference is not supported. On rwsd the model matches the majority class exactly, 0.564 against 0.554. It does not resolve pronouns.
• Multi-step reasoning is weak. rcb 0.450 and mera_qa 0.522. A 307M encoder does not chain inferences, and no amount of further training on this architecture will change that.
• Unfamiliar label wording degrades both the answer and the confidence, as the table above shows.
• Russian first. The multilingual base handles other languages, but they were not measured.

Speed on CPU

Apple M4, four threads, median of 25 runs after warmup, this checkpoint in float32.

Request	Tokens	Median	p95
yes/no, short context	23	24.5 ms	26.5 ms
choice of three	17	31.2 ms	32.6 ms
yes/no, full 256-token context	315, truncated to 256	133.2 ms	141.1 ms

On a single thread the short requests are unchanged and the full context costs 186 ms. A server CPU will give different figures.

Usage

import torch, torch.nn as nn, json
from transformers import AutoTokenizer, AutoModel
from huggingface_hub import hf_hub_download

REPO = "besktech/ru-jev"
tok = AutoTokenizer.from_pretrained(REPO)
backbone = AutoModel.from_pretrained(REPO).eval()
head = nn.Linear(backbone.config.hidden_size, 1)
head.load_state_dict(torch.load(hf_hub_download(REPO, "head.pt"), map_location="cpu"))
T = json.load(open(hf_hub_download(REPO, "calibration.json")))["temperature"]

@torch.no_grad()
def decide(context, options):
    enc = tok([context] * len(options), [str(o) for o in options],
              truncation=True, max_length=256, padding=True, return_tensors="pt")
    scores = head(backbone(**enc).last_hidden_state[:, 0]).squeeze(-1)
    probs = torch.softmax(scores / T, dim=0)
    return list(zip(options, probs.tolist()))

decide("Клиента списали дважды", ["да", "нет"])
decide("Не приходит счёт", ["оплата, счета и возвраты", "техническая проблема"])
decide("Отличный сервис, всё быстро", ["негативный", "нейтральный", "позитивный"])

Training data

Open datasets only, all permitting commercial use:

• Russian SuperGLUE (MIT): terra, danetqa, russe, rwsd, parus, rcb
• MASSIVE (CC-BY-4.0): intent and scenario routing
• Georeview (MIT): 5-level reviews
• MERA (MIT): reasoning questions
• RuReviews (Apache-2.0): sentiment
• RuDetoxifier data (MIT): toxicity
• RuCoLA (Apache-2.0 labels): acceptability

Boolean tasks are trained with five wordings of the same answer, and options are reordered on every epoch, so the model cannot learn a position or a single surface form.

License

MIT.

Vladimir Beskorovainyi, Enterprise AI Architect & CTO, author of the Amazon bestseller "Enterprise AI: The KATA Way", founder of Besk Tech (besk.tech)