JevK5-Lite (preview, experimental)

JevK5-Lite is a CPU classifier. It reads a text and any number of label sets ("heads") in one encoder pass. It returns a calibrated probability for every label: a softmax within a single-label head, and a sigmoid per label in a multi-label head. It is a 437M-parameter DeBERTa-v3-large, fine-tuned by the JevK5 project. This is the preview-1 release; the runtime is jevk5 0.3.1.

Where it is better: calibration. On a neutral test of seven public datasets, chosen before either model was run, its probabilities are better calibrated than GLiNER2.5-Decide's. The top-label expected calibration error is lower on all six single-label sets: 0.035-0.103 against 0.091-0.250.

Where it is not: accuracy. This is a preview, and it does not beat the model it was built against.

• Clear wins: JevK5-Lite is significantly more accurate on 2 of the 7 neutral sets, AG News and Yahoo Answers.
• Within noise: it is ahead on Enron spam and tweet_eval emotion only by the 0.02 margin; the 95% intervals include zero. tweet_eval sentiment is a tie.
• Decide's wins: GLiNER2.5-Decide is clearly better on Financial PhraseBank (-0.262) and GoEmotions (-0.074), and its mean over the seven sets is higher (0.643 against 0.629).
• fast-decisions: on fastino/fast-decisions dev, GLiNER2.5-Decide is stronger: head accuracy 0.637 against 0.587, all heads right 0.488 against 0.442. It is also better calibrated there (ECE 0.137 against 0.255), so JevK5-Lite's calibration does not hold under every shift.

GLiNER2.5-Decide's training data is not public, so "neither model trained on these sets" rests on Fastino's disclosure. Both models are DeBERTa-v3-large; this card makes no speed claim.

Use

pip install "jevk5[lite] @ git+https://github.com/allebee/jevk5@v0.3.1"

from jevk5 import JevK5Lite

lite = JevK5Lite.from_pretrained("alibiserikbay/JevK5-Lite", threads=16)
lite.classify(
    "I was charged twice for the same order, please refund one of them.",
    {"intent": ["refund_request", "order_status", "cancel_subscription"],
     "areas": {"labels": ["billing", "shipping", "account"], "multi_label": True}},
)
# {'intent': {'labels': ['refund_request'], 'probabilities': {'refund_request': 0.9995, ...}},
#  'areas': {'labels': ['billing'], 'probabilities': {'billing': 0.957, 'shipping': 0.004, 'account': 0.005}}}

• Precision: dtype=torch.bfloat16 roughly halves the latency on CPUs with bf16 support (AMX, AVX512-BF16).
• Length: inputs are cut to 512 tokens, labels first, so a very long label list leaves less room for the text.
• Threshold: a multi-label head returns every label with p >= 0.5; set threshold per task to change it.

How it reads

One sequence per call: [CLS] [TASK] task (one|any) [LABEL] label ... [TASK] ... [SEP] text [SEP]. Each label is scored from the hidden state of its [LABEL] marker and the mean of the text's hidden states, by a small MLP on [label, text, label * text]. There is one calibration temperature for single-label heads (1.87) and one for multi-label heads (1.32). The design follows the label-conditioned uni-encoder idea used by GLiNER2 and GLiClass.

Evaluation

fastino/fast-decisions dev

1,700 rows, 17 domains, head accuracy averaged over domains. CPU, 16 threads. This split was used for evaluation only, once, after the model was chosen.

Model	Params	Head accuracy	Single-label	Multi-label	All heads right	ECE (single)
GLiNER2.5-Decide	340M (Fastino's count)	0.637	0.686	0.227	0.488	0.137
JevK5-Lite	437M	0.587	0.633	0.217	0.442	0.255

JevK5-Lite leads on restaurant reviews, screen tags and ticket routing. It trails most on paper field, product feedback, support intent, news topic and banking intent.

Neutral test (seven public datasets)

• Items and models: up to 500 items per set, drawn from validation or test splits. Every model got the same text, task name and the dataset's own label names. Both models ran in bf16 on a CPU with 16 threads, in one run.
• Fixed in advance: the sets, metrics and decision rule were written down before either model was run.
• Training data: none of these datasets is in JevK5-Lite's training data. GLiNER2.5-Decide's training data is not public, so "neither model trained on these sets" rests on Fastino's disclosure.
• Metric and wins: the metric is macro-F1, or exact-set accuracy for GoEmotions. A "win" means a margin of at least 0.02, or a paired bootstrap 95% CI (1,000 resamples) entirely above 0.

Dataset (split)	Task	JevK5-Lite	GLiNER2.5-Decide	Difference [95% CI]	ECE Lite / Decide
AG News (test)	4 topics	0.739	0.680	+0.059 [+0.027, +0.092]	0.103 / 0.250
Yahoo Answers (test)	10 topics	0.674	0.578	+0.095 [+0.057, +0.135]	0.050 / 0.174
Enron spam (test)	ham / spam	0.722	0.680	+0.041 [-0.004, +0.089]	0.035 / 0.098
tweet_eval emotion (test)	4 emotions	0.762	0.726	+0.036 [-0.011, +0.086]	0.051 / 0.091
tweet_eval sentiment (test)	3 classes	0.653	0.640	+0.012 [-0.032, +0.057]	0.082 / 0.104
GoEmotions (test), multi-label	28 emotions	0.214	0.288	-0.074 [-0.114, -0.030]	—
Financial PhraseBank, all-agree	3 classes	0.643	0.905	-0.262 [-0.323, -0.204]	0.046 / 0.152
Mean		0.629	0.643

• Significant wins: JevK5-Lite wins AG News and Yahoo Answers; their intervals lie above zero.
• Within noise: Enron spam and tweet_eval emotion are ahead by the 0.02 margin only, with intervals that include zero. tweet_eval sentiment is a tie.
• Decide's wins: GLiNER2.5-Decide wins Financial PhraseBank and GoEmotions, and has the higher mean.
• Calibration: JevK5-Lite's top-label calibration error is lower on all six single-label sets.
• Latency: the median per set was 54-92 ms per item for JevK5-Lite and 54-97 ms for GLiNER2.5-Decide, in bf16 on 16 CPU threads.

Calibration

The temperatures were fitted on held-out synthetic data from the same generator as part of the training data. On data unlike that, the fit can be wrong in either direction. On fast-decisions dev it was overconfident (ECE 0.255). On the six single-label neutral sets it was better calibrated than GLiNER2.5-Decide.

Training

• Start: from microsoft/deberta-v3-large, one epoch over a mix of two sources.
• Public rows: 189,026 rows from public train splits: BANKING77, CLINC150, MASSIVE, HWU64, SNIPS, four Bitext assistant datasets, DBpedia-14, Civil Comments, arXiv abstracts and Schema-Guided Dialogue.
• Synthetic documents: 20,822 operational documents, repeated three times in the mix. They are support chats and emails, tickets, banking, insurance, clinic and travel requests, HR, reviews, news, sports, moderation and more. GPT-6 Luna wrote them with 1-4 label heads each, and a head was kept only when two independent solves agreed with the author.
• Label augmentation: candidate subsets, shuffled order, naming styles and task names were varied, so the model learns what labels mean rather than a fixed class list.
• Selection: on held-out slices of the public data, on held-out synthetic settings, and on held-out BANKING77/CLINC150/MASSIVE items offered with every label.
• Checks: every training row was checked against fast-decisions dev, JevBench's public items and the Jev Decision Index's test and validation sets. The check covered exact matches and shared 8-word sequences.

Limitations

• English only. It is a preview: expect uneven quality across domains, as the tables show.
• Financial sentiment and fine-grained multi-label emotion are weak.
• Calibration holds on some data and not on others (see above).
• 437M parameters, above the 400M the project aimed for.

License and notices

The weights are released under Apache-2.0. The base model, microsoft/deberta-v3-large, is MIT.

GPT-6 Luna by OpenAI wrote and checked 20,822 of JevK5-Lite's training documents through OpenAI's API. Those outputs are subject to OpenAI's terms.

Training data under their own licenses:

• BANKING77, MASSIVE and HWU64: CC BY 4.0
• CLINC150: CC BY 3.0
• SNIPS, Civil Comments and arXiv metadata: CC0 1.0
• Bitext customer support, retail banking, travel and insurance: CDLA-Sharing 1.0
• DBpedia-14: CC BY-SA 3.0
• Schema-Guided Dialogue: CC BY-SA 4.0

fastino/fast-decisions dev was used for evaluation only.

JevK5-Lite is not affiliated with Fastino or TypeSafe AI.