TigreGotico/jevasio-4b-v2

Jevásio-4B-V2

Jevásio-4B-V2 is a decision model for European Portuguese (pt-PT). It reads a state (a text or a JSON object) and a set of typed questions about it. It returns a probability over the options of each question, in one forward pass and without generated text. The question types are choice (pick one option), score (an ordered level) and noul (yes or no).

It is Jevásio-4B trained further on TigreGotico/jevasio-decisions-v2. That dataset holds every record of the first version and adds 36 sources: the language variety of a text in six domains, intent and scenario, news section and genre, legal area, parliament votes, entailment and similarity, offensive language and hate speech, puns, generated pt-PT messages, and copies of the faithfulness pairs with one fact changed. The request format and the serving code are those of Kev, so a Kev or System One client works with it unchanged.

Model details

Backbone	Qwen/Qwen3.5-4B-Base (revision 1001bb4d), frozen
Initialised from	TigreGotico/jevasio-4b
Adapter	LoRA, rank 16, α 32, on the attention, MLP and DeltaNet projections (33.8M trainable parameters)
Head	Kev's pointer head: it scores each option's closing token against the question's final token
Calibration	One temperature, T = 1.78, stored in head.pt and applied at load time
Languages	European Portuguese values; English keys and option names
Licence	CC BY-NC-SA 4.0, for non-commercial research, because the training data is share-alike and non-commercial

How to use

Serve it with the Kev repository at commit 1f04cfa28e6a708485b28a9585010b9e436b776e:

git clone https://github.com/jaredpalmer/kev.git && cd kev && git checkout 1f04cfa28e6a708485b28a9585010b9e436b776e && uv sync --extra serve
uv run --extra serve python -m kev.serve --run TigreGotico/jevasio-4b-v2 --port 8008

from typesafe_sdk import Choice, Noul, TypeSafeClient

client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8008", model="kev-latest")
response = client.system_one(
    state={"request": "Fui cobrado duas vezes pela encomenda 1182. Podem devolver uma das cobranças?"},
    questions={
        "team": Choice(instructions="Que equipa deve tratar deste pedido?",
                       criteria={"billing": "Cobranças e reembolsos", "shipping": "Entregas", "returns": "Trocas"}),
        "urgent": Noul(instructions="Este pedido precisa de resposta hoje?"),
    },
)
print(response.choices["team"].choice, response.nouls["urgent"].noul)

Write the keys in English and the values in Portuguese, as in the training data: the state's field names, the question ids and the option names are code identifiers. KEV_TEMPERATURE=1.0 returns the probabilities before calibration.

Training

Training continues from Jevásio-4B and does not start again from Kev. It reads 267,950 of the 335,063 train requests of jevasio-decisions-v2:

• every request of the 36 new sources (196,762);
• every request of the 4 first-version sources that have edited copies, because each original is one side of a pair (38,008);
• a fixed third of the other first-version requests, chosen by a hash of the state, so the model keeps what it learned (33,180).

They are mixed with 12,576 English records of Kev's decision-v7 suite as replay. Records with a state over 1,024 tokens were dropped, which left 263,696 of 280,526 records.

Epochs	1
Learning rate	2e-05
Effective batch	8 (4 × 2 accumulation)
Precision	bf16 autocast, gradient checkpointing
Kev options	--p_none 0 --p_none_distract 0 --p_distract 0 (no English "none of the above" options or distractors), --length_sort 1
Optimizer steps	32,962
Wall time	13.6 hours on one RTX 4090

The temperature is fitted on Portuguese questions from both sides of training: the calib split, from the sources the model trained on, and the test questions of the evaluation-only sources, which it never saw. A fit on calib alone leaves the model overconfident on new tasks. A fit on the unseen sources alone leaves it underconfident on the trained ones. To keep the unseen figure honest, each evaluation-only source is left out in turn and scored at the temperature fitted without it:

	ECE, temperature from calib only (T = 1.55)	ECE, combined temperature
Unseen sources, each left out of its own fit	0.128	0.113
Trained sources, test split	0.012	0.008

scripts/refit_temperature.py in the kev-pt code does the fit, and Kev's scripts/calibrate_checkpoint.py --temperature writes it into head.pt.

Evaluation

The test split of jevasio-decisions-v2 holds 37,972 requests with 46,661 questions. Every request was scored, for this model and for Jevásio-4B, each at its own temperature. Nine sources are evaluation-only, so neither model trained on them: alba_mcq, alba_rating, cultura_viva, harmbench, porvid_gold, pt_exams, pt_text_completion, sst, xstest. English transfer is Kev's transfer-v4 test suite.

	Jevásio-4B	Jevásio-4B-V2
Accuracy, pt-PT test	0.743	0.839
ECE, pt-PT test	0.045	0.006
Brier, pt-PT test	0.352	0.227
Accuracy, English transfer-v4	0.777	0.797
ECE, English transfer-v4	0.066	0.075

Accuracy by kind of source, weighted by questions:

Sources	Questions	Jevásio-4B	Jevásio-4B-V2
New in V2, edited copies included	29,372	0.710	0.863
First-version sources it trained on	11,536	0.885	0.879
Evaluation-only, never trained	5,972	0.635	0.646

Per source, on the pt-PT test split. The ECE of the evaluation-only sources is partly in-sample, because their questions are in the temperature fit; Training gives the left-out figures.

Source	Questions	Stock acc	Tuned acc	Stock ECE	Tuned ECE
alba_mcq	237	0.679	0.705	0.089	0.116
alba_rating	719	0.346	0.371	0.291	0.298
assin2	2,000	0.637	0.789	0.099	0.026
assin_ptbr	2,000	0.589	0.805	0.113	0.043
assin_ptpt	2,000	0.622	0.792	0.080	0.049
bbh_pt	144	0.639	0.694	0.072	0.093
boolq	868	0.869	0.886	0.063	0.035
cb	13	0.846	0.923	0.166	0.227
copa	33	0.909	0.939	0.106	0.042
corege_fos	1,000	0.862	0.869	0.047	0.049
csqa_pt	421	0.798	0.777	0.065	0.049
cultura_viva	2,463	0.527	0.544	0.052	0.122
dpo_pt	1,000	0.960	0.951	0.019	0.029
dsl_tl	147	0.823	0.810	0.051	0.136
egsciqa	1,000	0.636	0.631	0.045	0.060
fortuna	279	0.713	0.789	0.150	0.049
frmt	4	0.750	0.750	0.180	0.227
harmbench	300	0.910	0.923	0.045	0.062
hatebr	344	0.698	0.930	0.127	0.045
massive	1,994	0.816	0.922	0.055	0.062
mrpc	142	0.739	0.859	0.097	0.045
olid_br	2,533	0.587	0.785	0.152	0.057
openbookqa_pt	478	0.849	0.851	0.038	0.027
parliament	272	0.408	0.651	0.278	0.102
piqa_pt	542	0.882	0.880	0.041	0.043
porvid_gold	219	0.886	0.927	0.106	0.039
porvid_journalistic	826	0.995	0.971	0.090	0.016
porvid_legal	917	0.800	0.816	0.108	0.144
porvid_literature	1,000	0.682	0.924	0.048	0.019
porvid_politics	985	0.994	0.987	0.154	0.008
porvid_social_media	697	0.802	0.703	0.054	0.209
porvid_web	996	0.999	0.993	0.044	0.006
pt_culture	1,000	0.962	0.966	0.013	0.010
pt_exams	1,346	0.797	0.788	0.047	0.026
pt_text_completion	138	0.551	0.572	0.117	0.151
ptradutor	987	1.000	0.989	0.002	0.006
ptradutor_hard	491	0.002	0.957	0.966	0.018
publico	1,812	0.724	0.893	0.084	0.058
puntuguese	1,000	0.630	0.791	0.044	0.026
rewrite_faithful	1,000	1.000	0.995	0.001	0.005
rewrite_faithful_hard	493	0.030	0.996	0.918	0.013
rte	101	0.842	0.911	0.096	0.032
saudade	458	0.832	0.830	0.042	0.053
siqa_pt	685	0.775	0.777	0.033	0.047
sst	100	0.910	0.910	0.058	0.068
sst2	298	0.909	0.936	0.020	0.023
stj	1,000	0.984	0.991	0.094	0.019
stsb	512	0.561	0.652	0.120	0.026
summary_faithful	994	1.000	0.982	0.002	0.018
summary_faithful_hard	508	0.157	0.959	0.767	0.040
synth_offensive	180	0.994	1.000	0.102	0.012
synth_sentiment	130	1.000	1.000	0.052	0.019
synth_support_route	139	0.871	0.957	0.103	0.018
synth_urgency	102	0.745	1.000	0.079	0.017
synth_variety	124	0.984	1.000	0.178	0.029
tierc_everyday_conv	260	0.819	0.846	0.051	0.031
tierc_persona_general	343	0.892	0.892	0.035	0.031
tierc_persona_nemotron	335	0.866	0.878	0.045	0.041
tierc_wikipedia_conversations	319	0.868	0.881	0.071	0.047
toxichat	478	0.992	0.987	0.011	0.021
tupy	2,000	0.830	0.884	0.046	0.055
tweet_sentiment	869	0.662	0.761	0.114	0.065
wildguardmix	305	0.925	0.928	0.025	0.031
winogrande_pt	396	0.790	0.760	0.051	0.134
wmt24pp	497	0.779	0.744	0.023	0.062
wmt24pp_hard	238	0.105	0.937	0.583	0.083
xstest	450	0.833	0.833	0.016	0.020

Limits

• The gain is on the tasks it trained on. On the nine evaluation-only sources, accuracy went from 0.635 to 0.646. Do not expect better answers on a new kind of question.
• Checking a fact costs some faithful answers. On the edited copies, which change one fact, accuracy went from 0.070 to 0.966. On the faithful originals it went from 0.968 to 0.954: the first version called almost every pair faithful.
• One temperature does not fit every task. On a Portuguese task it never saw, the ECE is about 0.113, and on alba_rating, a five-level quality rating, it is 0.311. Before you set a threshold, fit a temperature on labelled data from your own workload (python -m kev.calibrate).
• The variety answer is uneven between the two varieties. On the dsl_tl and FRMT test texts, the model names European Portuguese texts correctly 46 of 49 times, and Brazilian texts 76 of 102 times.
• Portuguese knowledge did not improve. Accuracy on cultura_viva went from 0.527 to 0.544, and on pt_exams from 0.797 to 0.788.
• English is not a target. Accuracy on transfer-v4 went from 0.777 to 0.797, and its ECE from 0.066 to 0.075.
• Part of the data is silver or generated. The variety labels of the porvid sources come from the source of each text, and a blind label by Claude Haiku removed the texts it disagreed with. Claude Haiku wrote the edited copies. Claude Sonnet wrote the generated messages, and a blind label by Claude Haiku kept only those it agreed with. AMALIA-9B-1225-DPO labelled the conversation questions (tierc). Most knowledge sources are machine translations.
• This is not a model for decisions with legal, medical, financial or employment consequences for people without human review. It does not generate text.

Citation

The first-version data comes from the AMALIA project. Cite it, and Kev:

@inproceedings{simplicio-etal-2026-amalia,
    title = "{AMALIA}: A Fully Open Large Language Model for {E}uropean {P}ortuguese",
    author = "Simpl{\'i}cio, Afonso and Vinagre, Gon{\c{c}}alo and Ramos, Miguel Moura and Tavares, Diogo and Ferreira, Rafael and Attanasio, Giuseppe and Alves, Duarte M. and Calvo, In{\^e}s and others",
    booktitle = "Proceedings of the 17th International Conference on Computational Processing of {P}ortuguese ({PROPOR} 2026) - Vol. 1",
    year = "2026",
    address = "Salvador, Brazil",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.propor-1.38/",
    pages = "380--391"
}