TigreGotico/jevasio-4b

Jevásio-4B

Jevásio-4B is a decision model for European Portuguese (pt-PT). It reads a state (a text or a JSON object) and a set of typed questions about it. It returns a probability over the options of each question, in one forward pass and without generated text. The question types are choice (pick one option), score (an ordered level) and noul (yes or no).

It is Kev-4B fine-tuned on TigreGotico/jevasio-decisions, a set of typed decisions built from the amalia-llm data of the AMALIA project. The request format and the serving code are those of Kev, so a Kev or System One client works with it unchanged.

Model details

Backbone	Qwen/Qwen3.5-4B-Base (revision 1001bb4d), frozen
Initialised from	jaredpalmer/kev-4b
Adapter	LoRA, rank 16, α 32, on the attention, MLP and DeltaNet projections (33.8M trainable parameters)
Head	Kev's pointer head: it scores each option's closing token against the question's final token
Calibration	One temperature, T = 1.70, stored in head.pt and applied at load time
Languages	European Portuguese values; English keys and option names
Licence	CC BY-NC-SA 4.0, for non-commercial research, because the training data is share-alike and non-commercial

How to use

Serve it with the Kev repository at commit 1f04cfa28e6a708485b28a9585010b9e436b776e:

git clone https://github.com/jaredpalmer/kev.git && cd kev && git checkout 1f04cfa28e6a708485b28a9585010b9e436b776e && uv sync --extra serve
uv run --extra serve python -m kev.serve --run TigreGotico/jevasio-4b --port 8008

from typesafe_sdk import Choice, Noul, TypeSafeClient

client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8008", model="kev-latest")
response = client.system_one(
    state={"request": "Fui cobrado duas vezes pela encomenda 1182. Podem devolver uma das cobranças?"},
    questions={
        "team": Choice(instructions="Que equipa deve tratar deste pedido?",
                       criteria={"billing": "Cobranças e reembolsos", "shipping": "Entregas", "returns": "Trocas"}),
        "urgent": Noul(instructions="Este pedido precisa de resposta hoje?"),
    },
)
print(response.choices["team"].choice, response.nouls["urgent"].noul)

Write the keys in English and the values in Portuguese, as in the training data: the state's field names, the question ids and the option names are code identifiers. KEV_TEMPERATURE=1.0 returns the probabilities before calibration.

Training

The training data is the train split of TigreGotico/jevasio-decisions (147,254 requests), mixed with 12,576 English records of Kev's decision-v7 suite as replay. Records with a state over 1,024 tokens were dropped, which left 140,058 of 159,830 records.

Epochs	1
Learning rate	2e-05
Effective batch	8 (4 × 2 accumulation)
Precision	bf16 autocast, gradient checkpointing
Kev options	--p_none 0 --p_none_distract 0 --p_distract 0 (no English "none of the above" options or distractors), --length_sort 1
Optimizer steps	17,508
Wall time	5.8 hours on one RTX 4090

The temperature is fitted on Portuguese questions from both sides of training: the calib split, from the sources the model trained on, and the test questions of the evaluation-only sources, which it never saw. A fit on calib alone leaves the model overconfident on new tasks. A fit on the unseen sources alone leaves it underconfident on the trained ones. To keep the unseen figure honest, each evaluation-only source is left out in turn and scored at the temperature fitted without it:

	ECE, temperature from calib only (T = 1.48)	ECE, combined temperature
Unseen sources, each left out of its own fit	0.092	0.086
Trained sources, test split	0.008	0.016

scripts/refit_temperature.py in the kev-pt code does the fit, and Kev's scripts/calibrate_checkpoint.py --temperature writes it into head.pt.

Evaluation

The test split holds 17,332 requests with 18,546 questions. Every request was scored, for this model and for stock Kev-4B. Eight sources are evaluation-only, so the model never trained on them: pt_exams, alba_mcq, alba_rating, cultura_viva, pt_text_completion, xstest, harmbench and sst. English transfer is Kev's transfer-v4 test suite.

	Stock Kev-4B	Jevásio-4B
Accuracy, pt-PT test	0.729	0.803
ECE, pt-PT test	0.045	0.013
Brier, pt-PT test	0.371	0.260
Accuracy, English transfer-v4	0.805	0.781
ECE, English transfer-v4	0.037	0.062

Per source, on the pt-PT test split. The tuned ECE of the evaluation-only sources is partly in-sample, because their questions are in the temperature fit; Training gives the left-out figures.

Source	Questions	Stock acc	Tuned acc	Stock ECE	Tuned ECE
alba_mcq	237	0.616	0.675	0.069	0.093
alba_rating	719	0.359	0.346	0.036	0.291
bbh_pt	144	0.424	0.639	0.091	0.072
corege_fos	1,000	0.705	0.860	0.045	0.050
csqa_pt	421	0.689	0.796	0.063	0.062
cultura_viva	2,463	0.562	0.527	0.053	0.052
dpo_pt	1,000	0.949	0.961	0.117	0.020
dsl_tl	147	0.823	0.823	0.078	0.051
egsciqa	1,000	0.252	0.641	0.377	0.049
frmt	4	0.500	0.750	0.245	0.180
harmbench	300	0.850	0.910	0.078	0.048
openbookqa_pt	478	0.741	0.849	0.082	0.038
piqa_pt	542	0.803	0.884	0.077	0.036
pt_culture	1,000	0.871	0.962	0.114	0.011
pt_exams	1,346	0.763	0.794	0.110	0.049
pt_text_completion	138	0.558	0.558	0.121	0.110
ptradutor	987	0.987	1.000	0.161	0.001
rewrite_faithful	1,000	0.845	1.000	0.075	0.001
saudade	458	0.755	0.834	0.036	0.044
siqa_pt	685	0.688	0.775	0.033	0.034
sst	100	0.930	0.910	0.133	0.058
summary_faithful	994	0.982	1.000	0.109	0.002
tierc_everyday_conv	260	0.719	0.815	0.137	0.059
tierc_persona_general	343	0.802	0.892	0.120	0.035
tierc_persona_nemotron	335	0.660	0.869	0.119	0.037
tierc_wikipedia_conversations	319	0.790	0.868	0.165	0.068
toxichat	478	0.964	0.992	0.211	0.011
wildguardmix	305	0.872	0.925	0.109	0.025
winogrande_pt	396	0.626	0.793	0.190	0.053
wmt24pp	497	0.716	0.781	0.082	0.025
xstest	450	0.809	0.833	0.082	0.017

Limits

• One temperature does not fit every task. On a Portuguese task it never saw, the ECE is about 0.086, and on alba_rating, a five-level quality rating, it is 0.318. Before you set a threshold, fit a temperature on labelled data from your own workload (python -m kev.calibrate).
• The variety question leans towards Brazilian Portuguese. On the dsl_tl and FRMT test texts, the model names European Portuguese texts correctly 40 of 49 times, and Brazilian texts 84 of 102 times. The training texts are two-thirds Brazilian.
• Portuguese knowledge did not improve. Accuracy on cultura_viva and alba_rating, two native pt-PT sets the model never trained on, stayed level or fell.
• Some sources are too easy to measure. The faithfulness, translation and preference sources score near 1.0, because their wrong options come from other rows.
• Most knowledge sources are machine translations into Portuguese. The native pt-PT sources (saudade, pt_culture, corege_fos, egsciqa, pt_exams, alba, cultura_viva) are the better test of pt-PT ability.
• The conversation questions (tierc) have teacher labels. AMALIA-9B-1225-DPO labelled them, and nobody checked its answers.
• This is not a model for decisions with legal, medical, financial or employment consequences for people without human review. It does not generate text.

Citation

The training data comes from the AMALIA project. Cite it, and Kev:

@inproceedings{simplicio-etal-2026-amalia,
    title = "{AMALIA}: A Fully Open Large Language Model for {E}uropean {P}ortuguese",
    author = "Simpl{\'i}cio, Afonso and Vinagre, Gon{\c{c}}alo and Ramos, Miguel Moura and Tavares, Diogo and Ferreira, Rafael and Attanasio, Giuseppe and Alves, Duarte M. and Calvo, In{\^e}s and others",
    booktitle = "Proceedings of the 17th International Conference on Computational Processing of {P}ortuguese ({PROPOR} 2026) - Vol. 1",
    year = "2026",
    address = "Salvador, Brazil",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.propor-1.38/",
    pages = "380--391"
}