Jevhakim
Collection
Decision models for Arabic, MSA and dialects: Kev fine-tuned on typed decisions from 80 public datasets. • 4 items • Updated
Jevhakim-4B
Jevhakim-4B is a decision model for Arabic: Modern Standard Arabic and the dialects. It reads a state (a text or a JSON object) and a set of typed questions about it. It returns a probability over the options of each question, in one forward pass and without generated text. The question types are choice (pick one option), score (an ordered level) and noul (yes or no).
It is Kev-4B trained further on Arabic decisions from 58 sources, all built from public datasets. A further 22 sources are held out for evaluation only. The request format and the serving code are those of Kev, so a Kev or System One client works with it unchanged.
Model details
| Backbone | Qwen/Qwen3.5-4B-Base (revision 1001bb4d), frozen |
| Initialised from | jaredpalmer/kev-4b |
| Adapter | LoRA, rank 16, on the attention, MLP and DeltaNet projections (33.8M trainable parameters) |
| Head | Kev's pointer head: it scores each option's closing token against the question's final token |
| Calibration | One temperature, T = 1.662, fitted on the calib split, stored in head.pt and applied at load time |
| Languages | Arabic values (MSA and dialects); English keys and option names |
| Licence | CC BY-NC-SA 4.0, for non-commercial research, because the training data is share-alike and non-commercial |
How to use
Serve it with the Kev repository at commit 1f04cfa28e6a708485b28a9585010b9e436b776e:
git clone https://github.com/jaredpalmer/kev.git && cd kev && git checkout 1f04cfa28e6a708485b28a9585010b9e436b776e && uv sync --extra serve
uv run --extra serve python -m kev.serve --run TigreGotico/jevhakim-4b --port 8008
from typesafe_sdk import Choice, Noul, TypeSafeClient
client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8008", model="kev-latest")
response = client.system_one(
state={"message": "السلام عليكم، انخصم مني المبلغ مرتين على الطلب 1182، ممكن ترجعوا واحد؟"},
questions={
"team": Choice(instructions="أي فريق يجب أن يتولى هذا الطلب؟",
criteria={"billing": "الفواتير والمبالغ المستردة", "shipping": "الشحن والتوصيل", "returns": "الإرجاع والاستبدال"}),
"urgent": Noul(instructions="هل يحتاج هذا الطلب إلى رد اليوم؟"),
},
)
print(response.choices["team"].choice, response.nouls["urgent"].noul)
Write the keys in English and the values in Arabic, as in the training data: the state's field names, the question ids and the option names are code identifiers. KEV_TEMPERATURE=1.0 returns the probabilities before calibration.
Training
The training split holds 442,627 Arabic requests from 58 sources, each capped so that no large source swamps the mix. They are mixed with 12,576 English records of Kev's decision-v7 suite as replay. Records with a state over 1,024 tokens were dropped, which left 453,426 of 455,203 records. Questions by type: 280,944 choice, 185,944 noul, 67,977 score.
The sources cover dialect identification, readability, sentiment, emotion, sarcasm, offensive language and hate speech, cyberbullying, propaganda, stance, check-worthiness and factuality, news topic, intent and routing, entailment and similarity, question similarity, answerability, faithfulness of summaries and translations, culture and religion questions, and safety of prompts and responses. Every label comes from the source dataset, or is made from it by construction; no teacher model labelled any of them.
| Epochs | 2 |
| Learning rate | 2e-05 |
| Effective batch | 8 (4 × 2 accumulation) |
| Precision | bf16 weights and autocast, gradient checkpointing |
| Kev options | --p_none 0 --p_none_distract 0 --p_distract 0, --length_sort 1 |
| Optimizer steps | 113,358 |
| Wall time | 35.0 hours on one RTX 5090 |
Training data
The full data is published as TigreGotico/jevhakim-decisions: the train, dev, calib and test splits this model was trained, calibrated and measured on, and a manifest with the revision, licence and drop counts of each source. The table gives each source with its requests per split. The 22 sources with no train, dev or calib requests are for evaluation only.
The text and labels belong to the authors of the source datasets. The dataset card's Attribution section names the publisher and the original corpus of each source, with the BibTeX to cite.
| Source | Dataset | Variety | Train | Dev | Calib | Test | Licence |
|---|---|---|---|---|---|---|---|
| aoc_aldi | arbml/AOC_ALDi |
mixed | 9,996 | 1,000 | 1,000 | 1,000 | not stated (AOC upstream: research) |
| arabic_nli | Omartificial-Intelligence-Space/Arabic-NLi-Pair-Class |
mt-ar | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| arafacts | QCRI/LlamaLens-Arabic |
mixed | 4,098 | 622 | 252 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| arcyc_cyberbullying | QCRI/LlamaLens-Arabic |
mixed | 2,082 | 443 | 186 | 887 | CC-BY-NC-SA-4.0 (Hub card) |
| arcyc_offensive | QCRI/LlamaLens-Arabic |
mixed | 2,082 | 446 | 173 | 891 | CC-BY-NC-SA-4.0 (Hub card) |
| arpro_propaganda | QCRI/LlamaLens-Arabic |
msa | 5,517 | 672 | 343 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| arsarcasm | QCRI/LlamaLens-Arabic |
mixed | 8,207 | 1,000 | 542 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| arsas | QCRI/LlamaLens-Arabic |
mixed | 10,000 | 1,000 | 817 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| asnd | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| barec | CAMeL-Lab/BAREC-Corpus-v1.0 |
msa | 9,981 | 1,000 | 1,000 | 0 | CC-BY-SA-4.0 (Hub card) |
| brad | arbml/BRAD |
mixed | 9,994 | 1,000 | 1,000 | 1,000 | not stated (BRAD upstream: research) |
| commonsense_validation | arbml/Commonsense_Validation |
mt-ar | 9,386 | 650 | 604 | 348 | not stated (SemEval-2020 task 4 upstream: CC-BY-SA-4.0) |
| covid_factuality | QCRI/LlamaLens-Arabic |
mixed | 1,704 | 315 | 230 | 917 | CC-BY-NC-SA-4.0 (Hub card) |
| ct22_attention | QCRI/LlamaLens-Arabic |
mixed | 1,684 | 1,000 | 158 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| ct22_claim | QCRI/LlamaLens-Arabic |
mixed | 1,704 | 315 | 230 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| ct22_harmful | QCRI/LlamaLens-Arabic |
mixed | 1,681 | 1,000 | 169 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| ct24_checkworthy | QCRI/LlamaLens-Arabic |
mixed | 5,025 | 1,000 | 470 | 610 | CC-BY-NC-SA-4.0 (Hub card) |
| dpo_norobots | 2A2I/NoRobots-AceGPT.13B.Chat-DPO |
mt-ar | 7,271 | 932 | 417 | 450 | Apache-2.0 (Hub card) |
| egy_translation | oddadmix/egyptian-arabic-english-translation-50k |
egy | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| emotional_tone | QCRI/LlamaLens-Arabic |
egy | 6,609 | 995 | 412 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| guardrail | oddadmix/arabic-guardrail |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| habibi | arbml/Habibi |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated (lyrics; research use only) |
| hard | Elnagara/hard |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated (card: unknown) |
| hate_superset | manueltonneau/arabic-hate-speech-superset |
mixed | 10,000 | 1,000 | 860 | 964 | not stated on the card; gated: no use that harms people; upstream licences vary |
| hatetweets_sentiment | QCRI/LlamaLens-Arabic |
mixed | 9,999 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| jev_darija | mballouch/jev-darija-telecom |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-4.0 (Hub card) |
| labr | mohamedadaly/labr |
mixed | 9,246 | 1,000 | 601 | 1,000 | not stated (card: unknown) |
| llamalens_stance | QCRI/LlamaLens-Arabic |
msa | 2,479 | 755 | 173 | 379 | CC-BY-NC-SA-4.0 (Hub card) |
| madar | macabdul9/NADI-Sample02 |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated; MADAR upstream: non-commercial research licence |
| massive_ar | SetFit/amazon_massive_intent_ar-SA |
msa | 10,000 | 1,000 | 629 | 1,000 | CC-BY-4.0 (upstream AmazonScience/massive) |
| mqa_faq | clips/mqa |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC0-1.0 (Hub card) |
| nadi | ashabrawy/NADI-Balanced-Country-DID |
mixed | 7,962 | 0 | 504 | 1,000 | not stated; NADI shared-task data, research use for registered teams |
| nawah | oddadmix/nawah-dialect-data |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| nemotron_safety | nvidia/Nemotron-Safety-Guard-Dataset-v3 |
mt-ar | 9,996 | 1,000 | 1,000 | 1,000 | CC-BY-4.0 (Hub card) |
| news_credibility | QCRI/LlamaLens-Arabic |
msa | 8,131 | 1,000 | 540 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| news_headline | QCRI/LlamaLens-Arabic |
msa | 890 | 100 | 49 | 153 | CC-BY-NC-SA-4.0 (Hub card) |
| nsurl_q2q | arbml/nsurl_2019_task8_train |
msa | 9,471 | 1,000 | 609 | 596 | not stated (NSURL-2019 task 8 upstream: research) |
| osact4_hate | QCRI/LlamaLens-Arabic |
mixed | 3,819 | 1,000 | 311 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| osact4_offensive | QCRI/LlamaLens-Arabic |
mixed | 3,846 | 1,000 | 303 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| palmx_culture | UBC-NLP/palmx_2025_subtask1_culture |
msa | 1,870 | 498 | 129 | 1,000 | PalmX Data License: non-commercial research use (Hub card) |
| palmx_islamic | UBC-NLP/palmx_2025_subtask2_islamic |
msa | 553 | 286 | 38 | 983 | PalmX Data License: non-commercial research use (Hub card) |
| polyguardmix | ToxicityPrompts/PolyGuardMix |
mt-ar | 9,998 | 1,000 | 1,000 | 1,000 | CC-BY-4.0 (Hub card) |
| prompt_routing | oddadmix/arabic-prompt-routing |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| qadi | Abdelrahman-Rezk/Arabic_Dialect_Identification |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated on the card; upstream QADI Apache-2.0 |
| rag_chat | oddadmix/arabic-rag-chat-30K |
synthetic | 10,000 | 1,000 | 1,000 | 850 | Apache-2.0 (Hub card) |
| rag_support | oddadmix/arabic-rag-support-25K |
synthetic | 10,000 | 1,000 | 1,000 | 889 | Apache-2.0 (Hub card) |
| reviews100k | QCRI/LlamaLens-Arabic |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| rule_checking | oddadmix/arabic-rule-checking |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| sanad_akhbarona | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| sanad_alarabiya | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| sanad_alkhaleej | QCRI/LlamaLens-Arabic |
msa | 9,896 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| shami | arbml/Shami |
lev | 10,000 | 1,000 | 1,000 | 1,000 | not stated (Shami corpus upstream: research) |
| stsb_ar | Omartificial-Intelligence-Space/Arabic-stsb |
mt-ar | 5,354 | 1,000 | 349 | 1,000 | Apache-2.0 (Hub card) |
| thatiar_subjective | QCRI/LlamaLens-Arabic |
msa | 2,227 | 467 | 146 | 748 | CC-BY-NC-SA-4.0 (Hub card) |
| tydiqa_answerable | copenlu/answerable_tydiqa |
msa | 10,000 | 1,000 | 1,000 | 605 | Apache-2.0 (Hub card) |
| ultimate_news | QCRI/LlamaLens-Arabic |
msa | 9,869 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| xlsum_faithful | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card; XL-Sum upstream CC-BY-NC-SA-4.0) |
| xnli_ar | facebook/xnli |
mt-ar | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-4.0 (XNLI upstream README) |
| abb | silma-ai/arabic-broad-benchmark |
mixed | 0 | 0 | 0 | 125 | Apache-2.0 (Hub card) |
| alghafa_native | OALL/AlGhafa-Arabic-LLM-Benchmark-Native |
mixed | 0 | 0 | 0 | 5,000 | not stated |
| arabculture | MBZUAI/ArabCulture |
mixed | 0 | 0 | 0 | 3,438 | CC-BY-NC-SA-4.0 (Hub card) |
| arabic_exams | inceptlabs/Arabic_EXAMS-Redux |
msa | 0 | 0 | 0 | 530 | CC-BY-SA-4.0 (Hub card) |
| arabic_mmlu | MBZUAI/ArabicMMLU |
msa | 0 | 0 | 0 | 5,000 | CC-BY-NC-4.0 (Hub card) |
| arapro | humain-ai/AraPro |
msa | 0 | 0 | 0 | 4,894 | not stated |
| aratrust | asas-ai/AraTrust |
msa | 0 | 0 | 0 | 519 | MIT (Hub card) |
| aratruthfulqa | humain-ai/AraTruthfulQA |
msa | 0 | 0 | 0 | 536 | not stated |
| barec_test | CAMeL-Lab/BAREC-Corpus-v1.0 |
msa | 0 | 0 | 0 | 5,000 | CC-BY-SA-4.0 (Hub card) |
| belebele_ar | facebook/belebele |
mixed | 0 | 0 | 0 | 5,000 | CC-BY-SA-4.0 (Hub card) |
| etec | humain-ai/Etec |
msa | 0 | 0 | 0 | 1,859 | not stated |
| global_piqa_ar | mrlbenchmarks/global-piqa-parallel |
mixed | 0 | 0 | 0 | 1,322 | CC-BY-SA-4.0 (Hub card) |
| guardrail_unseen | oddadmix/arabic-guardrail |
synthetic | 0 | 0 | 0 | 5,000 | Apache-2.0 (Hub card) |
| hatecheck_ar | Paul/hatecheck-arabic |
mixed | 0 | 0 | 0 | 3,530 | CC-BY-4.0 (Hub card) |
| ien_mcq | humain-ai/IEN_MCQ |
msa | 0 | 0 | 0 | 5,000 | not stated |
| ien_tf | humain-ai/IEN_TF |
msa | 0 | 0 | 0 | 5,000 | not stated |
| include_ar | CohereLabs/include-base-44 |
msa | 0 | 0 | 0 | 543 | Apache-2.0 (Hub card) |
| jev_ar_bench | atmaneayoub/jev-ar-bench |
synthetic | 0 | 0 | 0 | 4,974 | CC-BY-4.0 (Hub card) |
| madinahqa | MBZUAI/MadinahQA |
msa | 0 | 0 | 0 | 943 | CC-BY-NC-4.0 (Hub card) |
| polyguard_ar | ToxicityPrompts/PolyGuardPrompts |
mt-ar | 0 | 0 | 0 | 1,725 | CC-BY-4.0 (Hub card) |
| prompt_routing_unseen | oddadmix/arabic-prompt-routing |
synthetic | 0 | 0 | 0 | 5,000 | Apache-2.0 (Hub card) |
| system_one_ar | najdresearch/system-one |
mixed | 0 | 0 | 0 | 2,695 | per pack: CC-BY-4.0, CC-BY-SA-4.0, Apache-2.0, MIT (Hub card) |
| Total | 442,627 | 50,496 | 40,244 | 119,903 |
Evaluation
The test split holds 119,903 requests with 141,217 questions. It includes the test rows of the training sources and every row of the 22 evaluation-only sources, which the model never saw: abb, alghafa_native, arabculture, arabic_exams, arabic_mmlu, arapro, aratrust, aratruthfulqa, barec_test, belebele_ar, etec, global_piqa_ar, guardrail_unseen, hatecheck_ar, ien_mcq, ien_tf, include_ar, jev_ar_bench, madinahqa, polyguard_ar, prompt_routing_unseen, system_one_ar.
Every request was scored. No request was rejected or truncated.
| Jevhakim-0.8B | Jevhakim-4B | |
|---|---|---|
| Accuracy | 0.661 | 0.765 |
| ECE | 0.063 | 0.061 |
| Brier | 0.438 | 0.328 |
| Coverage at 5 % error | 0.339 | 0.507 |
| Coverage at confidence ≥ 0.9 | 0.366 | 0.551 |
| Accuracy at confidence ≥ 0.9 | 0.941 | 0.939 |
Limits
- The temperature is fitted on training sources only. On a kind of question it never saw, expect worse calibration than the figures above. Before you set a threshold, fit a temperature on labelled data from your own workload (
python -m kev.calibrate). - Some sources are for research use only. NADI, MADAR, BRAD, AOC, the Habibi lyrics and the PalmX sets carry research or non-commercial terms, and several sources state no licence. Check each source's terms before any use beyond research.
- Part of the data is machine-translated or generated. Of the 442,627 Arabic training requests, 62,005 come from sources translated from English and 60,000 from sources generated by language models.
- Dialect coverage is uneven and not measured per dialect. By source, 130,884 training requests are MSA, 163,129 come from sources that mix varieties, 16,609 are Egyptian and 10,000 are Levantine. No accuracy figure per dialect exists.
- This is not a model for decisions with legal, medical, financial or employment consequences for people without human review. It does not generate text.
- Downloads last month
- 19
Model tree for TigreGotico/jevhakim-4b
Base model
Qwen/Qwen3.5-4B-Base