Jevhakim
Collection
Decision models for Arabic, MSA and dialects: Kev fine-tuned on typed decisions from 80 public datasets. • 4 items • Updated
Jevhakim-0.8B
Jevhakim-0.8B is a decision model for Arabic: Modern Standard Arabic and the dialects. It reads a state (a text or a JSON object) and a set of typed questions about it. It returns a probability over the options of each question, in one forward pass and without generated text. The question types are choice (pick one option), score (an ordered level) and noul (yes or no).
It is Kev-0.8B trained further on Arabic decisions from 58 sources, all built from public datasets. A further 22 sources are held out for evaluation only. The request format and the serving code are those of Kev, so a Kev or System One client works with it unchanged.
Model details
| Backbone | Qwen/Qwen3.5-0.8B-Base (revision dc7cdfe2), frozen |
| Initialised from | jaredpalmer/kev-0.8b |
| Adapter | LoRA, rank 16, on the attention, MLP and DeltaNet projections (11.3M trainable parameters) |
| Head | Kev's pointer head: it scores each option's closing token against the question's final token |
| Calibration | One temperature, T = 1.203, fitted on the calib split, stored in head.pt and applied at load time |
| Languages | Arabic values (MSA and dialects); English keys and option names |
| Licence | CC BY-NC-SA 4.0, for non-commercial research, because the training data is share-alike and non-commercial |
How to use
Serve it with the Kev repository at commit 1f04cfa28e6a708485b28a9585010b9e436b776e:
git clone https://github.com/jaredpalmer/kev.git && cd kev && git checkout 1f04cfa28e6a708485b28a9585010b9e436b776e && uv sync --extra serve
uv run --extra serve python -m kev.serve --run TigreGotico/jevhakim-0.8b --port 8008
from typesafe_sdk import Choice, Noul, TypeSafeClient
client = TypeSafeClient(api_key="local", base_url="http://127.0.0.1:8008", model="kev-latest")
response = client.system_one(
state={"message": "السلام عليكم، انخصم مني المبلغ مرتين على الطلب 1182، ممكن ترجعوا واحد؟"},
questions={
"team": Choice(instructions="أي فريق يجب أن يتولى هذا الطلب؟",
criteria={"billing": "الفواتير والمبالغ المستردة", "shipping": "الشحن والتوصيل", "returns": "الإرجاع والاستبدال"}),
"urgent": Noul(instructions="هل يحتاج هذا الطلب إلى رد اليوم؟"),
},
)
print(response.choices["team"].choice, response.nouls["urgent"].noul)
Write the keys in English and the values in Arabic, as in the training data: the state's field names, the question ids and the option names are code identifiers. KEV_TEMPERATURE=1.0 returns the probabilities before calibration.
Training
The training split holds 442,627 Arabic requests from 58 sources, each capped so that no large source swamps the mix. They are mixed with 12,576 English records of Kev's decision-v7 suite as replay. Records with a state over 2,048 tokens were dropped, which left 455,184 of 455,203 records. Questions by type: 281,845 choice, 187,592 noul, 68,018 score.
The sources cover dialect identification, readability, sentiment, emotion, sarcasm, offensive language and hate speech, cyberbullying, propaganda, stance, check-worthiness and factuality, news topic, intent and routing, entailment and similarity, question similarity, answerability, faithfulness of summaries and translations, culture and religion questions, and safety of prompts and responses. Every label comes from the source dataset, or is made from it by construction; no teacher model labelled any of them.
| Epochs | 2 |
| Learning rate | 2e-05 |
| Effective batch | 64 (16 × 4 accumulation) |
| Precision | bf16 autocast, gradient checkpointing |
| Kev options | --p_none 0 --p_none_distract 0 --p_distract 0, --length_sort 1, --shared_prefix 1 |
| Optimizer steps | 14,226 |
| Wall time | 14.2 hours on one RTX 5090 |
Training data
The full data is published as TigreGotico/jevhakim-decisions: the train, dev, calib and test splits this model was trained, calibrated and measured on, and a manifest with the revision, licence and drop counts of each source. The table gives each source with its requests per split. The 22 sources with no train, dev or calib requests are for evaluation only.
The text and labels belong to the authors of the source datasets. The dataset card's Attribution section names the publisher and the original corpus of each source, with the BibTeX to cite.
| Source | Dataset | Variety | Train | Dev | Calib | Test | Licence |
|---|---|---|---|---|---|---|---|
| aoc_aldi | arbml/AOC_ALDi |
mixed | 9,996 | 1,000 | 1,000 | 1,000 | not stated (AOC upstream: research) |
| arabic_nli | Omartificial-Intelligence-Space/Arabic-NLi-Pair-Class |
mt-ar | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| arafacts | QCRI/LlamaLens-Arabic |
mixed | 4,098 | 622 | 252 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| arcyc_cyberbullying | QCRI/LlamaLens-Arabic |
mixed | 2,082 | 443 | 186 | 887 | CC-BY-NC-SA-4.0 (Hub card) |
| arcyc_offensive | QCRI/LlamaLens-Arabic |
mixed | 2,082 | 446 | 173 | 891 | CC-BY-NC-SA-4.0 (Hub card) |
| arpro_propaganda | QCRI/LlamaLens-Arabic |
msa | 5,517 | 672 | 343 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| arsarcasm | QCRI/LlamaLens-Arabic |
mixed | 8,207 | 1,000 | 542 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| arsas | QCRI/LlamaLens-Arabic |
mixed | 10,000 | 1,000 | 817 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| asnd | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| barec | CAMeL-Lab/BAREC-Corpus-v1.0 |
msa | 9,981 | 1,000 | 1,000 | 0 | CC-BY-SA-4.0 (Hub card) |
| brad | arbml/BRAD |
mixed | 9,994 | 1,000 | 1,000 | 1,000 | not stated (BRAD upstream: research) |
| commonsense_validation | arbml/Commonsense_Validation |
mt-ar | 9,386 | 650 | 604 | 348 | not stated (SemEval-2020 task 4 upstream: CC-BY-SA-4.0) |
| covid_factuality | QCRI/LlamaLens-Arabic |
mixed | 1,704 | 315 | 230 | 917 | CC-BY-NC-SA-4.0 (Hub card) |
| ct22_attention | QCRI/LlamaLens-Arabic |
mixed | 1,684 | 1,000 | 158 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| ct22_claim | QCRI/LlamaLens-Arabic |
mixed | 1,704 | 315 | 230 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| ct22_harmful | QCRI/LlamaLens-Arabic |
mixed | 1,681 | 1,000 | 169 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| ct24_checkworthy | QCRI/LlamaLens-Arabic |
mixed | 5,025 | 1,000 | 470 | 610 | CC-BY-NC-SA-4.0 (Hub card) |
| dpo_norobots | 2A2I/NoRobots-AceGPT.13B.Chat-DPO |
mt-ar | 7,271 | 932 | 417 | 450 | Apache-2.0 (Hub card) |
| egy_translation | oddadmix/egyptian-arabic-english-translation-50k |
egy | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| emotional_tone | QCRI/LlamaLens-Arabic |
egy | 6,609 | 995 | 412 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| guardrail | oddadmix/arabic-guardrail |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| habibi | arbml/Habibi |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated (lyrics; research use only) |
| hard | Elnagara/hard |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated (card: unknown) |
| hate_superset | manueltonneau/arabic-hate-speech-superset |
mixed | 10,000 | 1,000 | 860 | 964 | not stated on the card; gated: no use that harms people; upstream licences vary |
| hatetweets_sentiment | QCRI/LlamaLens-Arabic |
mixed | 9,999 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| jev_darija | mballouch/jev-darija-telecom |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-4.0 (Hub card) |
| labr | mohamedadaly/labr |
mixed | 9,246 | 1,000 | 601 | 1,000 | not stated (card: unknown) |
| llamalens_stance | QCRI/LlamaLens-Arabic |
msa | 2,479 | 755 | 173 | 379 | CC-BY-NC-SA-4.0 (Hub card) |
| madar | macabdul9/NADI-Sample02 |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated; MADAR upstream: non-commercial research licence |
| massive_ar | SetFit/amazon_massive_intent_ar-SA |
msa | 10,000 | 1,000 | 629 | 1,000 | CC-BY-4.0 (upstream AmazonScience/massive) |
| mqa_faq | clips/mqa |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC0-1.0 (Hub card) |
| nadi | ashabrawy/NADI-Balanced-Country-DID |
mixed | 7,962 | 0 | 504 | 1,000 | not stated; NADI shared-task data, research use for registered teams |
| nawah | oddadmix/nawah-dialect-data |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| nemotron_safety | nvidia/Nemotron-Safety-Guard-Dataset-v3 |
mt-ar | 9,996 | 1,000 | 1,000 | 1,000 | CC-BY-4.0 (Hub card) |
| news_credibility | QCRI/LlamaLens-Arabic |
msa | 8,131 | 1,000 | 540 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| news_headline | QCRI/LlamaLens-Arabic |
msa | 890 | 100 | 49 | 153 | CC-BY-NC-SA-4.0 (Hub card) |
| nsurl_q2q | arbml/nsurl_2019_task8_train |
msa | 9,471 | 1,000 | 609 | 596 | not stated (NSURL-2019 task 8 upstream: research) |
| osact4_hate | QCRI/LlamaLens-Arabic |
mixed | 3,819 | 1,000 | 311 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| osact4_offensive | QCRI/LlamaLens-Arabic |
mixed | 3,846 | 1,000 | 303 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| palmx_culture | UBC-NLP/palmx_2025_subtask1_culture |
msa | 1,870 | 498 | 129 | 1,000 | PalmX Data License: non-commercial research use (Hub card) |
| palmx_islamic | UBC-NLP/palmx_2025_subtask2_islamic |
msa | 553 | 286 | 38 | 983 | PalmX Data License: non-commercial research use (Hub card) |
| polyguardmix | ToxicityPrompts/PolyGuardMix |
mt-ar | 9,998 | 1,000 | 1,000 | 1,000 | CC-BY-4.0 (Hub card) |
| prompt_routing | oddadmix/arabic-prompt-routing |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| qadi | Abdelrahman-Rezk/Arabic_Dialect_Identification |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | not stated on the card; upstream QADI Apache-2.0 |
| rag_chat | oddadmix/arabic-rag-chat-30K |
synthetic | 10,000 | 1,000 | 1,000 | 850 | Apache-2.0 (Hub card) |
| rag_support | oddadmix/arabic-rag-support-25K |
synthetic | 10,000 | 1,000 | 1,000 | 889 | Apache-2.0 (Hub card) |
| reviews100k | QCRI/LlamaLens-Arabic |
mixed | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| rule_checking | oddadmix/arabic-rule-checking |
synthetic | 10,000 | 1,000 | 1,000 | 1,000 | Apache-2.0 (Hub card) |
| sanad_akhbarona | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| sanad_alarabiya | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| sanad_alkhaleej | QCRI/LlamaLens-Arabic |
msa | 9,896 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| shami | arbml/Shami |
lev | 10,000 | 1,000 | 1,000 | 1,000 | not stated (Shami corpus upstream: research) |
| stsb_ar | Omartificial-Intelligence-Space/Arabic-stsb |
mt-ar | 5,354 | 1,000 | 349 | 1,000 | Apache-2.0 (Hub card) |
| thatiar_subjective | QCRI/LlamaLens-Arabic |
msa | 2,227 | 467 | 146 | 748 | CC-BY-NC-SA-4.0 (Hub card) |
| tydiqa_answerable | copenlu/answerable_tydiqa |
msa | 10,000 | 1,000 | 1,000 | 605 | Apache-2.0 (Hub card) |
| ultimate_news | QCRI/LlamaLens-Arabic |
msa | 9,869 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card) |
| xlsum_faithful | QCRI/LlamaLens-Arabic |
msa | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-SA-4.0 (Hub card; XL-Sum upstream CC-BY-NC-SA-4.0) |
| xnli_ar | facebook/xnli |
mt-ar | 10,000 | 1,000 | 1,000 | 1,000 | CC-BY-NC-4.0 (XNLI upstream README) |
| abb | silma-ai/arabic-broad-benchmark |
mixed | 0 | 0 | 0 | 125 | Apache-2.0 (Hub card) |
| alghafa_native | OALL/AlGhafa-Arabic-LLM-Benchmark-Native |
mixed | 0 | 0 | 0 | 5,000 | not stated |
| arabculture | MBZUAI/ArabCulture |
mixed | 0 | 0 | 0 | 3,438 | CC-BY-NC-SA-4.0 (Hub card) |
| arabic_exams | inceptlabs/Arabic_EXAMS-Redux |
msa | 0 | 0 | 0 | 530 | CC-BY-SA-4.0 (Hub card) |
| arabic_mmlu | MBZUAI/ArabicMMLU |
msa | 0 | 0 | 0 | 5,000 | CC-BY-NC-4.0 (Hub card) |
| arapro | humain-ai/AraPro |
msa | 0 | 0 | 0 | 4,894 | not stated |
| aratrust | asas-ai/AraTrust |
msa | 0 | 0 | 0 | 519 | MIT (Hub card) |
| aratruthfulqa | humain-ai/AraTruthfulQA |
msa | 0 | 0 | 0 | 536 | not stated |
| barec_test | CAMeL-Lab/BAREC-Corpus-v1.0 |
msa | 0 | 0 | 0 | 5,000 | CC-BY-SA-4.0 (Hub card) |
| belebele_ar | facebook/belebele |
mixed | 0 | 0 | 0 | 5,000 | CC-BY-SA-4.0 (Hub card) |
| etec | humain-ai/Etec |
msa | 0 | 0 | 0 | 1,859 | not stated |
| global_piqa_ar | mrlbenchmarks/global-piqa-parallel |
mixed | 0 | 0 | 0 | 1,322 | CC-BY-SA-4.0 (Hub card) |
| guardrail_unseen | oddadmix/arabic-guardrail |
synthetic | 0 | 0 | 0 | 5,000 | Apache-2.0 (Hub card) |
| hatecheck_ar | Paul/hatecheck-arabic |
mixed | 0 | 0 | 0 | 3,530 | CC-BY-4.0 (Hub card) |
| ien_mcq | humain-ai/IEN_MCQ |
msa | 0 | 0 | 0 | 5,000 | not stated |
| ien_tf | humain-ai/IEN_TF |
msa | 0 | 0 | 0 | 5,000 | not stated |
| include_ar | CohereLabs/include-base-44 |
msa | 0 | 0 | 0 | 543 | Apache-2.0 (Hub card) |
| jev_ar_bench | atmaneayoub/jev-ar-bench |
synthetic | 0 | 0 | 0 | 4,974 | CC-BY-4.0 (Hub card) |
| madinahqa | MBZUAI/MadinahQA |
msa | 0 | 0 | 0 | 943 | CC-BY-NC-4.0 (Hub card) |
| polyguard_ar | ToxicityPrompts/PolyGuardPrompts |
mt-ar | 0 | 0 | 0 | 1,725 | CC-BY-4.0 (Hub card) |
| prompt_routing_unseen | oddadmix/arabic-prompt-routing |
synthetic | 0 | 0 | 0 | 5,000 | Apache-2.0 (Hub card) |
| system_one_ar | najdresearch/system-one |
mixed | 0 | 0 | 0 | 2,695 | per pack: CC-BY-4.0, CC-BY-SA-4.0, Apache-2.0, MIT (Hub card) |
| Total | 442,627 | 50,496 | 40,244 | 119,903 |
Evaluation
The test split holds 119,903 requests with 141,217 questions. It includes the test rows of the training sources and every row of the 22 evaluation-only sources, which the model never saw: abb, alghafa_native, arabculture, arabic_exams, arabic_mmlu, arapro, aratrust, aratruthfulqa, barec_test, belebele_ar, etec, global_piqa_ar, guardrail_unseen, hatecheck_ar, ien_mcq, ien_tf, include_ar, jev_ar_bench, madinahqa, polyguard_ar, prompt_routing_unseen, system_one_ar.
Every request was scored for this model and for stock Kev-0.8B, each at its own temperature. No request was rejected or truncated.
| Kev-0.8B | Jevhakim-0.8B | |
|---|---|---|
| Accuracy | 0.503 | 0.661 |
| ECE | 0.038 | 0.063 |
| Brier | 0.599 | 0.438 |
| Coverage at 5 % error | 0.036 | 0.339 |
| Coverage at confidence ≥ 0.9 | 0.050 | 0.366 |
| Accuracy at confidence ≥ 0.9 | 0.926 | 0.940 |
Accuracy by kind of source, weighted by questions:
| Sources | Questions | Kev-0.8B | Jevhakim-0.8B |
|---|---|---|---|
| Training sources, test rows | 60,417 | 0.521 | 0.763 |
| Evaluation-only, all 22 | 80,800 | 0.490 | 0.584 |
| Evaluation-only, without guardrail_unseen and prompt_routing_unseen | 64,273 | 0.484 | 0.516 |
guardrail_unseen and prompt_routing_unseen hold unseen rows of two task kinds the model trained on, so the last row is the figure for new kinds of question.
On the calib split, which holds only training sources, accuracy is 0.791 and ECE 0.022. That ECE is in-sample, because the temperature was fitted on the same split.
Per source, on the test split:
| Source | Questions | Stock acc | Tuned acc | Stock ECE | Tuned ECE |
|---|---|---|---|---|---|
| abb | 125 | 0.448 | 0.456 | 0.167 | 0.178 |
| alghafa_native | 5,000 | 0.427 | 0.402 | 0.063 | 0.207 |
| aoc_aldi | 1,894 | 0.516 | 0.772 | 0.156 | 0.030 |
| arabculture | 3,438 | 0.488 | 0.519 | 0.006 | 0.095 |
| arabic_exams | 530 | 0.472 | 0.449 | 0.025 | 0.089 |
| arabic_mmlu | 5,000 | 0.430 | 0.403 | 0.079 | 0.137 |
| arabic_nli | 1,000 | 0.668 | 0.788 | 0.067 | 0.036 |
| arafacts | 1,000 | 0.097 | 0.763 | 0.448 | 0.048 |
| arapro | 4,894 | 0.447 | 0.477 | 0.045 | 0.075 |
| aratrust | 519 | 0.551 | 0.620 | 0.052 | 0.051 |
| aratruthfulqa | 536 | 0.420 | 0.384 | 0.053 | 0.197 |
| arcyc_cyberbullying | 887 | 0.620 | 0.799 | 0.097 | 0.028 |
| arcyc_offensive | 891 | 0.663 | 0.836 | 0.025 | 0.019 |
| arpro_propaganda | 1,000 | 0.452 | 0.761 | 0.238 | 0.037 |
| arsarcasm | 1,000 | 0.642 | 0.756 | 0.042 | 0.033 |
| arsas | 1,000 | 0.542 | 0.791 | 0.042 | 0.067 |
| asnd | 1,000 | 0.640 | 0.886 | 0.041 | 0.009 |
| barec_test | 5,000 | 0.123 | 0.473 | 0.170 | 0.023 |
| belebele_ar | 5,000 | 0.417 | 0.410 | 0.062 | 0.198 |
| brad | 1,000 | 0.380 | 0.504 | 0.093 | 0.040 |
| commonsense_validation | 348 | 0.652 | 0.833 | 0.051 | 0.031 |
| covid_factuality | 917 | 0.250 | 0.844 | 0.474 | 0.045 |
| ct22_attention | 1,000 | 0.126 | 0.468 | 0.190 | 0.070 |
| ct22_claim | 1,000 | 0.535 | 0.748 | 0.063 | 0.052 |
| ct22_harmful | 1,000 | 0.777 | 0.818 | 0.098 | 0.024 |
| ct24_checkworthy | 610 | 0.407 | 0.515 | 0.237 | 0.203 |
| dpo_norobots | 450 | 0.562 | 0.989 | 0.067 | 0.006 |
| egy_translation | 1,000 | 0.614 | 0.980 | 0.105 | 0.008 |
| emotional_tone | 1,000 | 0.357 | 0.685 | 0.079 | 0.028 |
| etec | 1,859 | 0.405 | 0.404 | 0.042 | 0.086 |
| global_piqa_ar | 1,322 | 0.282 | 0.274 | 0.103 | 0.173 |
| guardrail | 2,315 | 0.392 | 0.894 | 0.122 | 0.010 |
| guardrail_unseen | 11,527 | 0.361 | 0.811 | 0.134 | 0.062 |
| habibi | 1,000 | 0.149 | 0.567 | 0.150 | 0.039 |
| hard | 1,000 | 0.466 | 0.797 | 0.059 | 0.016 |
| hate_superset | 964 | 0.576 | 0.759 | 0.070 | 0.019 |
| hatecheck_ar | 6,767 | 0.587 | 0.636 | 0.086 | 0.113 |
| hatetweets_sentiment | 1,000 | 0.246 | 0.487 | 0.295 | 0.022 |
| ien_mcq | 5,000 | 0.556 | 0.556 | 0.015 | 0.053 |
| ien_tf | 5,000 | 0.679 | 0.523 | 0.064 | 0.250 |
| include_ar | 543 | 0.379 | 0.350 | 0.080 | 0.176 |
| jev_ar_bench | 4,974 | 0.530 | 0.532 | 0.144 | 0.147 |
| jev_darija | 4,472 | 0.438 | 0.617 | 0.050 | 0.086 |
| labr | 1,000 | 0.273 | 0.532 | 0.197 | 0.063 |
| llamalens_stance | 379 | 0.206 | 0.913 | 0.353 | 0.034 |
| madar | 1,000 | 0.348 | 0.768 | 0.079 | 0.026 |
| madinahqa | 943 | 0.409 | 0.393 | 0.046 | 0.089 |
| massive_ar | 1,974 | 0.573 | 0.856 | 0.044 | 0.025 |
| mqa_faq | 1,000 | 0.566 | 0.950 | 0.164 | 0.012 |
| nadi | 1,000 | 0.074 | 0.154 | 0.141 | 0.027 |
| nawah | 1,000 | 0.386 | 0.824 | 0.102 | 0.030 |
| nemotron_safety | 1,494 | 0.627 | 0.773 | 0.066 | 0.070 |
| news_credibility | 1,000 | 0.769 | 0.936 | 0.133 | 0.019 |
| news_headline | 153 | 0.301 | 0.490 | 0.167 | 0.064 |
| nsurl_q2q | 596 | 0.711 | 0.950 | 0.046 | 0.017 |
| osact4_hate | 1,000 | 0.846 | 0.957 | 0.176 | 0.015 |
| osact4_offensive | 1,000 | 0.735 | 0.888 | 0.079 | 0.025 |
| palmx_culture | 1,000 | 0.367 | 0.475 | 0.063 | 0.034 |
| palmx_islamic | 983 | 0.450 | 0.584 | 0.029 | 0.044 |
| polyguard_ar | 5,128 | 0.718 | 0.867 | 0.033 | 0.033 |
| polyguardmix | 1,998 | 0.705 | 0.887 | 0.025 | 0.007 |
| prompt_routing | 1,000 | 0.842 | 0.936 | 0.164 | 0.013 |
| prompt_routing_unseen | 5,000 | 0.863 | 0.931 | 0.164 | 0.016 |
| qadi | 1,000 | 0.129 | 0.272 | 0.046 | 0.028 |
| rag_chat | 850 | 0.718 | 0.976 | 0.065 | 0.008 |
| rag_support | 889 | 0.777 | 0.984 | 0.173 | 0.003 |
| reviews100k | 1,000 | 0.547 | 0.689 | 0.183 | 0.018 |
| rule_checking | 1,000 | 0.489 | 0.974 | 0.112 | 0.007 |
| sanad_akhbarona | 1,000 | 0.875 | 0.946 | 0.022 | 0.007 |
| sanad_alarabiya | 1,000 | 0.902 | 0.976 | 0.040 | 0.013 |
| sanad_alkhaleej | 1,000 | 0.897 | 0.960 | 0.034 | 0.011 |
| shami | 1,000 | 0.406 | 0.723 | 0.034 | 0.040 |
| stsb_ar | 1,000 | 0.357 | 0.554 | 0.032 | 0.032 |
| system_one_ar | 2,695 | 0.466 | 0.511 | 0.077 | 0.130 |
| thatiar_subjective | 748 | 0.533 | 0.556 | 0.103 | 0.027 |
| tydiqa_answerable | 605 | 0.608 | 0.901 | 0.098 | 0.038 |
| ultimate_news | 1,000 | 0.759 | 0.919 | 0.040 | 0.019 |
| xlsum_faithful | 1,000 | 0.598 | 0.996 | 0.107 | 0.004 |
| xnli_ar | 1,000 | 0.616 | 0.712 | 0.107 | 0.022 |
Limits
- The gain is on the tasks it trained on. On new kinds of question, accuracy went from 0.484 to 0.516. On the knowledge and reading sets (arabic_mmlu, belebele_ar, alghafa_native, include_ar, madinahqa, aratruthfulqa, global_piqa_ar) it fell by up to 0.04, and on the true-or-false set ien_tf it fell from 0.679 to 0.523.
- The temperature is fitted on training sources only. On those knowledge sets the ECE rises to between 0.09 and 0.21, against 0.05 to 0.10 for stock Kev, so the model is overconfident on new tasks. Before you set a threshold, fit a temperature on labelled data from your own workload (
python -m kev.calibrate). - Some sources are for research use only. NADI, MADAR, BRAD, AOC, the Habibi lyrics and the PalmX sets carry research or non-commercial terms, and several sources state no licence. Check each source's terms before any use beyond research.
- Part of the data is machine-translated or generated. Of the 442,627 Arabic training requests, 62,005 come from sources translated from English and 60,000 from sources generated by language models.
- Dialect coverage is uneven and not measured per dialect. By source, 130,884 training requests are MSA, 163,129 come from sources that mix varieties, 16,609 are Egyptian and 10,000 are Levantine. No accuracy figure per dialect exists.
- This is not a model for decisions with legal, medical, financial or employment consequences for people without human review. It does not generate text.
- Downloads last month
- 1
Model tree for TigreGotico/jevhakim-0.8b
Base model
Qwen/Qwen3.5-0.8B-Base