3ssiri/Sev-Arabic-Preview-v0.1

Sev Arabic Preview v0.1

Research Preview. Not the final Sev model, not production-ready, and not a final architecture choice.

• Version: 0.1.0 · Date: 2026-09-26
• Author: Ali Asiri · © 2026 Ali Asiri
• Contact: @3li3 on X · assiri@gmail.com
• Licenses: model weights CC BY-SA 4.0; inference code Apache-2.0 (see "Licensing and attribution").
• Code and release notes: https://github.com/3ssiri/sev-arabic-preview (release v0.1.0)
• DOI (all versions): 10.5281/zenodo.22979051 · v0.1.0: 10.5281/zenodo.22979052

What it is

Sev is an Arabic-first typed decision engine. A request is a state (text or JSON) plus up to 8 typed questions; one encoder pass returns, for every question, a bounded answer with calibrated probabilities. There is no text generation in the decision path.

Supported question types in v0.1:

type	answer	notes
choice	one option key + a probability for every option	options are given per request
noul	P(true)	label verification, asked together with its choice question — see limitations

score exists in the broader Sev design but is not supported in v0.1: the runtime rejects it, and no quality claim is made.

Model

• Base: jhu-clsp/mmBERT-base, revision c5955035435e2bf121cde7f3c8863ef52ff35d82 (MIT). The base weights are downloaded from Hugging Face at that exact revision; they are not redistributed here.
• Sev parts: LoRA adapter (r=8, on Wqkv, Wi, Wo) + a typed marker head (Linear(768, 1) scored at each option marker).
• Architecture code: open-jev-typed-decision-engine at commit 5977431afb6a9272e034f63c3ec7f0ee5395df59 (Apache-2.0, © 2026 Intikhab Azam), vendored unchanged.
• Checkpoint: Track E, seed 42 (frozen before any external benchmark result existed).
• Calibration: per-type temperature scaling, choice T = 1.364155606583895, noul T = 1.5122454372719816, fitted on development/calibration data only.
• Input: 384 tokens. Questions are never cut; a longer state is truncated and the response reports meta.state_truncated = true.

Exact file hashes (SHA-256)

file	SHA-256
adapter/adapter_model.safetensors	361c213df6bb152a8ee39b61fdf9e5adf283fc6557b77d61b1777a0f1e9b9548
head.pt	e697101ce9bbfe16122238713adcf1024458b4b5d89a685dcb8eee6928652791
adapter/adapter_config.json	e82bd8e10ac417a2b7a352ec3b5ce0897a50c02b845eb6339b3d4260375fdc18
training_config.json (verified, not distributed — contains local paths)	6da12b54283ce7e1120c022d0969ea8cf32d155762a5ef6fa80a5f0e214eb3cf

The full package manifest (MANIFEST.sha256) and the package archive hash are recorded in the release notes and in sev_preview_config.json; the runtime refuses to load if a weight file's hash differs.

Training data and attribution

Sev fine-tuning for this checkpoint used Arabic text only: 29,753 training rows; 7,803 development rows.

• MASSIVE (ar-SA) — FitzGerald et al., "MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages" (ACL 2023), Amazon — CC BY 4.0. https://huggingface.co/datasets/AmazonScience/massive
• ArBanking77 — SinaLab — CC BY-SA 4.0. https://github.com/SinaLab/ArBanking77. Its license asks users to cite: Jarrar et al. (2023), "ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic" (ArabicNLP 2023), and Malaysha et al. (2024), "AraFinNLP 2024: The First Arabic Financial NLP Shared Task" (ArabicNLP 2024).

Both sources were converted into Sev's typed-question format. The noul questions were derived deterministically from the source labels ("is this label the correct one for this request?"). No raw data is included in this release.

Development results (not an external benchmark)

Sev public-dev split, 7,803 rows, each with one choice and one noul question:

	seed 42 (this checkpoint)	seeds 42/17/73
overall accuracy	94.55%	mean 94.53%, stdev 0.04 pp
choice accuracy	92.82%	—
noul accuracy (verification form)	96.28%	—
held-out ECE after temperature scaling	—	≈ 0.0053 (mean)
choice accuracy, same question asked without its paired noul (6,228 held-out rows, added 2026-09-27)	90.94% (paired on the same rows: 92.95%)	91.31% / 90.80% (seeds 17/73)
choice / standalone noul on a broader, harder development set (SevCorpus-v0 rev1 dev: 410 choice + 355 noul rows, one question per request, added 2026-10-01)	57.3% / 56.9% (ECE 0.135 / 0.130)	—

These come from Sev's own development split, drawn from the same two sources as training. 94.53% is not an external benchmark and not a release accuracy. It cannot be compared with published results of Jeb, Laya, Lev, Jev or any other system. An external, preregistered benchmark run is in progress and will be reported separately.

Runtime and hardware

Measured on one workstation (AMD Ryzen 5 7600, 32 GB RAM, RTX 4060 Ti 8 GB); not a guarantee.

• Python 3.11+ with torch 2.11.0, transformers 5.17.0, peft 0.21.0, huggingface_hub.
• Packaged smoke test passed on Windows 10, CPU (Python 3.11.9): p50 ≈ 106 ms per request, peak process RAM ≈ 3.0 GB; and on Linux (Ubuntu, WSL2), CPU (Python 3.12.14): p50 ≈ 277 ms, peak RAM ≈ 3.1 GB. Both CPU runs shared the machine with another GPU job, so latency is indicative only.
• NVIDIA RTX 4060 Ti 8 GB (Windows 10, CUDA): packaged smoke passed with p50 ≈ 21 ms per request, peak ≈ 1.2 GB VRAM, ≈ 3.0 GB process RAM. On a 151,476-request external workload with inputs up to 8,192 tokens the same checkpoint peaked at ≈ 2.2 GB allocated VRAM.
• macOS / Apple Silicon and Docker are not tested; no support claim is made for them.

Intended use

Structured Arabic routing, intent classification, and label verification experiments where a caller needs bounded decisions and probabilities.

Not for autonomous high-risk decisions (medical, legal, financial, employment, credit, safety, or decisions about people's rights) without human review.

Limitations

• noul is narrow. It was trained only as a verification question ("is this label the correct one for this request?") with explicit Arabic false/true criteria, always in the same request as the choice question it verifies (see the example). Use it only in that form. In the packaged smoke test the same true verification scored P(true) ≈ 0.99 when paired with its choice question but ≈ 0.25 when asked alone, and a clearly true free-form banking question without criteria scored ≈ 0.10. Standalone and free-form yes/no questions are not supported in v0.1.
• choice accuracy is ~2 points lower without the paired noul question. The derived noul question names a candidate label, and in half of the development rows that label is the correct answer. The encoder reads both questions in one sequence, so the 92.82% choice accuracy above includes a ~2-point contribution from that text (measured 2026-09-27 on 6,228 held-out dev rows: 92.95% paired vs 90.94% alone; 95% family-bootstrap CI of the difference −2.7 to −1.3 points; seeds 17/73: −1.7 and −2.1). Callers that send a choice question alone should expect ≈ 91%. Two further data-construction artefacts in the banking source (fixed 8-option candidate sets; the gold label always being the first option key, which this model does not see because keys are sorted) are documented in the development repository and are not corrected in v0.1.
• Much lower accuracy outside its two training sources (measured 2026-10-01). This checkpoint was run unchanged, with its published temperatures, on the development split of Sev's newer corpus (SevCorpus-v0 rev1), which adds Saudi-dialect call-centre dialogues and other Arabic sources and asks each question on its own. It scored 57.3% on 410 choice questions and 56.9% on 355 standalone noul questions (close to chance for yes/no), with confidence well above accuracy (ECE ≈ 0.13). The 94.53% above holds only for its own development split. Models trained on the newer corpus reach 81–87% on the same rows in development runs; they are not released, and those figures are internal development evidence, not a release claim.
• Domains. Training covers virtual-assistant intents (MASSIVE) and banking customer requests (ArBanking77). Other domains are untested.
• Saudi/Gulf coverage. No dedicated Saudi/Gulf dialect evaluation set exists yet; dialect behaviour is not measured separately. Short Gulf-dialect requests worked in the smoke test, which is not an evaluation.
• OOD, code-switching, ambiguity. Not evaluated. choice always picks one of the given options; there is no built-in abstention, so include an explicit "other" option and treat low probabilities as uncertain.
• Context. 384 input tokens; longer states are truncated.
• Balance. The corpus is not the planned balanced SevCorpus-v0.

Licensing and attribution

Owner decision 2026-09-26 — three separate layers:

1. Model weights (adapter/, head.pt): CC BY-SA 4.0, © 2026 Ali Asiri (LICENSE-WEIGHTS). Attribute as "Sev Arabic Preview v0.1 by Ali Asiri"; share adaptations under the same license. Chosen because the fine-tuning data include ArBanking77 (CC BY-SA 4.0), whose share-alike condition may extend to trained weights.
2. Code (runtime, examples, smoke test): Apache-2.0, © 2026 Ali Asiri (LICENSE). The vendored open-jev modules remain Apache-2.0, © 2026 Intikhab Azam.
3. Data and base model: MASSIVE (CC BY 4.0) and ArBanking77 (CC BY-SA 4.0) attribution and citations above; their licenses apply to the data itself. mmBERT-base keeps its MIT license and is not redistributed. Tokenizer files are the mmBERT-base tokenizer (MIT) with two added marker tokens.

Status

Separate from Sev's final release path: the balanced corpus work, finalist training, architecture selection and the final release validation continue under their own rules. Preview version numbers do not imply any of them.

Contact

Questions, issues and reproducible test reports: @3li3 on X · assiri@gmail.com

Citation

See CITATION.cff, or:

@software{asiri_2026_sev_arabic_preview,
  author  = {Asiri, Ali},
  title   = {Sev Arabic Preview v0.1},
  year    = {2026},
  version = {0.1.0},
  doi     = {10.5281/zenodo.22979051},
  url     = {https://github.com/3ssiri/sev-arabic-preview},
  note    = {Research preview}
}