ALIM Arabic Models โ diacritization (prose + verse) + meter classification
Character-level neural heads from the ALIM neuro-symbolic Arabic project: two specialist diacritizers (12M params each, BiLSTM + multi-head attention, three heads: vowel / shadda / tanwin) and one 16-class Arabic meter classifier (char-embedding + 2-layer BiLSTM). Weights are safetensors; every number below is measured on fixed eval sets with seed 42 โ see Evaluation.
alim-arabic-models/
โโโ README.md # this file
โโโ inference_example.py # self-contained CPU inference (torch + safetensors)
โโโ diac-prose/ # prose diacritizer โ prose DER 12.9%
โ โโโ model.safetensors, config.json, README.md
โโโ diac-verse/ # verse diacritizer โ verse DER 11.7%
โ โโโ model.safetensors, config.json, README.md
โโโ meter-16/ # 16 canonical buhur โ test 79.6%, gold-327 84.4%
โโโ model.safetensors, config.json, vocab.json, idx_to_meter.json, README.md
Why two diacritizers
A single 12M model cannot serve both domains (measured share curve, same 10-epoch budget, seed 42):
| Poetry share | Prose DER (clean-2k) | Verse DER (313 bayts) |
|---|---|---|
0% โ diac-prose |
12.9% | 45.8% |
| 35% | 15.0% | 24.6% |
100% โ diac-verse |
45.9% | 11.7% |
Each domain without its data collapses to ~46%. Route by source: poetry
corpora โ diac-verse, everything else โ diac-prose. Per-class recall
(prose / verse): bare 93/95%, fatha 88/89%, damma 88/81%, kasra 90/89%,
sukun 90/91%, tanwin 83/86%, shadda 94/95%.
Quick start
# pip install torch safetensors
import sys
sys.path.insert(0, ".")
from inference_example import vocalize, predict_meter
print(vocalize("diac-prose", "ุงูุนูู
ููุฑ ููุฏู ุตุงุญุจู ุงูู ุงูุฎูุฑ"))
print(predict_meter("meter-16", "ููุง ูุจู ู
ู ุฐูุฑู ุญุจูุจ ูู
ูุฒู", "ุจุณูุท ุงูููู ุจูู ุงูุฏุฎูู ูุญูู
ู"))
Tokenizer (both diacritizers): ord(char) % 50000 per Arabic letter;
diacritics and spaces carry no ids; pad id 0. Meter input: "sadr | ajuz",
first 128 chars, char ids from meter-16/vocab.json.
Training recipes (reproducible)
- diac-prose: 10,802 synthetic words + 205k real fully-vocalized prose lines (50%+ density, eval-deduped), 10 epochs, AdamW 1e-3, batch 32, class weights + nunation augmentation, seed 42. Loss 1.03 โ 0.48.
- diac-verse: same recipe on 54k vocalized poetry shatrs (ashaar).
- meter-16: 36,768 bayts / 16 canonical families, class weights clamped 10ร (raw 766ร inverse-frequency collapses training below chance โ documented failure), label smoothing 0.1, 10 epochs. Test 79.6% (was 68.8% unweighted); tail classes (nโค10) score above zero.
Evaluation protocol
- Prose: Shamela clean-2k (2,000 fully-vocalized lines, 0 eval contamination โ train lines exact-match deduped against it), DER over all letters + marked recall + per-vowel/shadda/tanwin recall.
- Verse: 313 bayts / 15 families, density โฅ 0.8, row meter labels mapped to canonical families; DER argmax. (A 32-bayt easy-only subset twice gave false readings โ retired.)
- Meter: held-out test (2,177) + hand gold-327 at bahr-family level. IAA flag: the 327-set is 25 gold + 302 silver-candidate rows with no second-annotator IAA โ 84.4% is diagnostic until IAA lands (plan ยง5.3).
- Tiny unseeded runs show ยฑ30pt noise: all runs use seed 42.
Limitations
- Verse gold is partially vocalized and noisy: ~8pp of verse DER is a partial-gold artifact; Arkashaar spellings vary (hamza seats).
- Meter-constrained decoding (family templates) hurts variant verses (ู ุฌุฒูุก etc.) โ ship argmax for verse until sub-meter-aware constraints land.
- Prose SOTA reference is 6.6% DER; these 12M specialists sit at ~12%.
Version / provenance
- v1.0 (2026-09-08):
diac-proseโ diac_mix100k ยทdiac-verseโ diac_poetryonly ยทmeter-16โ meter16w. Source: ALIM project (neuro-symbolic Arabic: sarf/nahw/prosody rules + neural heads). - License: MIT. Contact via repo discussions.