Request access to DialectFix
DialectFix is released for non-commercial research only. Requests are approved automatically.
By requesting access you agree to use DialectFix only for non-commercial research and to the terms in the LICENSE file.
Log in or Sign Up to review the conditions and access this model content.
DialectFix
DialectFix normalizes the spelling of dialectal Arabic into CODA (Conventional Orthography for Dialectal Arabic). It fixes inconsistent spelling, ta marbuta / ha confusion, letter substitutions, word splits and merges, letter elongation and punctuation spacing, while keeping the dialect's own words and grammar. It does not translate into Modern Standard Arabic.
It supports 12 dialects: Egyptian, Lebanese, Syrian, Palestinian, Qatari, Emirati, Iraqi, Yemeni, Sudanese, Libyan, Tunisian and Moroccan.
| Parameters | 143M |
| Type | character-level edit tagger (not generative) |
| Input | one or more sentences + the dialect name |
| Output | the same text with CODA spelling |
| License | research-only, non-commercial (see LICENSE) |
Architecture
DialectFix is a sequence tagger. It predicts one edit label per input character instead of generating text, so it is fast and never rewrites words it was not asked to fix.
- Encoder: a 12-layer Arabic BERT-base encoder (hidden size 768) reads the input. Each character receives the state of the subword token that contains it.
- Character features: a learned 64-dimensional character embedding is concatenated to the encoder state.
- Sequence layer: a 2-layer bidirectional LSTM (384 units per direction), with a residual linear projection and LayerNorm.
- Edit head: a softmax over 250 edit labels. Each label means keep, delete or replace the character, optionally followed by an inserted string (for example a space or a letter).
- Decoding: an edit is applied only if its probability passes a confidence threshold (0.9, or 0.5 for punctuation edits). Otherwise the character is kept. This keeps precision high.
- Dialect conditioning: the dialect name is prepended to the input as a prefix word. The model uses it to apply that dialect's spelling conventions, and it is removed from the output.
Long inputs are processed in chunks of 100 words.
How to use
pip install torch transformers safetensors huggingface_hub
huggingface-cli login # after your access request is approved
from huggingface_hub import snapshot_download
import sys
path = snapshot_download("ahmedsamirtarjama/dialectfix")
sys.path.insert(0, path)
from dialectfix import DialectFix
fixer = DialectFix.from_pretrained(path) # uses a GPU if available, otherwise the CPU
print(fixer.fix("ุงูุดุฌุฑู ุงูุฎุจูุซู ุงุตููุง ุฎุงูุณ ููุฑุนูุง ูู ุงูู
ุฌุงุฑู", dialect="Iraqi"))
# ุงูุดุฌุฑุฉ ุงูุฎุจูุซุฉ ุงุตููุง ุฎุงูุณ ููุฑุนูุง ูู ุงูู
ุฌุงุฑู
print(fixer.fix(["ุงู ุงูุชุฑ ุดู ุจุชุณุชุฎุฏู
ูุง ูุชุฑูุญูุง ุนูู ุงูู
ุฏุฑุณุฉ"], dialect="Palestinian"))
# ['ุงู ุงูุซุฑ ุดู ุจุชุณุชุฎุฏู
ูุง ูุชุฑูุญูุง ุนูู ุงูู
ุฏุฑุณุฉ']
Arguments of fix:
texts: a string or a list of strings.dialect: one ofEgyptian,Lebanese,Syrian,Palestinian,Qatari,Emirati,Iraqi,Yemeni,Sudanese,Libyan,Tunisian,Moroccan. The Arabic label also works.threshold(default 0.9): minimum confidence for a letter edit. Lower values fix more but make more mistakes.threshold_punct(default 0.5): the same, for punctuation edits.batch_size(default 32).
Diacritics and tatweel are removed from the input before correction.
More examples
| Dialect | Input | DialectFix output |
|---|---|---|
| Iraqi | ู ู ุฑุงุญ ูุงุฑุจูู ุงุบุณู ุงูุฏู ู ูู | ู ู ุฑุงุญ ูุฃุฑุจูู ุงุบุณู ุงูุฏู ู ูู |
| Emirati | ู ุฑ ุงุชุตู ุจุณ ูุงุชุตู ุจู ููุงููุงู | ู ุฑ ุงุชุตู ุจุณ ูุงุชุตู ุจู ููุง ููุงู |
| Emirati | ู ูุฑู : ุนููู ู ูุฑููููู | ู ูุฑุฉ : ุนููู ู ูุฑูู |
| Syrian | ุฅููุง ุชู ูุงูู ู ุงููุง ุชู ูุญูู | ุงููุง ุชู ูุงูู ู ุงููุง ุชู ูุญูู |
| Moroccan | ุฑุงู ุชู ุงุ ู ูุงุจู ู ูุชุจ ุงุณุชุนูุงู ุงุช ุงูุณูุงุญ ุจุงูุถุจุท. | ุฑุงู ุซู ุงุ ู ูุงุจู ู ูุชุจ ุงุณุชุนูุงู ุงุช ุงูุณูุงุญ ุจุงูุถุจุท. |
| Tunisian | ุชุญุจ ูุฑูู ุฉ ู ุณูุฑ ูู ูููุชูุ | ุชุญุจ ูุฑูู ุฉ ูุณูุฑ ูู ูููุชูุ |
Training data
The model was trained on about 121,000 sentence pairs (raw text โ CODA spelling) across the 12 dialects:
- About 55,000 human-annotated pairs. Native-speaker linguists corrected the spelling to CODA.
- About 66,000 automatically labelled pairs. These were added to widen vocabulary coverage for the five North African, Egyptian, Levantine and Gulf dialects with the most data. Their labels were produced automatically and filtered by agreement.
The text covers several styles:
- short everyday and travel conversation sentences;
- social-media posts and comments;
- web-forum fiction and novels (Gulf);
- informal discussions about local life, news and religion.
Spelling conventions follow CODA:
- word-initial hamza on alef is not written (ุฃูุง โ ุงูุง);
- ta marbuta is restored (ู โ ุฉ);
- dialect-specific letters are written etymologically (for example ุซ / ุฐ / ู where pronunciation differs);
- clitics are attached or separated consistently;
- repeated-letter elongation is removed;
- punctuation is attached to the preceding word.
All text is undiacritized.
Each training sentence carries a country-level dialect label, which is passed to the model as a prefix.
Evaluation
The model was evaluated on held-out, human-annotated test sets that were never used for training or model selection. The metric is the M2 scorer F0.5 over word-level edits, the standard metric for Arabic error correction; it weights precision twice as much as recall. All settings, including the 0.9 threshold and the checkpoint, were chosen on validation data only, and each test set was decoded once.
| Dialect | Test sentences | Precision | Recall | F0.5 |
|---|---|---|---|---|
| Qatari | 300 | 94.9 | 77.5 | 90.8 |
| Lebanese | 300 | 92.9 | 80.6 | 90.2 |
| Tunisian | 300 | 91.7 | 76.9 | 88.3 |
| Egyptian | 300 | 89.6 | 83.0 | 88.2 |
| Moroccan | 300 | 92.1 | 74.4 | 87.9 |
| Syrian | 567 | 91.5 | 72.9 | 87.0 |
| Emirati | 1,442 | 86.1 | 77.0 | 84.1 |
| Yemeni | 2,654 | 89.9 | 61.3 | 82.2 |
| Palestinian | 606 | 86.2 | 61.1 | 79.7 |
| Iraqi | 297 | 83.0 | 52.2 | 74.2 |
| Sudanese | 208 | 81.4 | 47.6 | 71.3 |
| Libyan | 249 | 81.2 | 44.6 | 69.7 |
MADAR CODA benchmark
MADAR CODA is the standard public benchmark for dialectal Arabic spelling normalization. It covers five city dialects: Beirut, Cairo, Doha, Rabat and Tunis. The scores below use its official test set (1,500 sentences) and the M2 F0.5 metric of the published work.
| System | Type | Test F0.5 |
|---|---|---|
| GPT-4o (few-shot), reported by Alhafni & Habash (2025) | LLM prompting | 55.9 |
| Alhafni et al. (2024) | single model | 86.3 |
| Alhafni & Habash (2025), best single model | single model | 86.5 |
| Alhafni & Habash (2025), 3-model ensemble | ensemble | 88.9 |
| DialectFix (143M) | single model | 88.9 |
DialectFix matches the best published ensemble with one 143M-parameter model, and it scores 2.4 points above the best published single model. Above, the five rows Qatari, Lebanese, Tunisian, Egyptian and Moroccan are this test set split by dialect.
Notes:
- Lebanese also has a second, very small social-media test set (43 sentences), where F0.5 is 68.3.
- Yemeni: some posts are reposted, and 126 test sentences also appear in the training data. Excluding them gives F0.5 81.97.
- Precision is above 80% for every dialect. The weaker dialects (Iraqi, Sudanese, Libyan) lose mainly on recall: the model misses some corrections rather than making wrong ones. They had the least training data, about 2,000 sentences each.
Limitations
- The dialect must be given. A wrong dialect label lowers quality, because spelling conventions differ by dialect. Dialects outside the 12 above have not been evaluated.
- Spelling normalization only. It does not fix grammar, word choice or style, and it does not translate.
- Domain. Formal Modern Standard Arabic, code-switched text (Arabizi, Latin script) and very long documents were not part of training.
- Output follows CODA conventions, which may differ from other spelling standards (for example, word-initial hamza is dropped).
License
DialectFix is released under a research-only, non-commercial license (see LICENSE). Access requires accepting these terms, and requests are approved automatically. For commercial use, contact the author.
- Downloads last month
- -