algerianDeBERTa

A compact DeBERTa-v2 encoder pre-trained from scratch on Algerian text — Darja (dialect), Arabizi (Arabic in Latin script), French, Modern Standard Arabic, and the heavy code-switching that real Algerians actually write.

algerianDeBERTa is a ~60M-parameter masked language model built specifically for the Algerian linguistic space. It was trained on a custom Algerian corpus with a purpose-built tokenizer, and is designed to be a small, fast, fine-tunable backbone for downstream Algerian NLP tasks (sentiment, intent, classification, retrieval, NER).

Headline: it matches ~96–98% of DziriBERT's macro-F1 on Algerian sentiment benchmarks while using ≈ half the parameters and half the memory (231 MB vs 475 MB).

---

Why another Algerian model?

Algerian is a genuinely hard, low-resource setting:

• Three+ languages in one sentence — Arabic, French and Tamazight vocabulary mixed freely.
• Two scripts — the same word is written in Arabic letters and in Latin "Arabizi" (e.g. wach rak / واش راك), often with digits standing in for sounds (3 = ع, 7 = ح, 9 = ق).
• No standard orthography — spelling is phonetic and varies per writer and per region.

General multilingual models and even MSA-centric Arabic models underperform here. algerianDeBERTa tackles this with (1) a tokenizer learned directly on Algerian text and (2) DeBERTa-v2's disentangled attention, which is strong at modeling the messy, non-canonical token order of dialectal writing — all in a deliberately small footprint.

---

Highlights

Architecture	DeBERTa-v2 encoder (disentangled attention, relative positions)
Size	≈ 60M parameters · 231 MB (fp32) — about half of DziriBERT
Tokenizer	Custom fast tokenizer, 30k vocab, trained on Algerian Darja/Arabizi/French
Languages	Algerian Darja, Arabizi, French, MSA, code-switched
Objective	Masked Language Modeling (MLM), trained from scratch
Best for	Fine-tuning on Algerian classification / retrieval / token tasks

---

Performance vs DziriBERT

DziriBERT is the reference Transformer for Algerian dialect and the prior state of the art on Algerian text classification (it beats mBERT, AraBERT, CAMeLBERT, QARiB and MARBERT on these tasks). It is the right yardstick.

Both models were fine-tuned identically (same heads, same hyper-parameters — 3 epochs, LR 2e-5, max length 128, seed 42, 85/15 split) on three Algerian sentiment datasets, and compared on macro-F1:

Dataset	DziriBERT (~124M)	algerianDeBERTa (~60M)	Retained
Herouini	0.819	0.785	95.8%
DzSentiA	0.877	0.859	97.9%
AbdouYT	0.790	0.764	96.7%

DziriBERT keeps a small edge in raw macro-F1 (≈ 2–3 points), which is expected from a model with roughly twice the parameters. The point of algerianDeBERTa is the trade-off: near-parity quality at half the size.

Memory footprint

Model	Parameters	On-disk (fp32)
algerianDeBERTa	≈ 60M	231 MB
DziriBERT	≈ 124M	475 MB

That ~51% reduction in size means lower memory, faster inference, and cheaper fine-tuning — useful for edge deployment and for stacking the model into multi-stage pipelines.

---

How to use

Fill-mask

from transformers import pipeline

fill = pipeline("fill-mask", model="81melody/algerianDeBERTa")
fill("راني [MASK] بزاف اليوم")
fill("salam, wach rak [MASK]?")

Get embeddings / features

import torch
from transformers import AutoTokenizer, AutoModel

tok = AutoTokenizer.from_pretrained("81melody/algerianDeBERTa")
model = AutoModel.from_pretrained("81melody/algerianDeBERTa")

text = "نحب نشري دار في وهران"
inputs = tok(text, return_tensors="pt")
with torch.no_grad():
    out = model(**inputs)
# CLS / pooled representation
cls = out.last_hidden_state[:, 0]
print(cls.shape)  # (1, 512)

Fine-tune for classification

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tok = AutoTokenizer.from_pretrained("81melody/algerianDeBERTa")
model = AutoModelForSequenceClassification.from_pretrained(
    "81melody/algerianDeBERTa", num_labels=3
)
# then train with transformers.Trainer on your Algerian dataset

---

Model details

Hyper-parameter	Value
Model type	deberta-v2 (DebertaV2ForMaskedLM)
Hidden size	512
Layers	12
Attention heads	8
Intermediate size	2048
Max sequence length	512
Vocabulary size	30,000
Position encoding	relative + absolute (disentangled attention, c2p + p2c)
Parameters	≈ 60M
Precision	fp32

---

Training

Data

Pre-trained from scratch on a custom Algerian corpus (~115 MB of cleaned text) assembled from public Algerian web and social-media content, including ~45k YouTube comments. The corpus deliberately spans the full Algerian register:

• Algerian Darja in Arabic script
• Arabizi (Latin-script Algerian with digit substitutions)
• French and French↔Arabic code-switching
• some Modern Standard Arabic

Text was cleaned and normalized (deduplication, noise/boilerplate removal) before training. The tokenizer was trained on this same corpus so that frequent Darja/Arabizi sub-words get dedicated tokens instead of being shattered into bytes.

Procedure

Setting	Value
Objective	Masked Language Modeling
Epochs	2 (best checkpoint at ~1.7 epochs / 7,000 steps)
Batch size	16
Peak learning rate	3e-5 (with warmup)
Final held-out MLM loss	3.40
Framework	🤗 Transformers

The held-out masked-LM loss decreases steadily over training, from ~3.46 to 3.40.

---

Models built on algerianDeBERTa

algerianDeBERTa is the shared backbone of the DZ Pulse Algerian NLP stack:

Model	Task
81melody/algerianDeBERTa-realestate-intent	Buyer / Seller / Irrelevant intent classification for Algerian real-estate posts
81melody/ouedkniss-search-crossencoder	Cross-encoder reranker for Algerian marketplace search

---

Intended uses & limitations

Intended uses

• A base encoder to fine-tune on Algerian-dialect downstream tasks: sentiment, intent, topic/spam classification, NER, retrieval/reranking.
• Masked-token infilling and feature extraction for Algerian text.

Limitations & biases

• It is a base MLM, not an instruction-tuned or chat model — it does not "answer" prompts; it needs fine-tuning for most tasks.
• Trained on public social-media text, so it can reflect the biases, slang, and toxicity present in that data.
• Coverage is strongest for the Darja/Arabizi/French mix it was trained on; performance on pure MSA or on Tamazight may be weaker than dedicated models.
• The corpus is region-skewed toward the dialects most common online; some regional variants are under-represented.

---

Citation

@misc{himeur2026algerianDeBERTa,
  title        = {algerianDeBERTa: A Compact DeBERTa-v2 Encoder Pre-trained from Scratch
                  for Algerian Darja, Arabizi and Code-Switching},
  author       = {Himeur, Ayoub},
  year         = {2026},
  publisher    = {Hugging Face},
  url          = {https://huggingface.co/81melody/algerianDeBERTa}
}

Acknowledgements

Benchmarked against DziriBERT (Abdaoui et al.), the prior state of the art for Algerian dialect modeling.

---

License

Released under the Apache 2.0 license.