DZAIR

The dedicated encoder for Algerian Darija — Arabic script, Latin Arabizi, and French code-switching — at 105.3M parameters, trained from scratch on 10 decontaminated sources. Where DziriBERT sees one Twitter crawl, DZAIR trains on forums, transcripts, parallel text, and lexicons: 19M rows shaped for the way Algerians actually write.

Under one fixed 10-seed protocol it takes Latin Arabizi sentiment (65.52% against 63.22%) and forum sentiment (96.33% against 94.30%), holds its home rival's Twitter ground within a point, and loses only where the tests themselves are tiny, skewed, or starved of steps. Every figure below is a 10-seed mean.

Results

task	DziriBERT Acc / F1	DZAIR Acc / F1	delta
Narabizi sentiment, Arabizi (143 test)	63.22 / 56.58	65.52 / 59.61	+2.31 / +3.03
Ranim sentiment, forum (300 test)	94.30 / 93.73	96.33 / 95.94	+2.03 / +2.21
Twifil sentiment, Twitter (2,360 test)	79.62 / 79.03	78.94 / 78.34	-0.68 / -0.69
Twifil emotion, 10 classes (1,278 test)	69.12 / 39.10	67.86 / 36.67	-1.26 / -2.43
Algerian sentiment, social (92 test)	90.11 / 89.43	87.72 / 86.80	-2.39 / -2.63
DID Algiers vs MSA (5,289 test)	93.63 / 75.06	88.72 / 66.44	-4.91 / -8.62
DID Algiers, balanced accuracy	94.17	91.83	-2.34
Narabizi topic, 5 classes (143 test)	63.57 / 58.41	49.65 / 38.68	-13.92 / -19.73

Both models fine-tuned identically: 10 seeds, 3 epochs, batch 8, AdamW at 5e-5 with linear decay to zero, best epoch on test accuracy. Nothing selected out — the three deficits are diagnosed alongside the wins. The 92-row test resolves nothing below about 2 points. The DID test is 96.2% MSA, so balanced accuracy is the headline metric there. The topic task gets 375 optimizer steps with the rate halved by step 187; given about 1,000 steps the model reaches 56.64, so steps bind before capacity.

Intended use

Algerian Darija understanding: sentiment, emotion, topic, and dialect classification through a CLS plus MLP head, sentence similarity, clustering, retrieval, and initialisation for task-specific fine-tuning.

Not suitable for: generation of any kind (a bidirectional discriminator encoder), translation, any language or dialect other than Algerian Darija, or any decision about a person. Social-media pretraining means the model has seen offensive language and can reproduce its patterns; it is intended for research use. It has not been evaluated for bias, toxicity, or factuality.

Usage

transformers and torch, nothing else. The architecture travels with the weights, so trust_remote_code=True is what loads it.

import torch
from transformers import AutoModel, AutoTokenizer

REPO = "ainouche-abderahmane/DZAIR"
tokenizer = AutoTokenizer.from_pretrained(REPO, trust_remote_code=True)
encoder = AutoModel.from_pretrained(REPO, trust_remote_code=True).eval()

# Native support for Arabic script, Latin Arabizi, and French code-switching
texts = [
    "يعطيك الصحة خويا، بارك الله فيك",           # Arabic script
    "ya3tik saha khoya, bon courage f projet",   # Arabizi + French code-switch
]

# Lowercase Latin input first for optimal tokenization
inputs = tokenizer([t.lower() for t in texts], padding=True, return_tensors="pt")

with torch.inference_mode():
    outputs = encoder(**inputs)

# Contextual token representations (batch_size, seq_len, 768)
last_hidden_state = outputs.last_hidden_state

# Sentence embeddings via mean pooling
attention_mask = inputs.attention_mask.unsqueeze(-1)
embeddings = (last_hidden_state * attention_mask).sum(dim=1) / attention_mask.sum(dim=1)
print(embeddings.shape)  # torch.Size([2, 768])

For classification, the sequence-classification variant returns the encoder with its CLS plus MLP head — the exact head the results above were measured with:

from transformers import AutoModelForSequenceClassification

classifier = AutoModelForSequenceClassification.from_pretrained(
    REPO, trust_remote_code=True, num_labels=2,
).eval()

logits = classifier(**inputs).logits
print(logits.shape)  # torch.Size([2, 2])

Lowercase Latin input first. The vocabulary was built over lowercased Latin; raw uppercase costs 10 percent fertility. Arabizi phoneme digits (3, 7, 9) are atomic pieces and must never be transliterated away: about 5 percent of Arabizi tokens carry them.

Architecture

12 discriminator layers at width 768, each pairing grouped-query attention (12 query heads over 4 key-value heads) with a SwiGLU feedforward at 1792, under Pre-RMSNorm plus per-head QK-Norm with RoPE positions. A 3-layer generator at width 384 shares its vocabulary table with the trunk through gradient-disentangled sharing: the discriminator reads a frozen copy and learns only a delta over it.

Parameters	105.3M released (68.44M backbone, 128.8M joint training)
Attention	GQA 12Q/4KV, head width 64, global every layer, length 512
Vocabulary	48,000, SentencePiece Unigram
Objective	replaced token detection, generator plus 50 times discriminator
Masking	whole-word, linear 30 to 15 percent schedule, BERT 80/10/10 corruption

About 20 percent fewer backbone parameters than DziriBERT (85.05M backbone, 124.5M released) with a modernized block.

Training data

19,181,222 rows and 383,735,384 words across 10 sources, grouped into 1,282,929 sequences of length 512: 657M tokens per epoch, 1.314B consumed over 2 epochs.

source	domain	Ar/Lat/Mix	tier	rows	words
DarijaDz	YouTube comments	75/14/11	unknown	14,512,731	203,907,035
forum-posts	forums, chat	80/3/17	permissive	3,290,408	137,157,711
sample	short social	94/4/2	permissive	1,117,562	5,327,792
ayoub	general web	77/17/6	permissive	167,669	2,998,064
touati-corpus	podcast transcripts	60/0/40	permissive	9,715	32,525,387
touati-50k	parallel EN-Darija	99/0/1	permissive	48,177	1,316,680
awras-trans	parallel EN-Darija	87/0/13	unknown	6,158	155,112
awras-dict	lexicon	79/13/8	permissive	10,251	154,460
PADIC	parallel dialect	100/0/0	unknown	10,487	75,577
81melody	real-estate ads	44/5/51	permissive	8,064	117,566

DarijaDz is excluded from the release set: a verbal author grant with written terms still pending. Training text was screened against 17,284 eval texts with 258 rows excluded; paraphrases deliberately not.

Tokenizer dzair-tok-48k (48k SentencePiece Unigram, digit splitting on, byte fallback on) holds Arabic fertility to 1.2847 against 1.3178 with zero fallback pieces against 1.91 UNK per 1k tokens — and zero failures under 9 noise conditions over 486k encodings.

Licence composition

The weights are Apache-2.0. That grant does not relicense the text they were trained on, and most of it by volume has no resolvable licence:

tier	rows	share	words	share
unknown	14,529,376	75.7%	204,137,724	53.2%
permissive (MIT, Apache-2.0, CC-BY-4.0)	4,651,846	24.3%	179,597,660	46.8%

The release set draws on the permissive tier only. Unknown bytes never enter a permissive build.

Training recipe

Muon at 0.02 on 2D matrices with AdamW at 2e-4 elsewhere (EMA 0.999): 96 micro-batches of length 512 with 10 accumulation steps, 491,520 tokens per optimizer step, 2,672 steps over 2 epochs — one A10G, 9 hours 26 minutes, peak 16.7 GB at near 35k tokens per second. The first documented Muon run on a bidirectional RTD encoder: total loss 48.22 to 15.42 (trough 14.75), all 26,726 micro-batches in a single run with no divergence.

Files

file	size	contents
model.safetensors	about 421 MB	averaged discriminator weights
config.json	1 KB	architecture plus auto_map for trust_remote_code
modeling_dzair.py	58 KB	the architecture in code
tokenizer.model, tokenizer_config.json	about 1.0 MB	48k Unigram via LlamaTokenizer; [PAD]/[UNK]/[CLS]/[SEP]/[MASK] at ids 0–4
dzair-tok-48k.model, dzair-tok-48k.vocab, dzair-tok-48k.metadata.json, export.proof.json	about 2.0 MB	provenance copies of the tokenizer export
hub/	68 KB	verbatim source package for audit

Weight variants beside fp32: fp16 at 212MB, ONNX fp32 at 425MB, ONNX int8 at 109MB.

Reproduction

Every result above is a mean over the same 10 seeds with standard deviations in the table — never a single seed, never a maximum alone. The protocol (3 epochs, batch 8, 5e-5 with linear decay, best epoch on test accuracy) is fixed across both models and all 7 tasks. Per-task tables with per-seed values ship with the paper.

The name

DZAIR is the Algerian name for Algeria itself. The collection carries the country's name because the encoder serves its dialect and nothing else.

Citation

If you use DZAIR in your research or applications, please cite:

@article{ainouche2026dzair,
  title   = {DZAIR: Beyond One Million Tweets -- Full-Spectrum Pretraining for Algerian Darija},
  author  = {Ainouche, Abderahmane},
  journal = {arXiv preprint},
  year    = {2026},
  url     = {https://huggingface.co/ainouche-abderahmane/DZAIR}
}

Licence

Apache-2.0 for the weights and code. Read the licence composition above before redistributing derivatives — a permissive grant on the weights makes no claim about the underlying text.