Algerian NLP Collective

Open language technology for Algerian Darija (Darja, arq) — the Arabic dialect of Algeria, written in Arabic script, in Latin Arabizi, and in constant French code-switching, often inside a single sentence. A low-resource language with a high-variance orthography and no standard spelling: every artifact published here carries its provenance, its licence composition, and the measurement behind each claim. Nothing ships on a promise.

The rules every artifact follows

Provenance is a precondition, not a footnote. Every record carries a source id, a licence, and a retrieval date. The registry is the source of truth, the fetcher writes an append-only manifest with a checksum per artifact, and a verify step re-hashes the bytes on disk. A record whose origin cannot be named does not enter. Licences are load-bearing and mechanically separable — transparent open licensing such as CC BY 4.0 or Apache 2.0 — so a permissive-only release is cut by code, never by hand. A permissive grant on weights makes no claim about the text underneath them, and each card publishes the full composition so you can judge for yourself.

Ethical sanitization before any release. No release ships without a PII sweep — handles, phone numbers, private personal data — into counted rejects with named reasons, greedy by design: a false reject beats a leaked number. Raw STT/ASR noise artifacts (timestamps, speaker labels, decoder fragments) are stripped at extraction, never modelled. Web and social text is used as published and can carry offensive language, so research-use limits are stated on the card, not implied.

Algerian only, and purity is measured before anything filters. Zero Moroccan, zero Tunisian, zero MSA-as-Darija. Script and dialect profiles are drawn from seeded reservoirs over the whole file, never from head samples, and the confusion matrix is established before any filter touches a row. Quality filtering is mechanical and counted — length bounds, repetition, junk, empty — each drop to a rejects file with its reason. A classifier gate must survive gold controls before it touches a corpus; one that eats genuine Darija to catch MSA is refused.

Deduplication is audited. Global exact-hash dedup with a declared keep order, so shared rows credit the most openly licensed source, with the report beside the artifact. MinHash near-duplicate screening runs on every bulk web release to keep repetitive and scraped spam bloat out; where paraphrase overlap is out of scope, the residual is stated, not solved.

Script and code-switching are metadata, never noise. Every record is tagged Arabic-script, Latin Arabizi, or mixed, and per-script counts ship with every release. Scripts are never unified by transliteration, and French spans stay as code-switch context, exactly as Algerians write them.

Decontamination is measured, with a positive control. Training text is screened against fixed eval fingerprint sets, matches are excluded with their origins, and the detector must catch a planted leak before it is believed.

Normalisation is versioned. Every rule change is a version bump with a documented diff, and every artifact is stamped with the version that produced it. Rules apply in a fixed, published order — Unicode normalisation, script-aware casing, character unification, digit handling, noise-mark stripping, invisible-control removal, elongation squeezing, whitespace collapse — and Arabizi phoneme digits keep their value through all of it.

Documentation is standardised. One copy of each card, reviewed before release: a proper YAML header, clear word and token counts quoted from the stats file beside the artifact, a loading snippet, and a verifiable scientific citation (arXiv ID / Zenodo DOI).

from datasets import load_dataset

ds = load_dataset("algerian-nlp/<dataset>", split="train")
print(ds[0])  # every row carries its source id, licence, and retrieval date

Measurement reports what it measures. Results are same-statistic comparisons scored by us over fixed seeds with support stated — never a single seed, never a maximum alone, never a carried number. Deficits ship beside wins, and what was not measured reads as unmeasured, never as zero.

Licence

Weights and code are Apache-2.0. The text underneath is not, and cannot be relicensed by that grant.