O96a/sudanizer-72k

🇸🇩 Sudanizer: Sudanese Arabic BPE Tokenizer

Creator: O96a
Initiative: Sudaverse
For the Sudanese NLP Community

🎯 Mission

This tokenizer is part of the Sudaverse initiative dedicated to preserving Sudanese linguistic heritage and advancing Sudanese NLP. Developed specifically for the Sudanese Arabic dialect to ensure our language is properly represented in AI systems.

Model Description

Sudanizer is a production-ready Byte-Pair Encoding (BPE) tokenizer trained on ~49 million words of authentic Sudanese dialect text. It recognizes words and expressions unique to Sudanese culture and communication.

Key Features

• Vocabulary Size: 72,000 tokens
• Algorithm: BPE with ByteLevel pre-tokenization
• Dialect-Specific: Optimized for Sudanese Arabic patterns
• Arabic-Aware: Handles clitics, prefixes, suffixes, emojis
• Normalization: Sudanese orthographic variants (كده→كدا, ده→دا, etc.)

Performance Metrics

Metric	Value
Average Characters per Token	4.536
Compression Ratio	0.2205
UNK Rate	0.0000%
Avg Subwords per Word	1.159
Vocabulary Size	72,000

🚀 Quick Start

# Load tokenizer
tokenizer = PreTrainedTokenizerFast.from_pretrained("O96a/sudanizer-72k")
# Tokenize Sudanese text
text = "لو مغالطني ارح كاكا ليو"
tokens = tokenizer.encode(text)
print(f"Tokens: {tokens}")
# Decode back
decoded = tokenizer.decode(tokens)
print(f"Decoded: {decoded}")
# Get token strings
token_strs = tokenizer.tokenize(text)
print(f"Token strings: {token_strs}")

Training Details

Configuration

• Tokenizer Version: 1.0.0
• Minimum Token Frequency: 3
• Pre-tokenization: ByteLevel (Arabic-optimized)
• Normalization: NFKC + Sudanese dialect normalization
• Corpus Domain: social+stories+youtube+qa

Special Tokens

Token	Purpose	ID
<pad>	Padding	0
<unk>	Unknown	1
<s>	Start of sequence (BOS)	2
</s>	End of sequence (EOS)	3
<mask>	Mask token (MLM)	4
<user>	User message marker	5
<assistant>	Assistant message marker	6
<system>	System message marker	7

💚 For the Sudanese Community

✅ Intended Use:

• Sudanese Arabic NLP research & development
• Building Sudanese AI assistants and chatbots
• Fine-tuning models on Sudanese data
• Preserving Sudanese linguistic heritage
• Educational purposes for Sudanese language technology

⚠️ Attribution Required: This work is developed by Aamer Mihaysi as part of the Sudaverse initiative for the Sudanese NLP community. When using this tokenizer, please cite appropriately and acknowledge the Sudaverse project.

Comparison with Other Tokenizers

Sudanizer is designed to outperform general Arabic tokenizers on Sudanese text:

Benchmark Tokenizers Tested

1. Aranizer-SP-64k (riotu-lab) — SentencePiece tokenizer tailored for Arabic (~64K vocab)
2. Aranizer-PBE-64k (riotu-lab) — PBE version optimized for Arabic morphology
3. arabic-bpe-tokenizer (IsmaelMousa) — Standard BPE for Arabic (~50K vocab)
4. SARF-Tokenizer (almaghrabima) — Morphologically-aware tokenizer with morpheme segmentation
5. deeplatent-tokenizer (almaghrabima) — SARF-based bilingual tokenizer
6. AraBERTv2 — General Arabic BERT tokenizer
7. CAMeLBERT-Mix — Mixed Arabic dialect tokenizer

Tokenizer	Sudanese Chars/Token	UNK Rate
Sudanizer-72K-v2 (this)	4.54	0.00%
AraBERTv2	~3.5	Higher
CAMeL-BERT	~3.2	Higher
Aranizers	~3.0-3.5	Moderate

Citation

If you use this tokenizer in your research, please cite: bibtex @misc{sudanizer2026, title={Sudanizer: A Production-Ready BPE Tokenizer for Sudanese Arabic}, author={Mihaysi, Aamer}, year={2026}, howpublished={\url{https://huggingface.co/O96a/sudanizer-72k}} } ## License MIT License — This work belongs to the Sudanese NLP community. ## Acknowledgments - Creator: Aamer Mihaysi - Initiative: Sudaverse - HuggingFace for the tokenizers library - Contributors to Sudanese Arabic corpora ## Contact For questions or contributions, please contact via Sudaverse.

Made with 💙 for Sudan. This work belongs to the Sudanese NLP community.