SamAI-Tokenizer

🚀 Try the interactive tokenizer demo

SamAI-Tokenizer is an Arabic-focused byte-level BPE tokenizer for training new Arabic-heavy language models. It has a vocabulary of 65,536 tokens, applies Unicode NFC normalization, supports lossless decoding, and retains English coverage for multilingual training.

Key specifications

Property	Value
Algorithm	Byte-level BPE
Vocabulary size	65,536
Normalization	Unicode NFC
Pre-tokenizer	ByteLevel
Unknown-token fallback	Byte-level coverage
Named special tokens	<pad>, <bos>, <eos>, <unk>, <user>, <assistant>, <system>
Reserved token IDs	7–255
Primary language	Arabic
Secondary language	English

Arabic tokenizer benchmark

SamAI-Tokenizer was evaluated on four complementary Arabic benchmarks. Lower tokens per word indicates more efficient tokenization.

Across the equal-weight macro average of the four benchmarks, SamAI-Tokenizer ranked second among 15 successfully tested tokenizers at 1.6207 tokens per word.

Rank	Tokenizer	Mean tokens/word	Extra tokens versus SamAI
1	ALLaM-7B	1.5021	-4.86%
2	SamAI-Tokenizer	1.6207	baseline
3	Falcon-H1-7B	1.8880	+19.30%
4	Mistral-Small-3.1	1.8929	+20.71%
5	Qwen3.8-Flash-Next	1.9039	+21.94%
6	Qwen3.5-4B	1.9039	+21.94%
7	tiktoken-o200k	1.9919	+26.37%
8	GPT-OSS-20B	1.9919	+26.37%
9	Fanar-1	2.1570	+36.81%
10	DeepSeek-V3	2.2774	+46.25%
11	Qwen2.5-7B	2.3740	+50.25%
12	Qwen3-8B	2.3740	+50.25%
13	GLM-5.3-Flash	2.5225	+60.21%
14	Kimi-K2.6	2.8693	+84.13%
15	tiktoken-cl100k	4.0800	+164.63%

Installation

pip install -U tokenizers transformers huggingface_hub

Usage

Transformers

from transformers import PreTrainedTokenizerFast

tokenizer = PreTrainedTokenizerFast.from_pretrained(
    "Samai-sa/SamAI-Tokenizer"
)

text = "المملكة العربية السعودية"
ids = tokenizer.encode(text, add_special_tokens=False)

print("Token count:", len(ids))
print("Token IDs:", ids)
print("Tokens:", tokenizer.convert_ids_to_tokens(ids))
print("Decoded:", tokenizer.decode(ids))

Tokenizers

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

path = hf_hub_download(
    repo_id="Samai-sa/SamAI-Tokenizer",
    filename="tokenizer.json",
)
tokenizer = Tokenizer.from_file(path)

encoding = tokenizer.encode(
    "السلام عليكم ورحمة الله وبركاته",
    add_special_tokens=False,
)

print("Token count:", len(encoding.ids))
print("Token IDs:", encoding.ids)
print("Tokens:", encoding.tokens)
print("Decoded:", tokenizer.decode(encoding.ids))

Interactive demo

The SamAI Tokenizer Playground lets users enter Arabic or English text and inspect:

• token count and tokens per word;
• token IDs and raw token pieces;
• original text spans and offsets;
• decoded text for round-trip verification.

Validation

The released validation run reported zero round-trip failures and zero unknown tokens across samples from schoolbooks, AraMix Arabic, FineWeb2 Arabic, and FineWeb English. Additional stress tests cover Modern Standard Arabic, dialects, Arabizi, English, code, mathematics, and Unicode edge cases.

See reports/validation.json and reports/stress.json.

Benchmark methodology

• All available examples were used, followed by exact-text deduplication within each benchmark.
• Token counts exclude chat templates and added special tokens.
• Tokens per word is total tokens divided by whitespace-separated words.
• The primary macro ranking gives each benchmark 25% weight.
• The pooled report sums all tokens and words before calculating the ratio.
• Tokenizers producing identical results are tokenizer-family duplicates, not independent confirmations.

To reproduce the benchmark:

pip install -r benchmark/requirements.txt
python benchmark/benchmark.py

Limitations

• Tokenization efficiency does not measure language-model quality, reasoning, factuality, safety, or downstream accuracy.
• Fully vocalized Arabic is the clearest weakness. One heavily diacritized ArabicMMLU category accounts for most of the gap.
• Whitespace-separated words are a transparent denominator, not a linguistic segmentation of Arabic morphology.
• Replacing the tokenizer of an already pretrained model requires resizing and retraining its embedding and output layers. This is not a drop-in vocabulary replacement.

License and data terms

The tokenizer files and repository code are released under Apache-2.0. Source and benchmark datasets remain subject to their own licenses and terms. No source-corpus or benchmark documents are redistributed here.