alinawazmahar/sindhi-tokenizer

Sindhi Tokenizer (سنڌي): SentencePiece Unigram Tokenizer for Sindhi NLP and LLMs

Sindhi Tokenizer is an open-source subword tokenizer for the Sindhi language (Perso-Arabic script). It is a SentencePiece Unigram model with byte-level fallback and full character coverage, trained on about 1.46 million sentences. It ships in four vocabulary sizes (32,000, 64,000, 100,000 and 150,000); 64,000 is the recommended default. On the project's benchmark of 18 candidate tokenizers across four Sindhi text registers, the 64k and larger sizes use the fewest tokens per word and keep the most words whole in every register, never emit <unk>, and lose no characters on encode/decode round trips.

GitHub repository · Documentation site · Paper · Technical explainer · Interactive demo

Quick answers

• What is it? A tokenizer (vocabulary + segmentation model) for Sindhi text. It converts Sindhi sentences into subword tokens and IDs, and back. It is not a language model and contains no neural weights.
• Which file should I use? sindhi_custom_64000.model. It is the smallest size that has the lowest fertility and highest single-token retention in every register tested.
• How good is it? Mean fertility 1.147 tokens per word and 88.6% of words kept as one token (64k), versus 1.512 / 65.4% for XLM-RoBERTa and 2.896 / 18.2% for Llama 3.2.
• Is it safe for data? Round-trip character loss is 0.0000% (after Unicode normalization) on all four registers. Two other Sindhi tokenizers silently delete characters (see Round-trip fidelity).
• What is it not? Not validated for speech recognition or text-to-speech, not for Devanagari-script Sindhi, and not yet tested in a downstream model. See Limitations.

Key facts

Property	Value
Language	Sindhi (sd), Arabic/Perso-Arabic script
Algorithm	SentencePiece Unigram language model
Byte fallback	Yes (byte_fallback=True), so unknown characters become <0xNN> byte tokens, not <unk>
Character coverage	character_coverage=1.0 (the nine Sindhi-specific letters can never be dropped)
Vocabulary sizes	32,000 · 64,000 (default) · 100,000 · 150,000
Training corpus	About 1.46 million sentences from four sources
Evaluation	18 candidate tokenizers (14 baselines + 4 sizes of this one), 4 registers, 5 metrics, no sampling
Fertility, 64k (mean of 4 registers)	1.147 tokens per word
Single-token words, 64k	88.6%
UNK rate	0.000%
Byte-fallback rate, 64k	0.138% of tokens
Round-trip character loss	0.0000% (all four sizes, all four registers)
Library	sentencepiece (load with SentencePieceProcessor)
Model license	CC BY-SA 4.0 (code in the GitHub repo: MIT)
Authors	Ali Nawaz (lead), Muqadas Fareed, Soonh Sadaf (Sadaf Khaliq); supervised by Attaullah Sahito, Shaikh Ayaz University, Shikarpur, Pakistan

Files in this repository

Names used in the paper map to file names as shown (CustomSindhi_64k = sindhi_custom_64000).

Paper name	Model file	Vocab file	Size	Notes
CustomSindhi_64k	sindhi_custom_64000.model	sindhi_custom_64000.vocab	(measured at upload) MB	Default / flagship. Smallest size with the best fertility and single-token retention in every register.
CustomSindhi_32k	sindhi_custom_32000.model	sindhi_custom_32000.vocab	(measured at upload) MB	Smallest embedding table. One exception: TinyLlama_Sindhi beats it on FLORES+ fertility (1.193 vs 1.233).
CustomSindhi_100k	sindhi_custom_100000.model	sindhi_custom_100000.vocab	(measured at upload) MB	Slightly better than 64k.
CustomSindhi_150k	sindhi_custom_150000.model	sindhi_custom_150000.vocab	(measured at upload) MB	Best fertility, largest embedding table.

How to use the Sindhi tokenizer

pip install sentencepiece huggingface_hub

from huggingface_hub import hf_hub_download
import sentencepiece as spm

# Download the recommended 64k model
model_path = hf_hub_download(
    repo_id="alinawazmahar/sindhi-tokenizer",
    filename="sindhi_custom_64000.model",
)
sp = spm.SentencePieceProcessor(model_file=model_path)

text = "سنڌ جي تاريخ تمام قديم آهي"
tokens = sp.encode(text, out_type=str)   # subword pieces
ids = sp.encode(text, out_type=int)      # token IDs

print(tokens)
print(ids)
print(sp.decode(ids))
print(sp.get_piece_size())               # 64000

Output of the code above, produced by the published 64k file at upload time (the ▁ symbol, U+2581, marks the start of a word):

tokens: (printed at upload time from your real 64k file)
ids:    (printed at upload time from your real 64k file)

Another example, a longer sentence (8 tokens for 8 words):

tokens: (printed at upload time from your real 64k file)
ids:    (printed at upload time from your real 64k file)

Other sizes: change filename to sindhi_custom_32000.model, sindhi_custom_100000.model or sindhi_custom_150000.model.

Using it for model training: set your embedding size to sp.get_piece_size(). The special IDs are <unk>=0, <s>=1, </s>=2, and the padding ID is -1 (a value of -1 means there is no pad token; add one yourself if your framework needs it).

Do not load these files with transformers.LlamaTokenizer. In this project it loaded the .model file without any error but reported a vocabulary of only 3 tokens (<unk>, <s>, </s>), while the file itself was correct. Always use sentencepiece.SentencePieceProcessor, or evaluation.metrics.SPMTokenizerAdapter from the GitHub repository.

What is the best tokenizer for Sindhi?

On this project's benchmark, Sindhi Tokenizer at 64k and above has the lowest fertility and the highest single-token retention of all 18 candidates in every one of the four registers, with zero UNK tokens and zero round-trip character loss. Competitors include XLM-RoBERTa, mT5, Qwen2.5, BLOOM, Gemma 3, Llama 3.2, SindhiLM v1 to v3, and four Sindhi-adapted tokenizers. The 32k size has one narrow exception (TinyLlama_Sindhi is better on FLORES+). Byte fallback is not a pure win: Gemma 3 averages 0.105% byte-fallback tokens versus 0.138% here, and tokenizers with no byte fallback (mBERT_Sindhi, SindhiLM_v3) report 0.000% by design but use <unk> instead. The advantage over the three strongest Sindhi-adapted competitors (TinyLlama_Sindhi, hellosindh_SindhiBERT_NER, mBERT_Sindhi_Kashif786) is real but modest; the advantage over general multilingual tokenizers is large. Benchmark scope: four registers (FLORES+, Wikipedia, Sentiment, Sindh Line news); no classical or literary register was available.

Headline benchmark (mean of four registers, unweighted)

Fertility is the average number of tokens per word (lower is better). Single-token % is the share of words kept as exactly one token (higher is better).

Tokenizer	Vocab size	Fertility ↓	Single-token % ↑	UNK % ↓	Byte-fallback % ↓
Sindhi Tokenizer 150k (CustomSindhi_150k)	150,000	1.125	89.7	0.000	0.154
Sindhi Tokenizer 100k (CustomSindhi_100k)	100,000	1.133	89.3	0.000	0.149
Sindhi Tokenizer 64k, default (CustomSindhi_64k)	64,000	1.147	88.6	0.000	0.138
Sindhi Tokenizer 32k (CustomSindhi_32k)	32,000	1.184	86.5	0.000	0.113
TinyLlama_Sindhi †	42,681	1.208	84.4	0.000 †	0.000 †
hellosindh_SindhiBERT_NER	32,001	1.218	83.7	0.194	0.197
mBERT_Sindhi_Kashif786	119,547	1.278	78.4	0.143	0.000
XLM-RoBERTa	250,002	1.512	65.4	0.034	1.402
SindhiLM_v3	32,000	1.730	55.8	3.605	0.000
FahadQazi_BPE ‡	13,031	1.919	39.1	0.000 ‡	7.127
hellosindh_SindhiBERT	32,000	1.960	35.5	0.122	0.114
mT5	250,100	2.280	21.7	0.022	12.746
Gemma 3	262,144	2.308	26.8	0.000	0.105
Qwen2.5 / SindhiLM_v2 §	151,643	2.580	23.8	0.000	8.665
BLOOM	250,680	2.627	24.7	0.000	31.660
Llama 3.2	128,000	2.896	18.2	0.000	28.472
SindhiLM_v1	43	4.306	0.1	14.017	0.000

† TinyLlama_Sindhi's zeros reflect silent character deletion, not robustness. ‡ FahadQazi_BPE silently deletes Latin letters in mixed Sindhi/English text. § SindhiLM_v2 is numerically identical to Qwen2.5 (a re-published wrapper of the Qwen2.5 tokenizer), so they share one row. The mean is unweighted: each register counts equally, because Sindh Line is about 2.5 times larger than the other three combined.

How much shorter are Sindhi sequences with this tokenizer?

Fewer tokens per word means shorter sequences, lower compute and more text per context window. Relative reduction in tokens for the same Sindhi text, 64k versus each baseline (computed from the fertility column):

Compared with	Fewer tokens	Words per 2,048-token window
Sindhi Tokenizer 64k	n/a	about 1,786
TinyLlama_Sindhi	5.0%	about 1,695
hellosindh_SindhiBERT_NER	5.8%	about 1,681
mBERT_Sindhi_Kashif786	10.3%	about 1,603
XLM-RoBERTa	24.1%	about 1,354
mT5	49.7%	about 898
Gemma 3	50.3%	about 887
Qwen2.5	55.5%	about 794
BLOOM	56.3%	about 780
Llama 3.2	60.4%	about 707

The gain is large against general multilingual tokenizers (roughly 50% to 60%) and small against the Sindhi-adapted ones (roughly 5% to 10%).

Per-register results for the four sizes

Tokenizer	Register	Fertility	Single-token %	UNK %	Byte-fallback %
150k	FLORES+	1.167	85.9	0.000	0.199
150k	Wikipedia	1.157	88.6	0.000	0.179
150k	Sentiment	1.099	91.0	0.000	0.137
150k	Sindh Line	1.078	93.3	0.000	0.100
100k	FLORES+	1.176	85.5	0.000	0.198
100k	Wikipedia	1.171	88.0	0.000	0.173
100k	Sentiment	1.103	90.8	0.000	0.133
100k	Sindh Line	1.084	93.0	0.000	0.091
64k	FLORES+	1.191	84.7	0.000	0.172
64k	Wikipedia	1.194	86.9	0.000	0.167
64k	Sentiment	1.110	90.3	0.000	0.126
64k	Sindh Line	1.093	92.4	0.000	0.085
32k	FLORES+	1.233	82.4	0.000	0.099
32k	Wikipedia	1.249	84.0	0.000	0.158
32k	Sentiment	1.130	89.0	0.000	0.126
32k	Sindh Line	1.124	90.5	0.000	0.068

Full per-register tables for all 18 candidates are in the paper (Appendix A).

Does the Sindhi tokenizer lose characters? Round-trip fidelity

No. Across all four sizes and all four registers, character loss after encode then decode is 0.0000%. Loss is measured after NFKC Unicode normalization and removal of zero-width and bidirectional control characters on both sides, so harmless normalization (collapsed whitespace, presentation-form letters converted to standard letters) is not counted as loss. Because SentencePiece normalizes text, decoded output is guaranteed equal to the input only in that normalized sense.

This check was added to the project after a competitor was found deleting characters without emitting <unk> or byte-fallback tokens, which UNK rate and byte-fallback rate cannot detect. Applying it to all 14 baselines found two previously undocumented defects:

• TinyLlama_Sindhi silently deletes the Sindhi letter ہ (U+06C1, ARABIC LETTER HEH GOAL) in every occurrence. It appears in common words such as تہ ("that/so"), بہ ("also") and نہ ("not"), in 24.8% of FLORES+ sentences and 6.2% of Wikipedia sentences. It also badly corrupts text written in the Arabic Presentation Forms Unicode block (worst case: a full sentence reduced to seven characters).
• FahadQazi_BPE silently deletes individual Latin letters inside code-switched text. For example, "An update on my situation, stay safe everyone" decodes with letters missing. Up to 16.15% of Wikipedia sentences are affected.

Tokenizer	FLORES+ loss % (sent. affected %)	Wikipedia	Sentiment	Sindh Line
Sindhi Tokenizer, all four sizes	0.0000 (0.00)	0.0000 (0.00)	0.0000 (0.00)	0.0000 (0.00)
Qwen2.5 / SindhiLM_v2, BLOOM, Gemma 3, Llama 3.2	0.0000 (0.00)	0.0000 (0.00)	0.0000 (0.00)	0.0000 (0.00)
XLM-RoBERTa, mT5	0.0000 (0.00)	0.0040 (0.04)	0.0000 (0.00)	0.0030 (0.02)
mBERT_Sindhi_Kashif786	0.0000 (0.00)	0.0021 (0.04)	0.0000 (0.00)	0.0001 (0.00)
hellosindh_SindhiBERT	0.0000 (0.00)	0.0089 (0.09)	0.0000 (0.00)	0.0031 (0.02)
SindhiLM_v1	0.0000 (0.00)	0.0790 (0.69)	0.0000 (0.00)	0.0005 (0.01)
SindhiLM_v3	0.0010 (0.10)	0.0754 (1.13)	0.0086 (0.31)	0.0029 (0.04)
hellosindh_SindhiBERT_NER	0.0021 (0.10)	0.0751 (0.49)	0.0017 (0.06)	0.0072 (0.11)
TinyLlama_Sindhi	0.4147 (24.90)	0.3114 (6.95)	0.0409 (1.67)	0.0413 (1.44)
FahadQazi_BPE	0.4105 (15.81)	0.8461 (16.15)	0.0847 (1.87)	0.2272 (6.08)

Frequently asked questions

What is a Sindhi tokenizer, and why does Sindhi need its own?

A tokenizer splits text into the units a language model reads. General multilingual tokenizers were not built around Sindhi's script: Sindhi extends the Arabic alphabet with letters for retroflex and implosive sounds that do not occur in Arabic, Urdu or Persian. Tokenizers that handle these letters poorly split Sindhi words into many pieces or into raw bytes, which inflates sequence length and compute. For example, Llama 3.2 produces 2.896 tokens per Sindhi word and BLOOM falls back to raw bytes for 31.7% of its tokens, while this tokenizer produces 1.147 tokens per word.

What is token fertility, and what is it for Sindhi?

Fertility is the mean number of tokens per whitespace-separated word. Lower is better. For Sindhi, this tokenizer's mean fertility is 1.147 at 64k (1.125 at 150k), against 1.512 for XLM-RoBERTa and 2.896 for Llama 3.2.

Which vocabulary size should I choose: 32k, 64k, 100k or 150k?

Use 64k unless you have a reason not to. It is the smallest size with the best fertility and single-token retention in every register. 32k has a smaller embedding table but loses to TinyLlama_Sindhi on FLORES+ fertility and single-token retention. 100k and 150k are only marginally better than 64k (fertility 1.133 and 1.125 versus 1.147) and have larger embedding tables.

How is it different from the XLM-RoBERTa, Llama 3.2, Qwen2.5 and BLOOM tokenizers for Sindhi?

It produces about 24.1% fewer tokens than XLM-RoBERTa, 60.4% fewer than Llama 3.2, 55.5% fewer than Qwen2.5 and 56.3% fewer than BLOOM on the benchmark. Byte-level fallback is also far lower: 0.138% of tokens, versus 28.5% for Llama 3.2, 31.7% for BLOOM and 8.7% for Qwen2.5. Gemma 3 is the only general-purpose tokenizer with comparably low byte fallback (0.105%), but its fertility is 2.308.

Does it ever output <unk>?

No. UNK rate is 0.000% in all four registers. Characters it has not seen are represented as raw bytes through byte fallback (0.138% of tokens on average at 64k).

Can I use it to train a Sindhi language model?

It is intended for that, but this has not yet been validated downstream. The repository contains a tokenizer only, with no model weights, and no language model, translation, classification or NER task has been trained with it so far. The measured benefit is shorter, lossless sequences. Intended uses are Sindhi LLM pretraining and fine-tuning, text classification, machine translation, named-entity recognition and information retrieval.

Why SentencePiece Unigram and not BPE?

Unigram selects the segmentation that maximizes sequence likelihood, which tends to give more linguistically sensible splits (a root and a suffix kept as separate pieces) in morphologically rich languages than greedy BPE merges. This was a design choice made at the start; no BPE-versus-Unigram experiment was run in this project.

Why does the training corpus include Urdu, Persian and Arabic?

Sindhi shares its script and many loanwords with these three languages. About 15% of the corpus (by character count) is Urdu, Persian and Arabic Wikipedia, to give the tokenizer exposure to shared loanwords. No ablation without this slice was run.

Which existing Sindhi tokenizers are there, and did you compare against them?

Yes. Fourteen were benchmarked: SindhiLM_v1, v2, v3, FahadQazi_BPE, mBERT_Sindhi_Kashif786, TinyLlama_Sindhi_jamalimubashirali, hellosindh_SindhiBERT, hellosindh_SindhiBERT_NER, plus six general-purpose tokenizers (XLM-RoBERTa, mT5, Qwen2.5, BLOOM, Gemma 3, Llama 3.2). Links are in Tokenizers compared.

Does it support Sindhi written in Devanagari?

No. This release targets Arabic-script Sindhi only.

Can I use it for speech recognition or text-to-speech?

It has not been validated for either. Speech recognition systems such as Whisper ship with their own fixed tokenizer built into the trained model, so a standalone tokenizer cannot be swapped in without retraining. Text-to-speech systems usually consume characters or phonemes rather than subword tokens, and no TTS architecture has been chosen.

What license is the Sindhi tokenizer under?

The model files are released under CC BY-SA 4.0; the code in the GitHub repository is MIT. See Data provenance and licensing for the licenses of the training sources.

How do I cite it?

See Citation.

How it was built

Training parameters

python training/train_tokenizer.py \
  --corpus corpus.txt \
  --vocab_size 64000 \
  --model_prefix models/sindhi_custom_64000

• model_type="unigram"
• character_coverage=1.0: every observed character is covered, so the nine Sindhi-specific letters (ٻ ڄ ڏ ڳ ڀ ٿ ڇ ٺ ڻ) are never dropped. This targets the main weakness of SindhiLM_v1, which has a 12% to 15% UNK rate on Sindhi text.
• byte_fallback=True: unseen characters become raw byte tokens <0xNN> instead of <unk>.
• Four vocabulary sizes (32,000, 64,000, 100,000, 150,000) were trained and benchmarked side by side.

Training corpus (about 1.46 million sentences)

Source	Volume	Notes
Sindhi Wikipedia (wikimedia/wikipedia, 20231101.sd)	6,000-article slice	Disjoint from the evaluation slice (same seeded shuffle; training starts after the evaluation cut-off). Sentences filtered to 3 to 100 words.
Sindhi-NLP (GitHub Sindhi-NLP/sindhi-NLP-dataset)	Four collections	Two Twitter accounts, Awami Awaz articles, Daily Kawish articles. About 98% of sentences are newspaper articles and about 2% are tweets.
shaikhsalman/sindhi-llm-corpus	20,000-row sample	11.3% of sampled sentences were mostly English; sentences with more than 30% Latin characters were removed and a re-check confirmed 0.0% remained.
Urdu + Persian + Arabic Wikipedia	About 15% of corpus characters	For shared loanwords. The Sindhi-side sources make up the other 85%.

Two ideas that were tested and rejected

Both came from advice for morphologically rich, clitic-heavy languages; both were implemented, measured, and removed.

1. Forced single-token postpositions and loanwords (via user_defined_symbols): the Unigram training objective value got worse (about 14.9 forced versus about 9.6 unforced) and the forced words took more tokens. All target words reach single-token status naturally without forcing.
2. Clitic pre-marking (inserting spaces around six common postpositions before training): apparent "fusion" rates of 17% to 39% for three of the six clitics were coincidental letter overlap with unrelated words, confirmed against a control character with a 0.2% baseline. Sindhi already writes these postpositions as separate words. The step was removed.

Evaluation data

Register	Source	Sentences	Words	Style
FLORES+ (snd_Arab, devtest)	openlanguagedata/flores_plus	1,012	25,356	Standardized translation benchmark
Wikipedia (formal)	wikimedia/wikipedia (20231101.sd), 3 to 100 words per sentence	25,480	435,947	Encyclopedic
Sentiment	alinawazmahar/Sindhi_Sentiment_dataset	3,536	47,156	Newspaper-derived sentiment text
Sindh Line Publishers	Mozilla Data Collective (2024 to 2025 Karachi Sindhi newspaper corpus, CC-BY-SA-4.0)	87,158	1,014,998	Contemporary news

All four registers were evaluated in full, with no sampling. They are disjoint from the training corpus, with one caveat: the Sentiment register and the Sindhi-NLP training source both draw on Kawish and Awami Awaz-adjacent news (different files, similar genre).

Metrics

• Fertility: mean tokens per word (lower is better).
• Single-token retention: percentage of words that stay one token (higher is better).
• UNK rate: percentage of output tokens that are the unknown token (lower is better).
• Byte-fallback rate: percentage of output tokens that, decoded individually, yield a UTF-8 replacement character (lower is better). This method works for every tokenizer family; an earlier pattern-matching version silently reported 0% for four tokenizers (Qwen2.5, BLOOM, Llama 3.2 and XLM-RoBERTa).
• Round-trip character fidelity: percentage of characters lost after encode then decode, after Unicode normalization.

Word-level test: 22 hand-picked Sindhi words

Genitive constructions, inflectional suffixes, Sindhi-specific letters, and loanwords were tested against all 13 non-duplicate baselines. Result: 21 ties, 0 outright wins, 1 loss (to FahadQazi_BPE and mBERT_Sindhi jointly, on the word هنجي). On curated, common words several competitors already match this tokenizer; its advantage shows up across full-register text, not on individual hand-picked words.

Category	Word	Best Sindhi Tokenizer (tokens)	Best competitor (tokens)	Result
Genitive	دنيا جو	32k (2)	2 (5 tokenizers)	Tie
Genitive	خدا جو	32k (2)	2 (6 tokenizers)	Tie
Genitive	پاڻي جي	32k (2)	2 (5 tokenizers)	Tie
Genitive	ماڻهن جو	32k (2)	2 (5 tokenizers)	Tie
Inflectional suffix	ڪتابن	32k (1)	1 (5 tokenizers)	Tie
Inflectional suffix	شهرن	32k (1)	1 (3 tokenizers)	Tie
Inflectional suffix	توهانڪي	64k (1)	1 (hellosindh_NER)	Tie
Inflectional suffix	منهنجو	32k (1)	1 (5 tokenizers)	Tie
Inflectional suffix	هنجي	32k (2)	1 (FahadQazi_BPE, mBERT_Sindhi)	FahadQazi_BPE
Sindhi-specific letter	ٻار	32k (1)	1 (5 tokenizers)	Tie
Sindhi-specific letter	ڄاڻ	32k (1)	1 (5 tokenizers)	Tie
Sindhi-specific letter	ڏينهن	32k (1)	1 (4 tokenizers)	Tie
Sindhi-specific letter	ڳوٺ	32k (1)	1 (7 tokenizers)	Tie
Sindhi-specific letter	ڀاءُ	32k (1)	1 (4 tokenizers)	Tie
Sindhi-specific letter	ٿورو	32k (1)	1 (5 tokenizers)	Tie
Sindhi-specific letter	ڇوڪرو	32k (1)	1 (4 tokenizers)	Tie
Loanword	علم	32k (1)	1 (8 tokenizers)	Tie
Loanword	تعليم	32k (1)	1 (9 tokenizers)	Tie
Loanword	دنيا	32k (1)	1 (6 tokenizers)	Tie
Loanword	اسڪول	32k (1)	1 (5 tokenizers)	Tie
Loanword	ڪمپيوٽر	32k (1)	1 (5 tokenizers)	Tie
Loanword	انٽرنيٽ	32k (1)	1 (5 tokenizers)	Tie

Intended use

• Tokenization for Sindhi language model pretraining and fine-tuning
• Text classification (for example sentiment analysis, topic modeling), machine translation, named-entity recognition, information retrieval
• Measuring and comparing tokenizer efficiency on Sindhi text

Limitations

• No classical or literary register. No evaluation on classical Sindhi such as Shah Jo Risalo. Five candidate sources failed: a corrupted CSV export, an unrelated SQL dump, two unusable OCR scans (one also non-commercial/no-derivatives licensed), and a platform behind a paid subscription.
• Partial overlap between one evaluation register and training data. The Sentiment register and the Sindhi-NLP training source share adjacent news provenance; full genre-level independence is not established.
• Modest margin over the strongest competitors. Against TinyLlama_Sindhi, hellosindh_SindhiBERT_NER and mBERT_Sindhi_Kashif786 the advantage is real but small, and the clean-sweep claim holds unconditionally only at 64k and above (32k loses to TinyLlama_Sindhi on FLORES+).
• Word-level test does not differentiate. On the 22 curated words, this tokenizer ties rather than wins; the advantage is aggregate.
• Competitor search may be incomplete. Four tokenizers were found after initial benchmarking; a later search found no more, but the space is active.
• Downstream use not validated. No model has been trained with it yet. Not validated for speech recognition (ASR) or text-to-speech (TTS).
• Register weighting. Headline numbers are an unweighted mean of four registers of very different size.
• Arabic-script Sindhi only.

Tokenizers compared

Tokenizer	Link
XLM-RoBERTa	https://huggingface.co/FacebookAI/xlm-roberta-base
mT5	https://huggingface.co/google/mt5-base
Qwen2.5	https://huggingface.co/Qwen/Qwen2.5-0.5B
BLOOM	https://huggingface.co/bigscience/bloom-560m
Gemma 3	https://huggingface.co/google/gemma-3-1b-pt
Llama 3.2	https://huggingface.co/meta-llama/Llama-3.2-1B
SindhiLM v1 / v2 / v3	https://huggingface.co/aakashMeghwar01/SindhiLM-Tokenizer-v1 · https://huggingface.co/aakashMeghwar01/SindhiLM-Tokenizer-v2 · https://huggingface.co/aakashMeghwar01/SindhiLM-Tokenizer-v3
FahadQazi_BPE	https://huggingface.co/fahadqazi/Sindhi-BPE-Tokenizer
mBERT_Sindhi_Kashif786	https://huggingface.co/Kashif786/mbert-sindhi-tokenizer
TinyLlama_Sindhi	https://huggingface.co/jamalimubashirali/sindhi-tinyllama-tokenizer
hellosindh_SindhiBERT	https://huggingface.co/hellosindh/sindhi-bert-base
hellosindh_SindhiBERT_NER	https://huggingface.co/hellosindh/sindhi-bert-ner

Data provenance and licensing

Source	Role	License status
Wikipedia (Sindhi, Urdu, Persian, Arabic)	Training	CC BY-SA
Sindhi-NLP (GitHub)	Training	Stated as MIT in the repository documentation; not independently verified
shaikhsalman/sindhi-llm-corpus	Training	No license stated; documented as research-use only
FLORES+	Evaluation only	See openlanguagedata/flores_plus
Sindh Line Publishers	Evaluation only	CC-BY-SA-4.0
Sindhi Sentiment dataset	Evaluation only	CC BY 4.0

The model files in this repository are released under CC BY-SA 4.0. Full provenance, volumes and licenses are in DATA_SOURCES.md.

Reproducing the results

The GitHub repository contains the training script (training/train_tokenizer.py), the metrics code (evaluation/metrics.py: fertility, single-token, UNK, byte-fallback and round-trip fidelity), the Kaggle notebooks (notebooks/training.ipynb, notebooks/benchmark.ipynb), the Gradio comparison demo (demo/app.py), the paper (docs/paper.md), and the technical explainer (docs/explainer.md, with every bug found and every rejected design).

Authors and acknowledgments

Developed at the Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan, by Ali Nawaz (lead developer and researcher), Muqadas Fareed and Soonh Sadaf (Sadaf Khaliq), under the supervision of Attaullah Sahito.

Citation

@misc{nawaz2026sindhitokenizer,
  author = {Nawaz, Ali and Fareed, Muqadas and Sadaf, Soonh and Sahito, Attaullah},
  title  = {A Corpus-Driven Sindhi Subword Tokenizer: Design, Empirical Validation, and Benchmark Comparison Against Fourteen Existing Tokenizers},
  year   = {2026},
  howpublished = {\url{https://huggingface.co/alinawazmahar/sindhi-tokenizer}},
  note   = {Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan. Code: https://github.com/alinawazcode/sindhi-tokenizer}
}

Keywords: Sindhi tokenizer, Sindhi NLP, Sindhi language model, Sindhi LLM, SentencePiece Unigram, subword tokenization, byte fallback, low-resource language, Perso-Arabic script, token fertility, tokenizer benchmark.