NeuroKurd/kurmanji_tokenizer

Kurmancî (Northern Kurdish) Tokenizer v4

A byte-level BPE tokenizer for Kurmancî (Kurmanji, Northern Kurdish, Latin script), built for generative language models.

Highlights

• Efficient on Kurmancî: 1.44 tokens per word on FLORES-200 (kmr_Latn), compared with 1.85 for XLM-R, 2.62 for Qwen2.5 and 2.91 for GPT-2 (see Evaluation).
• Lossless: decode(encode(text)) == text for every held-out line tested and for all edge cases (tabs, line breaks, repeated spaces, emoji, other scripts).
• No <unk>: Byte-level fallback means any input can be encoded.
• Robust to spelling variants: The training data includes text written without circumflexes (e i u instead of ê î û) and in capital letters.
• Digits are split one by one, and words with an apostrophe or hyphen are kept together by the pre-tokenizer before BPE merges are applied.

Quick start

from transformers import PreTrainedTokenizerFast

tok = PreTrainedTokenizerFast.from_pretrained("NeuroKurd/kurmanji_tokenizer")

text = "Ez ji hemû hevalên naskirî dûrtir im."
ids = tok.encode(text, add_special_tokens=False)
print(tok.convert_ids_to_tokens(ids))
print(tok.decode(ids) == text)   # True

Byte-level tokenizers display the space as Ġ and non-ASCII characters (such as ê, î, û) as byte symbols when you call convert_ids_to_tokens. To see readable pieces, decode token by token:

[tok.decode([i], clean_up_tokenization_spaces=False) for i in ids]

A word at the very beginning of a text has no leading space and is therefore a different token from the same word in the middle of a sentence. This is the same behaviour as in GPT-2 and similar tokenizers.

Details

Property	Value
Type	Byte-level BPE (Hugging Face tokenizers)
Vocabulary size	40,000
Normalization	Unicode NFC
Pre-tokenization	Regex split (words incl. internal ', ’, -; single digits; punctuation runs; whitespace), then byte-level mapping
Decoder	ByteLevel
Special tokens	<s> (bos), </s> (eos), <unk>, <pad>, <mask>
Reserved chat tokens	None yet (see Limitations)
Case	Case-sensitive

The pre-tokenizer regex is:

 ?[\p{L}\p{M}]+(?:['’\-][\p{L}\p{M}]+)*| ?\p{N}| ?[^\s\p{L}\p{M}\p{N}]+|\s+(?!\S)|\s+

To check the special token IDs of your copy:

print({t: tok.convert_tokens_to_ids(t) for t in ["<s>", "</s>", "<unk>", "<pad>", "<mask>"]})

Training data

The tokenizer was trained on text from the public dataset kurdish-tech/KurdishCorpus-clean (Hugging Face). Only Kurmancî was used. Sorani and Zazaki were excluded.

Processing steps:

1. Kept documents with dialect == "Kurmanji", lid_label == "kmr_Latn" and lid_prob >= 0.8.
2. Unicode NFC normalization.
3. Split documents into lines. Kept lines of 20 to 2,000 characters with more than 60 % letters and no Arabic-script characters.
4. Removed exact duplicate lines (case-insensitive).
5. About 1 % of documents (chosen by a hash of the first line) were set aside as a held-out set that was never used for training.
6. Added extra variants of some lines: about 12 % with ê î û replaced by e i u, about 4 % additionally with ç ş replaced by c s, and about 1.5 % in capital letters.

Result: about 1.2 million training lines (about 270 MB of text), about 10,000 held-out lines.

Important: The corpus was read as a stream and the run was stopped early. Only roughly the first 400,000 documents of the dataset were read, and about 125,000 of them passed the filters. The mix of sources in this sample was not balanced on purpose. A more balanced sample is planned for a later version.

The text sources have their own terms of use. Please check the dataset card of kurdish-tech/KurdishCorpus-clean and the licenses of the original sources before you build on this work for anything beyond the tokenizer itself. The tokenizer contains a vocabulary of word pieces, not the source texts.

Evaluation

Metric: Fertility = number of tokens divided by number of words, with words counted by splitting on whitespace and without special tokens. Lower is better.

FLORES-200 (kmr_Latn, dev + devtest, 2,009 sentences)

FLORES-200 was not used for training in any way.

Tokenizer	Vocabulary	Fertility
This tokenizer (v4)	40,000	1.44
XLM-R (xlm-roberta-base)	about 250,000	1.85
Qwen2.5 (Qwen/Qwen2.5-0.5B)	about 151,000	2.62
GPT-2 (gpt2)	50,257	2.91

This tokenizer was trained only on Kurmancî, while the others cover many languages. The table shows how efficiently each one splits Kurmancî. It does not show that one is better overall. Llama and Gemma tokenizers were not measured.

Held-out lines from the training dataset (10,229 lines)

Test	Result
Fertility, normal spelling	1.318
Fertility, same lines without circumflexes	1.366
Bytes per token	4.75
Round-trip identical	100 % (10,229 / 10,229)
<unk> tokens	0

The held-out lines come from the same sources as the training data, so these numbers are likely a bit optimistic compared with unseen text types. The FLORES result is the more reliable figure.

Edge cases (selection)

Input	Tokens
nexwendiyên	ne xwend iyên
nexwestibûnên	ne xwest ibûn ên
Ez li mala xwe dixebitim.	Ez ▁li ▁mala ▁xwe ▁dixebitim .
Ez ji hemu hevalen naskiri durtir im (no circumflex)	8 tokens for 7 words
2026	2 0 2 6
21.10.2026	single digits and dots

(▁ here stands for a leading space.)

Intended use

• Training or continuing to pre-train generative language models for Kurmancî.
• Text analysis and tokenization experiments for Kurmancî.
• A reference for research on tokenization of morphologically rich, low-resource languages.

Limitations

• Not suitable for other languages. Sorani (Arabic script), Zazaki; Turkish, German, English and others are encoded losslessly but inefficiently. Sorani costs about 10 tokens per word, and emoji cost several tokens each.
• Imbalanced training sample. The training data comes from a limited part of the dataset. Text types that are rare there (for example literature, spoken-style language, specialist vocabulary) may be split less efficiently.
• Some verb forms are split unevenly, for example dixwînim becomes di xw înim. The tokenizer follows frequency, not grammar, and does not know morphology.
• Capital letters are handled less efficiently than lowercase (ÇIYAYÊN becomes Ç IYA YÊN).
• Apostrophe spellings such as ç'ê or pe'rê are not always kept as single units, because BPE merges decide the final split.
• Possible personal data in the vocabulary has not been systematically checked yet. Web-derived vocabularies can contain names, e-mail fragments or similar strings.

Reproducing the tokenizer

The script train_tokenizer.py in this repository builds the corpus, trains the tokenizer and runs the tests (round-trip, fertility, edge cases). Settings used for this version: VOCAB_SIZE = 40000, min_frequency = 2, seed 42. Results can differ slightly between runs, because the dataset is read as a shuffled stream.

Changelog

• v4 (October 2026): Byte-level BPE, NFC instead of NFKC, lossless round-trip, spelling variants added to the training data. Replaces an earlier internal version that used a whitespace pre-tokenizer, which lost spaces and mapped unknown characters to <unk>.

License

This tokenizer is released under the Apache-2.0 license (see the header of this card). Please also respect the terms of the underlying data sources listed on the dataset card of kurdish-tech/KurdishCorpus-clean.

The FLORES-200 evaluation data is licensed separately (CC BY-SA 4.0) and is not part of this repository.

Acknowledgements

• kurdish-tech/KurdishCorpus-clean for the text data.
• FLORES-200 (NLLB Team et al., Meta AI) for the evaluation sentences.
• Everyone who writes, translates and teaches Kurmancî.

Citation

If you use this tokenizer, please link to this repository. A formal citation will be added with the public release.

Contact

Open a discussion in the Community tab of this repository.