uzbek-bpe-16k
A byte-pair-encoding (BPE) tokenizer built specifically for Uzbek (Latin script). Off-the-shelf multilingual tokenizers split Uzbek text into too many pieces and mishandle the language's own characters — oʻ, gʻ, and the apostrophe. This tokenizer is trained on Uzbek text and treats those characters correctly, so it produces fewer tokens per word.
Fewer tokens per word (fertility 1.839) means cheaper training, longer effective context, and a model whose capacity goes to the language instead of to inefficient splitting. In the paired from-scratch model project, the tokenizer turned out to be the single highest-leverage decision.
- Author: Islombek Turdiyev (GitHub · website)
- Language: Uzbek (Latin script,
uzn_Latn) - Algorithm: byte-pair encoding (BPE)
- Vocabulary size: 16,384
- Fertility: 1.839 tokens/word
Why it exists
Multilingual tokenizers share one vocabulary across dozens of languages, so a low-resource language like Uzbek gets few merges and fragments into many subword tokens. Uzbek-specific characters (oʻ, gʻ) and apostrophes are frequently broken apart, which inflates sequence length and wastes model capacity. Training a dedicated tokenizer on Uzbek fixes this — the merges reflect real Uzbek morphology.
Intended use
Tokenizing Uzbek (Latin-script) text for language modeling, fine-tuning, and NLP experiments; as a drop-in tokenizer for training Uzbek models; and as a baseline for tokenizer / fertility studies on low-resource languages.
Paired model
Used to train IslombekT/uzbek-gpt-103m, a 103M-parameter Uzbek model trained from scratch that reaches 1.105 bits/byte — better than a 1.3B multilingual model adapted with QLoRA.
Citation
@misc{turdiyev2025uzbekbpe16k,
title = {uzbek-bpe-16k: a byte-pair-encoding tokenizer for Uzbek},
author = {Turdiyev, Islombek},
year = {2025},
howpublished = {\url{https://huggingface.co/IslombekT/uzbek-bpe-16k}}
}
Related
- Model — IslombekT/uzbek-gpt-103m
- Code & study — uzbek-gpt-from-scratch
- More of my work — turdiyevislombek.github.io · EduBoost