MorphBPE tokenizer for Turkish (XNLI)
Faithful implementation of MorphBPE (Asgari et al., Findings of ACL 2026): a standard BPE tokenizer whose merges were constrained during training to never cross a morpheme boundary. Inference is identical to standard BPE on raw text — no morphological analyzer is required at test time.
| Parameter | Value |
|---|---|
| Vocabulary size | 21,623 |
| Special tokens | <s>, <pad>, </s>, <unk>, <mask> |
| Training data | facebook/xnli / tr / train split (785,404 sentences) |
| Normalization | NFC + Turkish lowercase (I→ı, İ→i, Lowercase) (built into the tokenizer) |
| Pre-tokenizer | Whitespace (built into the tokenizer) |
| Morpheme source (training only) | Morfessor Baseline (unsupervised) |
| Metric (validation+test) | MorphBPE | plain BPE |
|---|---|---|
| Fertility (ϕ) | 1.5519 | 1.2079 |
| Tokens/char | 0.2533 | 0.1971 |
| Morph. Edit Distance (µₑ, lower=better) | 0.7764 | 1.4362 |
| Morph. Consistency F1 (µ_c, higher=better) | 0.5460 | 0.1441 |
Usage
from transformers import PreTrainedTokenizerFast
tok = PreTrainedTokenizerFast.from_pretrained("NIRVLab/xnli-morphbpe-tr-32k")
tok.tokenize("Evlerimizden kitaplarımızı getirdik!") # raw text, no pre-segmentation
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support