NIRVLab — MorpheL Tokenizer for Turkish XNLI — ABLATION: Random-segmentation null baseline
This is NOT the real MorpheL algorithm. Every algorithmic component is disabled: no MI scoring (Eq.1-4), no vowel-consonant plausible-boundary pre-filter, no TopK candidate ranking (Eq.5), no Gumbel sampling (Eq.6-9). Boundary count/position come from a plain uniform RNG. Exists purely as a worst-case sanity control for MorpheL's own pipeline (Proposal: MorpheL: Morphology-Aware Tokenizer Adaptation for Pretrained Models in Low-Resource Languages) — if the real MI+Gumbel pipeline doesn't clearly beat this, the added complexity isn't earning its keep.
Trained on the Turkish (tr) subset of
facebook/xnli — all splits.
Algorithm (ablated)
Words are segmented by drawing a uniform-random number of cuts (0-2) and uniform-random cut positions — no MI, no vowel filter, no Gumbel, no learned signal of any kind. Segmentation is cached at induction time with a fixed seed for reproducibility, but carries no linguistic information.
Vocab induction also differs from the real MorpheL pipeline here: the shared span-recombination step (counting every contiguous merge of a word's pieces, e.g. [a,b,c] -> a,b,c,ab,bc,abc) was REMOVED for this ablation. That step lets a frequent whole word win a vocab slot regardless of how it was cut (the full-span "abc" always carries the word's true corpus frequency), which would silently undo the random segmentation and make this not a true know-nothing floor. Only the literal pieces produced by random cutting are counted here.
Training Config
| Parameter | Value |
|---|---|
| Algorithm | RANDOM (ablation — MI+Gumbel+vowel-filter all OFF) |
| Vocabulary size | 32,054 |
| random_max_cuts | 2 |
| seed | 42 |
| min_frequency | 2 |
| Special tokens | <s>, <pad>, </s>, <unk>, <mask> |
| Corpus | facebook/xnli/tr — all splits (800,404 sentences) |
| Vowel set | N/A — no plausible-boundary filter in this ablation |
Evaluation Metrics (vs Baselines, vocab_size=32000, same corpus)
| Metric | BPE | WordPiece | Unigram | MorpheL (full) | Random (this ablation) | |---|---|---|---|---| | Fertility ↓ | — | — | — | — | 3.0540 | | Tokens/char ↓ | — | — | — | — | 0.4077 | | Avg seq len ↓ | — | — | — | — | 33.83 | | Vocab coverage ↑ | — | — | — | — | 1.0000 | | OOV rate ↓ | — | — | — | — | 0.0000 |
Fill baseline columns after running baseline notebooks.
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("NIRVLab/xnli-morphel-tr-32k-ablation-random-baseline")
Note: this checkpoint uses RANDOM segmentation (ablation control) — do not use it for actual MorpheL comparisons other than as the worst-case sanity baseline. No temperature / Gumbel parameter applies here.