NIRVLab/xnli-morphel-ru-32k-opt

NIRVLab — MorpheL Russian 32k Residual-Optimized v2

MorpheL is an MI-guided stochastic segmentation tokenizer for morphology-aware subword modeling.

This candidate preserves the original Russian MorpheL segmentation algorithm and optimizes only the residual vocabulary allocation.

Optimization

The original high-frequency vocabulary head is protected, while only the residual vocabulary tail is reranked using:

1. corpus frequency;
2. cross-word type productivity;
3. a bonus for direct atomic MorpheL pieces.

No alternative tokenizer algorithm is introduced.

Preserved MorpheL core

The following components remain unchanged:

• vowel–consonant plausible-boundary heuristic;
• global prefix/suffix MI computation;
• positive-MI candidate filtering;
• top-k candidate selection;
• Gumbel cut-selection mechanism;
• deterministic T=0 segmentation cache;
• recursive vocabulary fallback;
• dynamic-programming piece coalescing.

Training configuration

Parameter	Value
Candidate	ru_k4_residual_v2
Algorithm	MorpheL MI + Gumbel
Vocabulary size	32,000
top_k	4
Temperature	1.0
Vocabulary-induction temperature	0.0
MI threshold	0.0
Minimum frequency	2
Protected vocabulary ratio	90%
Productivity weight	0.55
Atomic-piece bonus	0.35
Corpus	facebook/xnli/ru
Corpus text fields	800,404
Core algorithm changed	No
Exact vocabulary budget	Yes

Residual vocabulary allocation

The protected head is selected using the original raw-frequency ranking.

Only the remaining vocabulary slots use the following residual score:

residual score =
    log(1 + corpus frequency)
  + productivity_weight * log(1 + type productivity)
  + atomic_piece_bonus * atomic_piece_indicator

An atomic piece is a direct piece produced by the deterministic MorpheL segmentation cache rather than only a longer merged span.

Tokenizer diagnostics

Metric	Value
Fertility ↓	1.4247
Tokens per character ↓	0.2071
Average sequence length ↓	19.23
Vocabulary coverage ↑	1.0000
OOV rate ↓	0.0000
Fallback-event rate ↓	0.1153
Character-shatter rate ↓	0.0000

Dataset

The tokenizer was induced from the Russian (ru) subset of facebook/xnli.

The tokenizer-training flow uses the same corpus splits as the original Russian MorpheL tokenizer notebook.

Native MorpheL artifacts

This repository contains:

• morphel_vocab.json
• mi_index.pkl
• segmentation_cache.pkl
• morphel_config.json
• residual_allocation_report.csv
• morphel_metrics_report.json

Downstream usage

Downstream Regime A, B, and C notebooks must load the native MorpheL artifacts rather than relying only on the generic WordLevel wrapper.

The raw-text pipeline is:

raw text
→ simple tokenization
→ deterministic MorpheL segmentation
→ recursive fallback
→ DP coalescing
→ token IDs

For Regime A, the intended tokenizer repository is:

NIRVLab/xnli-morphel-ru-32k-opt

Important: Intrinsic tokenizer metrics do not establish downstream superiority. The candidate must be evaluated using the fixed Russian Regime A/B/C protocol and the predetermined seeds.