NIRVLab/xnli-morphel-sw-32k-beta

MorpheL Tokenizer for Swahili XNLI

MorpheL MI-guided tokenizer trained on facebook/xnli subset sw, using all train/validation/test premise and hypothesis text under the same corpus protocol as the project baselines.

Configuration

• Vocabulary budget: 32,000
• Actual vocabulary size: 32,000
• top_k: 4
• Downstream stochastic temperature: 1.0
• Vocabulary induction/evaluation temperature: 0.0
• MI threshold: 0.0
• Minimum frequency: 2
• Swahili vowel bases: a, e, i, o, u
• Unicode vowel handling: NFD base-letter classification
• Protected digraphs: ch, dh, gh, kh, ng, ny, sh, th

Native artifacts

Downstream notebooks must load morphel_vocab.json, mi_index.pkl, segmentation_cache.pkl, and morphel_config.json; tokenizer.json alone is only a WordLevel wrapper.

Evaluation

{
  "fertility": 1.259849551558451,
  "tokens_per_char": 0.232687970096,
  "avg_seq_len": 17.329933333333333,
  "vocab_coverage": 1.0,
  "oov_rate": 0.0,
  "fallback_event_rate": 0.1561573736720706,
  "char_shatter_rate": 0.0,
  "n_sentences": 15000,
  "n_word_types": 18059
}