Title: IndicSuperTokenizer: An Optimized Tokenizer for Indic Multilingual LLMs
Primary area: foundation or frontier models, including LLMs
Keywords: Tokenizer, Multilingual LLMs, Multilingual Tokenizer, Indic, Tokenization Efficiency, LLMs, Indic Tokenizer, Indian Languages, BPE

Abstract: Tokenizers play a crucial role in determining the performance, training efficiency, and the inference cost of Large Language Models (LLMs). Designing effective tokenizers for multilingual LLMs is particularly challenging due to diverse scripts and rich morphological variation. While subword methods such as Byte Pair Encoding (BPE) are widely adopted, their effectiveness in multilingual settings remains underexplored. We present IndicSuperTokenizer, a tokenizer for Indic multilingual LLMs, that combines both subword and multi-word tokenization, along with language-specific pre-tokenization, leading to more linguistically aligned tokens and achieving a new state-of-the-art in fertility score. Evaluated across English, $22$ Indian languages and code data, our tokenizer improves the average fertility score by $39.5$% over LLaMA4 and by $18$% over Sutra (the current best). This translates to $44$% improvement in inference throughput over LLaMA4 while maintaining comparable performance on English and Indic benchmarks. We also present detailed ablations across tokenizer training data size, vocabulary size, merging techniques, and pre-tokenization strategies, demonstrating the robustness of our design choices.

TL;DR: We present IndicSuperTokenizer, a tokenizer for Indic multilingual LLMs, that combines both subword and multi-word tokenization along with language-specific pre-tokenization, achieving a new state-of-the-art in fertility rate.