Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling
Authors: name: Hongzhi Huang; name: Defa Zhu; name: Banggu Wu; name: Yutao Zeng; name: Ya Wang; name: Qiyang Min; name: Xun Zhou
Published: 2025-01-28 14:15:42

Abstract: Tokenization is a fundamental component of large language models (LLMs), yet
its influence on model scaling and performance is not fully explored. In this
paper, we introduce Over-Tokenized Transformers, a novel framework that
decouples input and output vocabularies to improve language modeling
performance. Specifically, our approach scales up input vocabularies to
leverage multi-gram tokens. Through extensive experiments, we uncover a
log-linear relationship between input vocabulary size and training loss,
demonstrating that larger input vocabularies consistently enhance model
performance, regardless of model size. Using a large input vocabulary, we
achieve performance comparable to double-sized baselines with no additional
cost. Our findings highlight the importance of tokenization in scaling laws and
provide practical insight for tokenizer design, paving the way for more
efficient and powerful LLMs.