Predicting the Order of Upcoming Tokens Improves Language Modeling
Authors: name: Zayd M. K. Zuhri, name: Erland Hilman Fuadi, name: Alham Fikri Aji

Multi-Token Prediction (MTP) has been proposed as an auxiliary objective to
improve next-token prediction (NTP) in language model training but shows
inconsistent improvements, underperforming in standard NLP benchmarks. We argue
that MTP's exact future token prediction is too difficult as an auxiliary loss.
Instead, we propose Token Order Prediction (TOP), which trains models to order
upcoming tokens by their proximity using a learning-to-rank loss. TOP requires
only a single additional unembedding layer compared to MTP's multiple
transformer layers. We pretrain models of 340M, 1.8B, and 7B parameters using
NTP, MTP, and TOP objectives. Results on eight standard NLP benchmarks show
that TOP overall outperforms both NTP and MTP even at scale. Our code is
available at https://github.com/zaydzuhri/token-order-prediction