BOW: Bottlenecked Next Word Exploration
Authors: name: Ming Shen, name: Zhikun Xu, name: Xiao Ye, name: Jacob Dineen, name: Ben Zhou

Large language models (LLMs) are typically trained via next-word prediction
(NWP), which provides strong surface-level fluency but often lacks support for
robust reasoning. We propose BOttlenecked next Word exploration (BOW), a novel
RL framework that rethinks NWP by introducing a reasoning bottleneck where a
policy model first generates a reasoning path rather than predicting the next
token directly, after which a frozen judge model predicts the next token
distribution based solely on this reasoning path. We train the policy model
using GRPO with rewards that quantify how effectively the reasoning path
facilitates next-word recovery. Compared with other continual pretraining
baselines, we show that BOW improves both the general and next-word reasoning
capabilities of the base model, evaluated on various benchmarks. Our findings
show that BOW can serve as an effective and scalable alternative to vanilla
NWP.