Reinforcement Mid-Training
Authors: name: Yijun Tian, name: Shaoyu Chen, name: Zhichao Xu, name: Yawei Wang, name: Jinhe Bi, name: Peng Han, name: Wei Wang

The development of state-of-the-art large language models is commonly
understood as a two-stage process involving pre-training and post-training. We
point out the need for an additional intermediate stage called reinforcement
mid-training with potential for strong performance gains. In this paper, we
formally define the problem and identify three key challenges: (1) inefficient
training due to excessive reasoning steps, (2) disregard of the imbalanced
token entropy distribution, and (3) underutilization of token information. To
address these challenges, we propose RMT, a framework for efficient, adaptive,
and unified reinforcement mid-training with various innovative components. In
particular, we first introduce a dynamic token budget mechanism that constrains
unnecessary reasoning steps and mitigates model overthinking. Next, we design a
curriculum-based adaptive sampling method that fosters a progressive learning
trajectory from easy to hard tokens. Finally, we present a dual training
strategy that combines reinforcement learning with next-token prediction,
ensuring targeted learning on key tokens and full exploitation of all token
information. Extensive experiments demonstrate the superiority of RMT over
state-of-the-art methods, achieving up to +64.91% performance improvement with
only 21% of the reasoning length in language modeling. We also show that
checkpoints obtained after reinforcement mid-training can benefit the
subsequent post-training, yielding up to +18.76% improvement in the
mathematical domain.