Low-probability Tokens Sustain Exploration in Reinforcement Learning
  with Verifiable Reward
Authors: name: Guanhua Huang; name: Tingqiang Xu; name: Mingze Wang; name: Qi Yi; name: Xue Gong; name: Siheng Li; name: Ruibin Xiong; name: Kejiao Li; name: Yuhao Jiang; name: Bo Zhou
Published: 2025-10-03 17:56:13

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has propelled Large
Language Models in complex reasoning, yet its scalability is often hindered by
a training bottleneck where performance plateaus as policy entropy collapses,
signaling a loss of exploration. Previous methods typically address this by
maintaining high policy entropy, yet the precise mechanisms that govern
meaningful exploration have remained underexplored. Our analysis suggests that
an unselective focus on entropy risks amplifying irrelevant tokens and
destabilizing training. This paper investigates the exploration dynamics within
RLVR and identifies a key issue: the gradual elimination of valuable
low-probability exploratory tokens, which we term \textit{reasoning
sparks}. We find that while abundant in pre-trained models, these sparks are
systematically extinguished during RLVR due to over-penalization, leading to a
degeneracy in exploration. To address this, we introduce Low-probability
Regularization (Lp-Reg). Its core mechanism regularizes the policy towards a
heuristic proxy distribution. This proxy is constructed by filtering out
presumed noise tokens and re-normalizing the distribution over the remaining
candidates. The result is a less-noisy proxy where the probability of
reasoning sparks is amplified, which then serves as a soft
regularization target to shield these valuable tokens from elimination via KL
divergence. Experiments show that Lp-Reg enables stable on-policy training for
around 1,000 steps, a regime where baseline entropy-control methods collapse.
This sustained exploration leads to state-of-the-art performance, achieving a
60.17% average accuracy on five math benchmarks, an improvement of 2.66%
over prior methods. Code is available at https://github.com/CarlanLark/Lp-Reg.