RL-PLUS: Countering Capability Boundary Collapse of LLMs in
  Reinforcement Learning with Hybrid-policy Optimization
Authors: name: Yihong Dong, name: Xue Jiang, name: Yongding Tao, name: Huanyu Liu, name: Kechi Zhang, name: Lili Mou, name: Rongyu Cao, name: Yingwei Ma, name: Jue Chen, name: Binhua Li, name: Zhi Jin, name: Fei Huang, name: Yongbin Li, name: Ge Li

Reinforcement Learning with Verifiable Reward (RLVR) has significantly
advanced the complex reasoning abilities of Large Language Models (LLMs).
However, it struggles to break through the inherent capability boundaries of
the base LLM, due to its essentially on-policy strategy coupled with LLM's
immense action space and sparse reward. Critically, RLVR can lead to the
capability boundary collapse, narrowing the LLM's problem-solving scope. To
address this problem, we propose RL-PLUS, a novel hybrid-policy optimization
approach for LLMs that synergizes internal exploitation with external data to
achieve stronger reasoning capabilities and surpass the boundaries of base
models. RL-PLUS integrates two core components, i.e., Multiple Importance
Sampling to address distributional mismatch from external data, and
Exploration-Based Advantage Function to guide the model towards high-value,
unexplored reasoning paths. We provide both theoretical analysis and extensive
experiments to demonstrate the superiority and generalizability of our
approach. Compared with existing RLVR methods, RL-PLUS achieves 1)
state-of-the-art performance on six math reasoning benchmarks; 2) superior
performance on six out-of-distribution reasoning tasks; 3) consistent and
significant gains across diverse model families, with average relative
improvements up to 69.2\%. Moreover, the analysis of Pass@k curves indicates
that RL-PLUS effectively resolves the capability boundary collapse problem.