Does Reinforcement Learning Really Incentivize Reasoning Capacity in
  LLMs Beyond the Base Model?
Authors: name: Yang Yue, name: Zhiqi Chen, name: Rui Lu, name: Andrew Zhao, name: Zhaokai Wang, name: Yang Yue, name: Shiji Song, name: Gao Huang

Reinforcement Learning with Verifiable Rewards (RLVR) has recently
demonstrated notable success in enhancing the reasoning capabilities of LLMs,
particularly in mathematics and programming tasks. It is widely believed that
RLVR enables LLMs to continuously self-improve, thus acquiring novel reasoning
abilities that exceed corresponding base models' capacity. In this study,
however, we critically re-examines this assumption by measuring the
pass@k metric with large values of k to explore the reasoning
capability boundary of the models across a wide range of model families and
benchmarks. Surprisingly, the RL does not, in fact, elicit fundamentally
new reasoning patterns. While RL-trained models outperform their base models at
smaller values of k (\eg, k=1), base models can achieve a comparable or
even higher pass@k score compared to their RL counterparts at large k
values. The reasoning paths generated by RL-trained models are already included
in the base models' sampling distribution, suggesting that most reasoning
abilities manifested in RL-trained models are already obtained by base models.
Further analysis shows that RL training boosts the performance by biasing the
model's output distribution toward paths that are more likely to yield rewards,
therefore sampling correct responses more efficiently. But this also results in
a narrower reasoning capability boundary compared to base models. Similar
results are observed in visual reasoning tasks trained with RLVR. Moreover, we
find that distillation can genuinely introduce new knowledge into the model,
different from RLVR. These findings underscore a critical limitation of RLVR in
advancing LLM reasoning abilities which requires us to fundamentally rethink
the impact of RL training in reasoning LLMs and the need of a better paradigm.
Project Page: https://limit-of-RLVR.github.io