Reinforcement Learning for Reasoning in Large Language Models with One
  Training Example
Authors: name: Yiping Wang; name: Qing Yang; name: Zhiyuan Zeng; name: Liliang Ren; name: Lucas Liu; name: Baolin Peng; name: Hao Cheng; name: Xuehai He; name: Kuan Wang; name: Jianfeng Gao; name: Weizhu Chen; name: Shuohang Wang; name: Simon Shaolei Du; name: Yelong Shen
Published: 2025-04-29 09:24:30

Abstract: We show that reinforcement learning with verifiable reward using one training
example (1-shot RLVR) is effective in incentivizing the math reasoning
capabilities of large language models (LLMs). Applying RLVR to the base model
Qwen2.5-Math-1.5B, we identify a single example that elevates model performance
on MATH500 from 36.0% to 73.6%, and improves the average performance across six
common mathematical reasoning benchmarks from 17.6% to 35.7%. This result
matches the performance obtained using the 1.2k DeepScaleR subset (MATH500:
73.6%, average: 35.9%), which includes the aforementioned example. Similar
substantial improvements are observed across various models (Qwen2.5-Math-7B,
Llama3.2-3B-Instruct, DeepSeek-R1-Distill-Qwen-1.5B), RL algorithms (GRPO and
PPO), and different math examples (many of which yield approximately 30% or
greater improvement on MATH500 when employed as a single training example). In
addition, we identify some interesting phenomena during 1-shot RLVR, including
cross-domain generalization, increased frequency of self-reflection, and
sustained test performance improvement even after the training accuracy has
saturated, a phenomenon we term post-saturation generalization. Moreover, we
verify that the effectiveness of 1-shot RLVR primarily arises from the policy
gradient loss, distinguishing it from the "grokking" phenomenon. We also show
the critical role of promoting exploration (e.g., by adding entropy loss with
an appropriate coefficient) in 1-shot RLVR training. As a bonus, we observe
that applying entropy loss alone, without any outcome reward, significantly
enhances Qwen2.5-Math-1.5B's performance on MATH500 by 27.4%. These findings
can inspire future work on RLVR data efficiency and encourage a re-examination
of both recent progress and the underlying mechanisms in RLVR. Our code, model,
and data are open source at https://github.com/ypwang61/One-Shot-RLVR