ExGRPO: Learning to Reason from Experience
Authors: name: Runzhe Zhan; name: Yafu Li; name: Zhi Wang; name: Xiaoye Qu; name: Dongrui Liu; name: Jing Shao; name: Derek F. Wong; name: Yu Cheng
Published: 2025-10-02 17:31:30

Abstract: Reinforcement learning from verifiable rewards (RLVR) is an emerging paradigm
for improving the reasoning ability of large language models. However, standard
on-policy training discards rollout experiences after a single update, leading
to computational inefficiency and instability. While prior work on RL has
highlighted the benefits of reusing past experience, the role of experience
characteristics in shaping learning dynamics of large reasoning models remains
underexplored. In this paper, we are the first to investigate what makes a
reasoning experience valuable and identify rollout correctness and entropy as
effective indicators of experience value. Based on these insights, we propose
ExGRPO (Experiential Group Relative Policy Optimization), a framework that
organizes and prioritizes valuable experiences, and employs a mixed-policy
objective to balance exploration with experience exploitation. Experiments on
five backbone models (1.5B-8B parameters) show that ExGRPO consistently
improves reasoning performance on mathematical/general benchmarks, with an
average gain of +3.5/7.6 points over on-policy RLVR. Moreover, ExGRPO
stabilizes training on both stronger and weaker models where on-policy methods
fail. These results highlight principled experience management as a key
ingredient for efficient and scalable RLVR.