RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning
Authors: name: Hongzhi Zhang; name: Jia Fu; name: Jingyuan Zhang; name: Kai Fu; name: Qi Wang; name: Fuzheng Zhang; name: Guorui Zhou
Published: 2025-07-10 05:58:55

Abstract: Reinforcement learning (RL) for large language models is an energy-intensive
endeavor: training can be unstable, and the policy may gradually drift away
from its pretrained weights. We present RLEP\, -- \,Reinforcement
Learning with Experience rePlay\, -- \,a two-phase framework that first
collects verified trajectories and then replays them during subsequent
training. At every update step, the policy is optimized on mini-batches that
blend newly generated rollouts with these replayed successes. By replaying
high-quality examples, RLEP steers the model away from fruitless exploration,
focuses learning on promising reasoning paths, and delivers both faster
convergence and stronger final performance. On the Qwen2.5-Math-7B base model,
RLEP reaches baseline peak accuracy with substantially fewer updates and
ultimately surpasses it, improving accuracy on AIME-2024 from 38.2% to 39.9%,
on AIME-2025 from 19.8% to 22.3%, and on AMC-2023 from 77.0% to 82.2%. Our
code, datasets, and checkpoints are publicly available at
https://github.com/Kwai-Klear/RLEP to facilitate reproducibility and further
research.