Data-Efficient RLVR via Off-Policy Influence Guidance
Authors: name: Erle Zhu, name: Dazhi Jiang, name: Yuan Wang, name: Xujun Li, name: Jiale Cheng, name: Yuxian Gu, name: Yilin Niu, name: Aohan Zeng, name: Jie Tang, name: Minlie Huang, name: Hongning Wang

Data selection is a critical aspect of Reinforcement Learning with Verifiable
Rewards (RLVR) for enhancing the reasoning capabilities of large language
models (LLMs). Current data selection methods are largely heuristic-based,
lacking theoretical guarantees and generalizability. This work proposes a
theoretically-grounded approach using influence functions to estimate the
contribution of each data point to the learning objective. To overcome the
prohibitive computational cost of policy rollouts required for online influence
estimation, we introduce an off-policy influence estimation method that
efficiently approximates data influence using pre-collected offline
trajectories. Furthermore, to manage the high-dimensional gradients of LLMs, we
employ sparse random projection to reduce dimensionality and improve storage
and computation efficiency. Leveraging these techniques, we develop
Curriculum RL with Off-Policy
Influence guidance (CROPI), a multi-stage RL framework that
iteratively selects the most influential data for the current policy.
Experiments on models up to 7B parameters demonstrate that CROPI significantly
accelerates training. On a 1.5B model, it achieves a 2.66x step-level
acceleration while using only 10\% of the data per stage compared to
full-dataset training. Our results highlight the substantial potential of
influence-based data selection for efficient RLVR.