Angles Don't Lie: Unlocking Training-Efficient RL Through the Model's
  Own Signals
Authors: name: Qinsi Wang; name: Jinghan Ke; name: Hancheng Ye; name: Yueqian Lin; name: Yuzhe Fu; name: Jianyi Zhang; name: Kurt Keutzer; name: Chenfeng Xu; name: Yiran Chen
Published: 2025-06-02 21:40:38

Abstract: Current Reinforcement Fine-tuning (RFT) paradigms for Large Language Models
(LLMs) suffer from sample inefficiency due to the redundant exposure of
identical queries under uniform data sampling. While previous work has explored
curriculum learning via heuristic difficulty metrics, these strategies exhibit
limitations by neglecting the intrinsic learning signals generated by the model
itself, thus leading to suboptimal training regimes. In this paper, we identify
a model-inherent signal termed angle concentration that effectively reflects an
LLM's capacity to learn from specific data. We theoretically and empirically
demonstrate a correlation between the angular distribution of token hidden
state vectors and the resulting gradient, revealing a learning preference for
data exhibiting higher angle concentration. Inspired by this finding, we
propose GAIN-RL, a Gradient-driven Angle-Informed Navigated RL framework. By
leveraging the model's intrinsic angle concentration signal, GAIN-RL
dynamically selects training data in each epoch, ensuring consistently
impactful gradient updates and thus significantly enhancing overall training
efficiency. Empirical evaluations show that GAIN-RL (GRPO) achieves over a 2.5x
acceleration in training efficiency across diverse mathematical and coding
tasks and varying model scales. Furthermore, GAIN-RL (GRPO)'s efficient
sampling yields data-efficient training, achieving better performance with half
the original data compared to vanilla GRPO with full training data. Code is
realsed at https://github.com/wangqinsi1/GAINRL/tree/main.