On Predictability of Reinforcement Learning Dynamics for Large Language
  Models
Authors: name: Yuchen Cai; name: Ding Cao; name: Xin Xu; name: Zijun Yao; name: Yuqing Huang; name: Zhenyu Tan; name: Benyi Zhang; name: Guiquan Liu; name: Junfeng Fang
Published: 2025-10-01 06:13:50

Abstract: Recent advances in reasoning capabilities of large language models (LLMs) are
largely driven by reinforcement learning (RL), yet the underlying parameter
dynamics during RL training remain poorly understood. This work identifies two
fundamental properties of RL-induced parameter updates in LLMs: (1) Rank-1
Dominance, where the top singular subspace of the parameter update matrix
nearly fully determines reasoning improvements, recovering over 99\% of
performance gains; and (2) Rank-1 Linear Dynamics, where this dominant subspace
evolves linearly throughout training, enabling accurate prediction from early
checkpoints. Extensive experiments across 8 LLMs and 7 algorithms validate the
generalizability of these properties. More importantly, based on these
findings, we propose AlphaRL, a plug-in acceleration framework that
extrapolates the final parameter update using a short early training window,
achieving up to 2.5 speedup while retaining \textgreater 96\% of reasoning
performance without extra modules or hyperparameter tuning. This positions our
finding as a versatile and practical tool for large-scale RL, opening a path
toward principled, interpretable, and efficient training paradigm for LLMs.