On-Policy RL with Optimal Reward Baseline
Authors: name: Yaru Hao, name: Li Dong, name: Xun Wu, name: Shaohan Huang, name: Zewen Chi, name: Furu Wei

Reinforcement learning algorithms are fundamental to align large language
models with human preferences and to enhance their reasoning capabilities.
However, current reinforcement learning algorithms often suffer from training
instability due to loose on-policy constraints and computational inefficiency
due to auxiliary models. In this work, we propose On-Policy RL with Optimal
reward baseline (OPO), a novel and simplified reinforcement learning algorithm
designed to address these challenges. OPO emphasizes the importance of exact
on-policy training, which empirically stabilizes the training process and
enhances exploration. Moreover, OPO introduces the optimal reward baseline that
theoretically minimizes gradient variance. We evaluate OPO on mathematical
reasoning benchmarks. The results demonstrate its superior performance and
training stability without additional models or regularization terms.
Furthermore, OPO achieves lower policy shifts and higher output entropy,
encouraging more diverse and less repetitive responses. These results highlight
OPO as a promising direction for stable and effective reinforcement learning in
large language model alignment and reasoning tasks. The implementation is
provided at https://github.com/microsoft/LMOps/tree/main/opo.