Accelerated Preference Optimization for Large Language Model Alignment
Authors: name: Jiafan He; name: Huizhuo Yuan; name: Quanquan Gu
Published: 2024-10-08 18:51:01

Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a pivotal
tool for aligning large language models (LLMs) with human preferences. Direct
Preference Optimization (DPO), one of the most popular approaches, formulates
RLHF as a policy optimization problem without explicitly estimating the reward
function. It overcomes the stability and efficiency issues of two-step
approaches, which typically involve first estimating the reward function and
then optimizing the policy via proximal policy optimization (PPO). Since RLHF
is essentially an optimization problem, and it is well-known that momentum
techniques can accelerate optimization both theoretically and empirically, a
natural question arises: Can RLHF be accelerated by momentum? This paper
answers this question in the affirmative. In detail, we first show that the
iterative preference optimization method can be viewed as a proximal point
method. Based on this observation, we propose a general Accelerated Preference
Optimization (APO) framework, which unifies many existing preference
optimization algorithms and employs Nesterov's momentum technique to speed up
the alignment of LLMs. Theoretically, we demonstrate that APO can achieve a
faster convergence rate than the standard iterative preference optimization
methods, including DPO and Self-Play Preference Optimization (SPPO).
Empirically, we show the superiority of APO over DPO, iterative DPO, and other
strong baselines for RLHF on the AlpacaEval 2.0 benchmark.