Self-Play Preference Optimization for Language Model Alignment
Authors: name: Yue Wu; name: Zhiqing Sun; name: Huizhuo Yuan; name: Kaixuan Ji; name: Yiming Yang; name: Quanquan Gu
Published: 2024-05-01 17:59:20

Abstract: Traditional reinforcement learning from human feedback (RLHF) approaches
relying on parametric models like the Bradley-Terry model fall short in
capturing the intransitivity and irrationality in human preferences. Recent
advancements suggest that directly working with preference probabilities can
yield a more accurate reflection of human preferences, enabling more flexible
and accurate language model alignment. In this paper, we propose a
self-play-based method for language model alignment, which treats the problem
as a constant-sum two-player game aimed at identifying the Nash equilibrium
policy. Our approach, dubbed Self-Play Preference Optimization (SPPO),
approximates the Nash equilibrium through iterative policy updates and enjoys
theoretical convergence guarantee. Our method can effectively increase the
log-likelihood of the chosen response and decrease that of the rejected
response, which cannot be trivially achieved by symmetric pairwise loss such as
Direct Preference Optimization (DPO) and Identity Preference Optimization
(IPO). In our experiments, using only 60k prompts (without responses) from the
UltraFeedback dataset and without any prompt augmentation, by leveraging a
pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain
a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the
state-of-the-art length-controlled win-rate of 28.53% against GPT-4-Turbo on
AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench and
the Open LLM Leaderboard. Notably, the strong performance of SPPO is achieved
without additional external supervision (e.g., responses, preferences, etc.)
from GPT-4 or other stronger language models.