Self-Play Preference Optimization for Language Model Alignment
Authors: Yue Wu
Zhiqing Sun
Huizhuo Yuan
Kaixuan Ji
Yiming Yang
Quanquan Gu
Venue: ICLR 2025 Poster (2025)
Keywords: self play
preference optimization
large language model
RLHF
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose a self-play algorithm (SPPO) with strong theoretical motivations and guarantees, achieving state-of-the-art results on key benchmarks without external supervision.

Abstract: Standard reinforcement learning from human feedback (RLHF) approaches relying on parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements suggest that directly working with preference probabilities can yield a more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose a self-play-based method for language model alignment, which treats the problem as a constant-sum two-player game aimed at identifying the Nash equilibrium policy. Our approach, dubbed *Self-Play Preference Optimization* (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium. 
Additionally, we propose a new SPPO objective which is both strongly motivated by theory and is simple and effective in practice.
In our experiments, using only 60k prompts (without responses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain a model from fine-tuning Mistral-7B-Instruct-v0.2 that achieves the state-of-the-art length-controlled win-rate of 28.53\% against GPT-4-Turbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO on MT-Bench, Arena-Hard, and the Open LLM Leaderboard.
Starting from a stronger base model Llama-3-8B-Instruct, we are able to achieve a length-controlled win rate of 38.77\%.
Notably, the strong performance of SPPO is achieved without additional external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models.

Published as a conference paper at ICLR 2025
SELF-PLAY PREFERENCE OPTIMIZATION FOR LAN- GUAGE MODEL ALIGNMENT
Yue Wu' Zhiqing Sun Huizhuo Yuan" '", Kaixuan Ji', Yiming Yang", Quanquan Gu
'Department of Computer Science. University of California, Los Angeles
Language Technologies Institute, Carnegie Mellon University {ywu, hzyuan, ggu}ecs wwl.. edu, (odu,(zhor {zhiqings, perlinng,yiiinnnn yiming)ecs cmu, edu
ABSTRACT
Standard reinforcement learning from human feedback (RLHF) approaches rely- ing parametric models like the Bradley-Terry model fall short in capturing the intransitivity and irrationality in human preferences. Recent advancements sug- gest that directly working with preference probabilities can yield more accurate reflection of human preferences, enabling more flexible and accurate language model alignment. In this paper, we propose self-play- guage model alignment, which treats the problem as a constant-sum two-player game aimed identifying the Nash equilibrium policy. Our approach, dubbed Self-Play Preference Optimization (SPPO), utilizes iterative policy updates to provably approximate the Nash equilibrium. Additionally, we propose new SPPO objective which is both strongly motivated by theory and is simple and effective practice.
method for lan-
our experiments, using only 60k prompts (without re- sponses) from the UltraFeedback dataset and without any prompt augmentation, by leveraging a pre- pre-trained preference model PairRM with only 0.4B parameters, SPPO can obtain model from fine-tuning Mistral-7B Baaal------nntutt--tt Instruct-vO.2 that achieves the state-of offte--f-th-arr contthlco win-rate of 28.53% against GPT-4-TuB Turrbo on AlpacaEval 2.0. It also outperforms the (iterative) DPO and IPO MT-Bench, Arena-Hain Hard, and the Open LLM Leaderboard. Starting from a stronger base model Llama-3-8B-Instre
we able achieve a length-controlled win rate of 38%.7%. Notably, the strong performance of SPPO is achieved without addi- tional external supervision responses, preferences, etc.) from GPT-4 or other stronger language models.
INTRODUCTION
Large Language Models (LLMs) (e.g.. Ou..ng al. 2022 OpenAl et al. 2023). have shown re- markable capabilities in producing human-like text, fielding questions, and coding. Despite their advancements, these models encounter challenges tasks requiring high levels of reliability, safety, and ethical alignment. To address these challenges, Reinforcement Learning from Human Feedback (RLHF), also known as Preference- Reinforcement Learning (PhRL), presents a promising so- lution. This framework for policy optimization, highlighted in works by Christiano et al. (2017) and recently in Ouyang et al. 22022), has led to significant empirical success in fine-tuning instruction- following LLMs, making them more aligned with human preferences and thus more helpful.
Most existing approaches to RLHF rely on either explicit or implicit reward models. Taking In- structGPT (Ouyang et al. 2022) as an example, a reference policy Tet is first established, typically instruction-
(supervised) fine-tuning. An explicit reward func- tion is obtained by training a reward model based on human preference feedback data, employing the Bradley-Terry (BT) model (Bradley & Terry 1952). Subsequently, reinforcement learning al- gorithms such as Proximal Policy Optimization (Schulman et al., 2017] PPO) are used to fine-tune the reference LLM Thef by maximizing the reward function. The reward model provides "reward score" r(y; x) for the given response and prompt x, approximately reflecting how humans value these responses. More recently, methods like Direct Preference Optimization (Rafailov et al., 2202bb] DPO) have been introduced. These methods forgo the training of a separate reward model but still
from supervised pre- pre-training
"Equal Contribution
1 Corresponning Author

---

Published as a conference paper at ICLR 2025
fundamentally adhere to the reward maximization objective and are determined by parametric mod- els such as the BT model.
These models presuppose a monotonous and transitive relationship among preferences for different choices. However, empirical evidence suggests otherwise. For instance. Tversky (1969) observed human decisions can be influenced by different factors and exhibit inconsistency. Such observations indicate that human preferences do not always adhere to single, value-based hierarchy and can even appear irrational, such as exhibiting loops in preference relations. For LLMs, another motivating evidence is that Munos al. (2023) has empirically shown that directly predicting the pairwise preference can achieve higher accuracy than predicting the preference via BTT-based reward model. To address the inconsistency in human preference, researchers have proposed to work directly with the preference probability and design algorithms that can more flexibly represent human preferences (Luu et al. 2022 Wu et al.) 2023) in the ranking or bandit setting. Recently, an emerging line of work @Wang al. [2024] Munos et al. [023] Swamy et al., (024)) also proposed to study RLHF for LLMs under such general preference Pyyy > y' xx), where y and y' are two different responses and is prompt. The goal is to identify the Nash equilibrium or von Neumann winner of the two-player
constant-sum game
(TT" x²) arg max min Ex-x () (Ply > y
where each player is an LLM that outputs responses and aims to maximize its probability of being preferred over its opponent.
Independent from our work, Swamy et al. (2024) proposed Self-play Preference Optimization ( I for the same (unregularized) two-player constant-sum game. They provide a general re- duction of preference optimization no-regret online learning for the multi-step Markov Decision Process. When constrained to the bandit setting for LLMs, their proposed algorithmic framework reduces the famous Hedge algorithm (Freund & Schapire 1997). which admits the exponential update rule as described in (3.I. To approximately solve the exponential update, [Swamy et al. (202)) then proposed to employ typical policy optimization algorithms such Proximal Policy Optimization (PPO) (Schulman et al. (017) or Soft Actor-Critic (SAC) (Haarnoja et al.) 2018) to maximize the win rate against the reference policy and evaluated the performance of their self-play algorithms in robotic and game tasks. However, it typically requires more effort to apply PPO or SAC to large-scale fine-tuning of LLM and make them work stably. Therefore, it remains unclear how their self-play framework can be applied to large-scale language model alignment efficiently. In this paper. motivated by these developments mentioned above, we propose a new self-play algo- rithm that (1) enjoys provable guarantees solve the two-player constant-sum game; and (2) can scale up to large-scale efficient fine-tuning of large language models. In detail, we formulate the RLHF problem as a constant-sum two-player game. Our objective to identify the Nash equilib- rium policy, which consistently provides preferred responses over any other policy average. To identify the Nash equilibrium policy approximately, we adopt the classic online adaptive algorithm with multiplicative weights (Freund & Schapire 1999) high-level framework that solves the two-player game. Further, each step of the high-level framework can be approximated by a self-play mechanism, where in each iteration the policy is playing against itself in the previous iteration by fine-tuning it on synthetic data that are generated by the policy and annotated by the preference model.
Our contributions are highlighted follows:
Starting from the exponential weight update algorithm which provably converges to the Nash equi- librium of the two-player constant-sum game, we propose the Self-Play Preference Optimization (SPPO) algorithm for large language model alignment. The algorithm converges to an approxi- mate Nash equilibrium provably and admits simple form of loss function for easy optimization.
Unlike the symmetric pairwise loss such DPO and Identity Preference Optimization (IPO) (Azar et al. 2023). we propose a new optimization objective that does not rely on pairwise comparisons. The new loss objective (3.4), initially driven by game-theoretical concepts, turns out strongly motivated by the policy gradient theory and implicitly encourages the LLM to learn token-level optimal value function.
'The SPO framework does not pertain to the efficient fine-tuning of LLMs. Our Self- Self-Play Preference Optimization (SPPO) focuses on L.M alignment and was developed independently. To distinguish it from the SPO framework, use the abhreviation SPPO.
2

---

Published as a conference paper at ICLR 2025
Empirically, SPPO significantly enhances the well-aligned 3-8B-Instruct model, achieving an increase of over 11% on the length-controlled win rate against GPT-4-Turbo on the AlpacaEval 2.0 (Dubois et al.] 2024a) test set. Additionally, SPPO exhibits strong generalist abilities across different tasks, including MT-Bench, the Open L.M Leader- board, and the more recent, more challenging benchmark, Arena-Hard. Unlike iterative DPO/IPO, which tends to show performance decay on other benchmarks when optimized towards the PairRM score, SPPO's performance gain is consistent. Notably, all the strong performances are achieved without external supervision (e.g., responses, preferences, etc.) from GPT-4 or other stronger language models.
-nni---t--nttuut-0...... and Llama-
Concurrent to work, several studies, including Direct Nash Optimization (Rosset et al. 2024 and REBEL (Gao et al. 2024) have also explored using either cross-entropy loss or square loss minimization approximate the exponential update. Specifically, they used the same trick proposed in DPO (Rafailov et al. 2024b) to cancel out the log-partition factor and directly regress on the win- rate difference. However, it is shown theoretically and empirically by Pal et al. (2024) that the pairwise loss may only drive the relative likelihood gap to be large, but may not necessarily drive up the likelihood of the preferred responses. Our method instead has deeper connection the policy gradient theory and can effectively match the likelihood of the response to its win rate. We postpone the detailed discussion of related works Appendix ^
PRELIMINARIES
We consider the preference learning scenario as follows. Given a text sequence (commonly referred as prompt) x [I1, ...] two text sequences y [y1 .2....] and y' are generated as re- sponses to the prompt x. An autoregressive language model æ given the prompt x can generate responses y following the probability decomposition
# (y x)
II
T m
Given the prompt and two responses y and y', preference oracle (either human annotator or a language model) will provide preference feedback oly > y'lx) {0, 1} indicating whether y is preferred over y'. We denote P(y y' xx) Efo(y [ as the probability of y "winning the duel" over y'. The KL divergence of two probability distributions of density and is defined as (L(pl) Ey-p(r) log p(y)
s(y)
2.1 RLHF WITH REWARD MODELS
Christiano et al.
(0177) first learn a reward function x(y:x) following the Bradley-Terry model (Bradley Terry 1952). For prompp-respons--rssoons triplet yx),,'), the Bradley-
Terry model specifies the probability of being chosen over as
exp(r(y;x))
Fly y/xx exp(r(y;x)) exp(r(y';x)) (r(y)x)) -ay r(y::))),
(2.1)
where (Ix) et /ce +1) is the logistic function. The reward function associated with the Bradley- Terry model can be estimated by maximizing the log-likelihood log Pyy > y'(x). Suppose the true reward function x)y; is available, Christiano et al. (2017) proposed to solve the following optimization problem with policy optimization algorithms in RL. such as PPO (Schulman et al. 2017):
max Ex-xX y-aa [xy; x -1 -7 Exx (x)) 0
(x))]
(2.2)
where is the prompt distribution. Rafailov et al.] (2024b) identified that the optimization problem above has closed-form solution such that for any y.
ा (ylx) oc Terry x) exp(rr(y;:)),
which can be further converted to the DPO loss for any triplet (ww where the winner yw is chosen over the loser y::
To(Yw Txer(yw<x))
) -
(xpo(x, Yw, Yon Y1; 0; Teet) == - loog a 11 log
log
Trer(y|(x)
3

---

Published as a conference paper at ICLR 2025
2.2 RLHF WITH GENERAL PREFERENCE
Following Wang et al. (024): d: Munos et al. 2023), we aim to establish RLHF methods without reward model, as the human preference can be non-transitive (Tiverkyy) T969p. Under general preference oracle P(y > y'xx,, we follow Dudík et al. (2015) and aim to identify the von Neumann
winner. More specifically, the von Neumann winner . is the (symmetric) Nash equilibrium of the following two-player constant-sum game:
(T ग = arg max min Exwo (y (Ply > y/|x)]
(2.3)
In addition, we define the winning probability of one response y against distribution of responses T as
P(y m|x) = (xx) (Ply y'(x)],
and the winning probability of one policy against another policy n'
P(( > n' x = By ( [P > y'|x)]-
Furthermore, we define PP(T > x¹) = Ex~x [TT > n x' where x is a prompt drawn from the prompt distribution X. The two-player constant-sune game 2..3 can be simplified as
(TT x²) arg max min P(T > T²).
3 SELF-PLAY PREFERENCE OPTIMIZATION (SPPO)
In this section, we introduce the Self-Play Preference Optimization (SPPO) algorithm, derived from the following theoretical framework.
3.1 THEORETICAL FRAMEWORK
There are well-known algorithms to approximately solve the Nash equilibrium in constant-sun two-player game. In this work, we follow Freund & Schapire (1999) establish an iterative frame- work that can asymptotically converge to the optimal policy on average. We start with a theoretical framework that conceptually solves the two-player game as follows:
t++1 (ylx) o Te (y(x)) exp(nP(y > Tx(x)), for 1.2.....
(3.1)
3.1 is an iterative framework that relies on the multiplicative weight update in each iteration and enjoys clear structure. Initially, we have base policy πi usually from some supervised fine-tuned model. In each iteration, the updated policy T++11 is obtained from the reference policy following the multiplicative weight update. More specifically, response y should have higher probability weight if it has a higher average advantage over the current policy Nt.
Equivalently, 33. can be written as
Tx+1(y|x)= Te(y(x)ex) exp (niP(y > T((x))
(3.2)
Zm, (x)
where Zr, (x)
Ey Te (y(x)) exp (np(y > ne (x)) is the normalizing factor (a.k.a., the partition function). For any fixed and y, the ideal update policy m4t1 should satisfy the following equation:
(xrx()
log
)
Ply > xx(x) log Zn, (x).
(3.3)
(y|x))
Unlike the pair-wise design in DPO or IPO that cancels the log normalizing factor log Zn, (x) by differentiating (3.3) between y and y', we choose to approximate (3.3) directly in terms of La
distance:
T = argmin ExxxX yoottay log x(ylx) - (nP(y > Tx(x) - log Zo, (x))) (3.4) (y(x)x)
4

---

Published as a conference paper at ICLR 2025
Estimation of the Probability The optimization objective (3.4) can be approximated with finite samples. We choose to sample K responses Y2.Y2..... .K ~ (-(x for each prompt x, and de- note the empirical distribution by म... The finite-sample optimization problem can be approximated
as
x(x(x) Te(y)x)
(nP(y > मे (x) log Zzp (xx) (3.5)
At-1 = argmin Ex X y-mm())))))
log
Specifically, P(y > xx) Σ Ply > ys|x)/K and Zap (x) = By (x) (e > TH [x)]] Zape (x), treated as an expectation, can be further estimated by B new samples with in total B(KB) queries of the preference oracle P. (3.5) is an efficiently tractable optimization problem. Informally speaking, when K + oo, 3.5 will recover (3.4). We have the following guarantee on the convergence (3.4p:
Theorem 3.1. Assume the optimization problem (3.4) is realizable. Denote m. as the policy ob- 1= Tt. By setting = (1/vV)), we have
tained via 3.4 and the mixture policy कr = + > that
max [T(m का) min [P( का) = 0(1/V))
Theorem 3.1 characterizes the convergence rate of the average policy across the time horizon T towards the Nash equilibrium, in terms of the duality gap. The proof is based on Theorem I in Freund & Schapire (1999) with slight modification. For completeness, we include the proof in Appendix[[]
Alternatively, we can avoid estimating log 7 (x) by replacing it with constant based on the human preference model. The choice of the constant is discussed in detail in Appendix E. Here, we replace log Zą x (x) with m/22 in (3.5) to obtain more clear objective:
x(ylx) Tx(y|x)
Plyy > xx 1x) -x))
T++1 = argmin X
log
(3.6)
Intuitively, if tie occurs (i.e., Ply 7 (x) 1/2). we prefer the model does not update weight at y. If y wins over सर on average (i.e., Ply > n. (x) > 1/2), then we increase the probability density at y employ the advantage of y over म. In our experiments, choose to minimize the objective (3.6). In Appendix B we provide a detailed comparison between (3.6) and other loss functions such DPO and IPO.
3.2 THE SPPO ALGORITHM
Based on the aformentioned theoretical framework, we propose the Self-Play Preference Optimiza- tion algorithm in Algorithm [
Algorithm Self Play Preference Optimization (SPPO)
1: input: base policy ne,, preference oracle IP, learning rate number of generated samples K.
2: for t 1, 2.... do
3: Generate synthetic responses by sampling x ~ X' and y1:k ~ (-)x).
4: Annotate the win-rate P(ys Yk /e/|x), Vk, k' [K].
5: Select responses from Yı.k form dataset D. ((xi, Ply: > n. xx(x:)))
6: Optimize Te according to (3.6):
(o())xx)
'n( (Ply mx(x) 5 1 2
0c+1 + argmin E (x. P(y))))))) log
(x(x)x)
(3.7)
7: end for
In each iteration 4, Algorithm for each prompt x (Line 3 Then, the preference oracle IP will be queried to calculate the win rate
1 will first generate K responses Y1, .2,..... according to me( (x)
Assuming the winning probability between any given pair is either 1 or with equal chance, when K o0, can show that indeed Z (x) e²/2, Also Appendix F for complete derivation.
5

---

Published as a conference paper at ICLR 2025
among the K responses (Line 4). At Line certain criteria can be applied to determine which sponse should be kept in the constructed dataset Dj and construct the wewpt--e--ooobaaobbbiiie triplet (x P(y > TT(x)). We will discuss the design choices later in Section 4 One straightfor- ward design choice is to include all K responses into Dr and each Ply; > ny |X] is estimated by comparing y, to all K responses. In total, O(K2) queries will be made. Then the algorithm will optimize 3.6 on the dataset D, (Li
3.3 CONNECTION TO POLICY GRADIENT
While SPPO is derived from the iterative framework (Freund & Schapire 1999) for two-player games, the square loss in the SPPO objective (3.4) provides an alternative interpretation for SPPO as semi-online variant of policy gradient method due to its special loss form. The difference from standard policy gradient is that it collects samples from iTe at the start of iteration t, rather than perform on on-policy sampling each gradient step.
Consider general reward function x(y:x), the RLHF problem 2.2 can be written as:
r(y:x) log (y(
max ((0)
(3.8)
Trel(y /
The policy gradient of the objective J(0) is:
log To(y|x)
(e) = Ex y
(((x:)) -nn
b(x)) V log (v(xx)
(3.9)
(
To Ty (x)
X
r(y;x) - . log
b(x)
(3.10)
Tear (y x)
where the first line follows the policy gradient theorem (Sutton et is an arbitrary constant relying only on x used for variance reduction 3.10p with the SPPO objective (3.4) (rewritten below):
1999) and the baseline b(x)
Comparing the square loss
( P(y > To, (x) - 17 log
To (y [x)
27
yooso (-bx) (-bx) 0++1 = argmin Ex-x y
17 log ZTo, (x)
Te, (y xx)
one can see that the win rate Pyy > Te, x) exactly the reward SPPO aims maximize, and log Zno, (x) is in fact the best possible baseline-the the (soft) value function. When the value function is not available in practice, it can be replaced by any constant baseline to reduce the variance of the policy gradient. We choose 1/2 as good approximation to 17 1 log 7Ta, (x) but the constant can vary depending on the human preference model (see Appendix
Comparing with the general framework proposed by [wwamy et al. (2024), SPPO can be seen as a new, straightforward variant of policy gradient method without the need of extra modifications such as gradient clipping in PPO, Hessian calculation TRPO, or maintaining multiple components (Q-critic, V-critic, actor, etc.) in many policy optimization algorithms.
3.4 TOKEN-LEVEL Q' LEARNING
Rafailov al. 2024a) showed that under the Max-Entropy RL formulation, the token-level log- ratio log =ely
=atly]x) can be seen as an implicit token-level reward or advantage function (invariant under reward shaping). Below we show the square loss in SPPO can also lead to the optimal Max-Entropy policy n², with token-level optimal value/advantage function.
We first briefly restate the setting and results in Rafailov et al. (2024b). The token-level MDP defines as the prefix tokens, and the action a 1/h as the next token. token-level policy π (a,, (sp)) and the
the state S₁ (x, y1, ................................ 9h-11
An nuto-regressive language model π (y]) can be viewed transition kernel is known and deterministic because it only concatenates the next token to the prefix form new token sequence 1 (x, yl, y/,..... , 1/hh)
The Max- Max-Entropy RL setting again considers the reverse- KL. regularized reward maximization prob- lem 2.2):
max X yo [ry;) -1 -Ex (KL(Te ((a()
'Equation 3.9 is also discussed in Munos et al. 2023 More recently, the same gradient form
3.10 has
also been analyzed by Tang al. 2025

---

Published as a conference paper at ICLR 2025
X y- (x)))(((00((()())) + +n= log Tee ( (y[x)]
Exxxx (+((
We denote the optimal solution for the problem above Rafailov et al. (024a)) showed that the Bradley- Terry preference model (B.2) can be rewritten as:
T (ay (s% Tea / 8%)
log x (a), (s%)) h=1 Trat (a), (s)))
wy + vi(x) o
log
where the state and action is defined as in the token-level MDP introduced above, with superscription (.)" and (-) denoting if it is for the winner y or the loser yt. And maximizing the log likelihood with कt replaced by mo gives the DPO loss.
From now on we assume the horizon is fixed at H for simplicity. The derivation of the Max-Entropy RL formulation relies on the (soft) optimal value function Q" and V a
V ($##11 r(sH+1)) r(y:x), (reward EOS)
(Sh, aA) = 17 log (a,((a))s)) (S)--),
(*(ss) - log Σ exp (n (sh, a)), when h S H.
Rafailov al. (2024a) showed that the optimal policy X² satisfies:
" log x=" (ax (s.) = Q" (sh, an) - (sx)
= . log Tea (as + V (sh+1) - (sw).
can be verified that for $1 =x),, we have (s1) = log Ey Terly//x) exp (vr(y: x)) Going
back to the SPPO objective (3.4) at t-th iteration. if we set Tee
and r(y:x) = Pyy > x(xx),
we have (siss) 17 log 2T. (x), and the learning objective at f-th iteration becomes:
x (yyxx)
2
T++1 = argmin Xx
log
(PPly > Telx) - log nz, (x)
(x
H
T(an
argmin E, Ma
log
(3.11)
(an (sA)
Similar DPO, SPPO "secretly" encourages the policy he converge the optimal policy "' at token level via the square loss form (3.1]. Additionally, one may realize that minimizing the square-loss form is related minimizing the KL divergence KL(To
[| via policy gradient:
To (a(a)(8h) n (a/a/ (sh))
VoKL(Tel|=") Es X.
log
Ve log To Te(a, (n))
hml
hm1
1
To (a x *a(a/ss>
2
V
Σ /=1
log
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
We briefly summarize experiment setup below. For full description of our experiment setup, see Section C
Base Model and Datasets: We follow Snorkel's experimental setup, using
and Llama-3-8B-Il Instructt as our base models and Ultrafeedback for prompts. We split the dataset into three portions to avoid overfitting and ensure fair comparison with Snorkel.
Preference Model: We use PairRM, 0.4B pair-wise preference model based on DeBERTA-V3, trained on high-quality human-preference datasets. PairRM outputs "relative reward" between any given pair.
Here we restated with the sequence-level reward x(y:)). Rafailov et al. 2024a) started their derivation from ground bouund-tuuth token-level reward r(sa, ax), which is under-specified due to the reward reshaping is- sue (Ng et al. 1999): reshaping the reward will not affect the Bradley-Tens preference probability it is impossthle recover the ground-truth reward from the preference signal (Rafailov al. 2024a Section 4.2).

---

Published as a conference paper at ICLR 2025
Table AlpacaEval 2.0 evaluation of various models (detailed Baselines) terms of both normal and length-controld (LC) win rates in percentage (%). 7B-rPO Iter3 model achieves the highest LC win rate of 28.53% and normal win rate of 51.02%. SPPO demonstrates steady per- formance gains across iterations and outperforms other baselines which show a tendency to produce longer responses. Additionally, re-ranking with the PairRM reward model (best-of- of-)--o-16) at test time consistently enhances the performance actoss all models and SPPO (best-of-10) 16) achieves high win rate without strong external supervision like GPT-4. We additionally include the results obtained from fine-tuning Llama-3-8B-Insttee which also show steady performance improvement.
Al AlpacaEval 2.0
Model
LC Win Rate
Win Rate
Avg. Len
Mistral-7B Baaed-----Inntttttiie (hest-of-16)
17.11
14.72
1676
22.45
17.94
1529
Snorkel PMistral-PairRRM-DPPO)
26.39
30.22
2736
Snorkel Milistral-PaiRMM-- DPO best-of-16)
29.97
34.86
2616
Self Self-Rewarding 70B Iterl
9.94
1092
Self-Rewarding 70B Iter2
15.38
1552
Self Ref-RRearddig 70B Iter3
20.44
2552
Mistral-7B-DPP Iterl
23.81
20.44
1723
Mistral- (B-rOO Iter2
24.23
24.46
2028
Mistral-7B-DDP Iter3
22.30
23.39
2189
Iterl
23.78
20.77
1693
Mistral-7B-Ii Iter2
21.08
23.38
2660
Mistral Mistral-7B-IPO 7B-IPO Iter3
20.06
22.47
2760
Iterl
21.79
(3.51(.8.79)
1855
Mistral-BB-SPP Iter2
26 26,89
27.63 27.62
2019
Mistral-7B-SPPPO Iter3
28.53 (1.64)
31.02
2163
Mistral-7B-SPPP Iterl (best-of-16)
28.7 28.71(46.26)
27. (7.77(+9.83)
1901
Mistral-7--SPPP Iter2 (best-of-16)
31. 31.23
32. 32.12,44.35
2035
Mistral-7B-SPPO Iter3 (best-of-16)
32.13
34.94
2174
Llama Llama-3-8B- 3-St-Instruct
22.92
22.57
1899
Llama-3- 3-8B
3-8---88PPPP- SPPO Iterl
31.73
31.74
1962
Llama-3-8B-SPPO Iter2
35. 15.11
35.98
2021
Llama-3-8B-SPPO Iter3
38.77 (43.62)
39.85
2066
We sample K responses per prompt with top = 1.0 and temperature 1.0. We select the responses with the highest and lowest PairRM scores as the winning and losing responses respectively.
Response Generation and Selection:
Snorkel, iterative DPO and IPO for Mistral, and Self-rewarding LM. Benchmarks include AlpacaEval 2.0, MT-Bench, Arena-Hard and the Open LLM Leaderboard. We also evaluate Llama-3-8B--Inse on AlpacaFval 2.0.
Baselines and Benchmarks: We evaluate
4.2 EXPERIMENTAL RESULTS
Evaluation using GPT-4 as judge Human evaluation remains the benchmark for quality and accuracy (Askell al. 202T Ouyang al.) (022). However, due its limitations in scalability and reproducibility lity, we explore the alternative approach of using the advanced capabilities of GPT-4
(OpenAl et al. 2023) as an automatic evaluation tool. We conduct GPT-4-based automatic eval- uation Aval 2.0 (Li et al. 2023b), MT-Bench (Zheng et al.) 2023), and Arena-Hare Li cct al. 2024) to measure the chatbot capability of our model. The results can be found in Table I for AlpacaEval 2.0, Figure 3 (left) for MT-Bench, and Figure 3 (right) for Arena-Hard. We found that the performance of SPPO models consistently improves throughout all iterations (iterations).
Table (AlpacaEva 2.0) shows the win rate over the GPT-4-Turbo baseline of different models on 805 prompts. We also include column indicating the length contth-cnntollled win rate, and one col- umn on the average length each model, to account for the tendency of the LLM-based judge to favor longer sequence outputs
issue colloquially termed the "reward hacking" phenomenon. According to the table, Mistral-7B-SPPO Iter3 has the highest win rate, 28.52% for the length- controlled version, and 31.02% for the overall win rate. The performance gains over previous iterations are 7.69% (Mistral-7P) Marral--B-Bd Baaal-BBIInstiie InIttrctt + Iterl), 2.10% (Iterl - Iter2), and 1 1.64% (Iter2 +
8

---

in Published as a conference paper at ICLR 2025
Table AlpacaEval 2.0 leaderboard results of both normal and length-controlled (LC) win rates in percentage (%). Mistral-7B-SPPO can outperform larger models and Mistral-7B-SPPP (best-of-16) can outperform proprietary models such as GPT-4(6/13). Llama-3-8B-SPP exhibits even better performance.
AlpacaEval 2.0
Model
LCC. Win Rate
Win Rate
GPT-4 Turbo
50.0
50.0
Claude Opus
40.5
29.1
Llama-3-8B-SPPP Iter3
38.8
39.9
GPTT4 0314
35.3
22.1
Llama 3 70B Instruct
34.4
Mistral-7B-SPPO Iter3 (best-of-16)
32.1
34.9
GPT-4
30.2
15.8
Snorkel MMistral-ParrRM-DOOO best-of-16)
30.0
34.9
Mistral Medium
28.6
21.9
Mistral-7B-SPPP Iter3
28.5
Claude 2
28.2
17.2
Snorkel (Mistral- PairRM-DPR)
26.4
30.2
Gemini Pro
24.4
18.2
Mistral 8x v0.1
23.7
Llama Instruct
22.9
22.6
MT--Bench
Model
1st Tum 2nd Tum
Avenge
Model
Arena Arena-Hard Hard-Auto- v0.1
Mistral-7B-Insttussss
7.78
Snorkel (Mistral-Pairke Sairral-PiiRM--PPOO
7.83
7,33
7.58
Cestrll7B-Instreeee
12.6
Mistral78--DPO Iterl
7.45
6.58
7.02
Snorkel MMistral-PairRM-DOOO))
20.7
Mistml-7B-DDR DPO lter2
7.57
6.56
7.06
Mistral-BB DB-Da-B--POO lter3
7.49
6.69
7.09
Mistral-7B-SPP Iterl
18.7
Mistral-BB-SPPP Iterl
7.63
6.79
7.21
Mistral-7B-SPPE Iter2
20.4
Mistral-BB.5 SPPO lter2
7.90
7.08
7.49
Mistral-788 PPPPOO lter
7.84
7.34
7.59
Mistral-7B-SPPO Iter3
23.3
Mistral-7B-SIe Iter3 outperforms all baseline models by achieving an average score of 7.59 in MT-Bench. Despite initial drops in per- formance in the first two iterations, SPPO Iter3 improves upon the base model by the final iteration. Right: Mistral-7B-SPPP Iter3 outperforms the baseline model Arrna-Hard. The improvement across different iterations is consistent.
Table 3: MT-Bench & Arena-Hard Evaluation. Left:
Iter3), respectively, indicating steady improvements across iterations, as illustrated in Figure We also apply SPPO stronger baseline model, i.c., -lamm-3-8B--nsttiee Llama-3-8BBSSPPO has higher length-controlled win rate 38. 77% and overall win rate 39.85%. The performance gains are more significant: 8.81% % (llma----B-Instruee - Iterl), 3.42% (Iterl > Iter2), and 3.62% (Iter2 + Iter3), summing up total gain of 15.85%.
and the fine-tuned model
Additionally, the result indicates that SPPO achieves superior performance compared to the iterative variants of DPO and IPO. The length-controlled win rate for SPPO reaches 28. 28.53,, outperform- ing the DPO's best rate of 26.39% (by Snorkel) and IPO's rate of 25.45%. Notably, while DPO and IPO training tend to significantly increase the average output length Length-2736 2736 and 2654, respec- tively-SPPO shows at the third iteration. Finally, we present the best-of-16 results for each model, selected using the PairRM reward model. We find that re-ranking with the preference model at test time can sistently improve the performance of base model M SPPO (Iter3) by 5.34%. 3.57%, and 3.6%. respectively. Notably, this shows that while SPPO sig- nificantly enhances model alignment using PairRM-0.4B as the sole external supervision, it has not resulted in over-optimization against the preference model (Gao et al.) (002)).
moderate length increase, moving from 1676 in the base model to 2163
DPO (Snorkel), and
In Table 2 we compare SPPO on the AlpacaEval 2.0 leaderboard with other state-of-the-art AI chatbots. Our SPPO model outperforms many competing models trained on proprietary alignment data (e.g., Claude 2, Gemini Pro. Llama 3 8B Instruct). When applied to Llama 8B Instruct, our -lama-3-8B-SPPO exhibits an even higher win rate. With test-time reranking, Mistral-7B-SPPO Iter3 (best-of-16) even competitive to GPT-4 0613 and Llama 3 70B Instruct.
In (left, we evaluate the performance of SPPO on MT-Bench. We can see that Mistral-7B- SPPO Iter3 outperforms all baseline models, achieving an average score of 7.59. While we are not certain why the MT-Bench performance drops at the first two iterations, the performance SPPO at the final iteration still improves over the base model.

---

Published as a conference paper at ICLR 2025
Table Open LLM Leaderboard Evaluation. SPPO fine-tuning improves the base model's per- formance on different tasks, reaching a state-of-the-art average score of 66.75 for Mistral-7B and 70.29 for Llama-3-8B. For Mistral-7B, subsequent iterations of DPO, IPO, and SPPO see decline in performance. It is possible that aligning with human preferences (simulated by the PairRM pref- erence model our study) may not always enhance, and can even detract from, overall performance.
Models
Are
TruthfulQA
WinoGrande
GSM8k
HellaSwag
MMLU
Average
63.65
66.85
77.98
41.93
84.89
59.15
65.74
Snorkel
66.04
70.86
77.74
36.77
85.64
60.83
66.31
Mistral-7B-DPP Iterl
63.14
68.39
77.19
40.33
85.25
59.41
65.62
Mistral-7B-DPO Iter2
64.16
67.84
76.09
39.95
85.23
59.03
65.38
Mistral-78-DPO Iter3
65.19
67.89
77.27
32.30
85.49
59.00
64.52
Mistral-7B-In 7B-IPO Iterl
64.68
68.60
77.98
43.75
85,08
59.04
66.52
Mistral-7B-Ii 7B-IPO Iter2
62.12
66.30
77.51
39.20
83,15
59,70
64.66
(Barall-B-IPPP Iter3
62.97
67.12
77.51
37.45
83,69
59.57
64.72
Mistral-7B Saaral--B-SPPPP 7BSPPO Iterl
65.02
69.40
77.82
43.82
85.11
58.84
66.67
Mistral-7B-SPPE Iter2
65.53
69.55
77.03
44.35
85,29
58.72
66.75
Mistral-7B-SPPO Iter3
65.36
69.97
76.80
42.68
85.16
58.45
66.40
Llama-3- Moma---888 3--3--88-mme 3B-IInstrcct
62.29
51.65
76.09
75.89
78.73
65.59
68.37
Llama-3 3 8B- SPPO
63.82
54,96
76.40
75.44
79.80
65.65
69,35
Llama-3 3- 3-PP Iter2
64.93
56.48
76.87
75.13
80.39
65.67
69,91
Llama-3-4 3-8B-SPPO Iter3
65.19
58.04
77.11
74.91
80.86
65.60
70.29
Arena-Hard Hard (Li et al. 2024) contains 500 challenging user queries and follows the same evaluation method as AlpacaEval 2.0. In Table (right), we evaluate the performance of SPPO on Arena-Hard. We can see that Mistral-7B-SPPe exhibits steady performance gain across iterations. Mistral-7B- SPPO Iter 3 outperforms the baseline models, achieving an average score of 23.3.
Open LLM Leaderboard We further evaluate the capabilities of SPPO models using Hugging- face Open LLM Leaderboard (Beeching et al. 2023a). This leaderboard encompasses 6 differ- ent datasets, each focusing on specific capability of LLMs: Are (Clark et al. 2018). HellaSwag (Zellers et al. 2019), Winogrande (Sakaguchi et al. (021). MMLU (Hendrycks al. 2020), Truth- fulQA (Lin al. 2021), and GSMSk (Cobbe et al.) 2021). The models are prompted with zero or few-shot exemplars. The results, presented in Table demonstrate that SPPO can enhance the performance of the base model on Are, TruthfulQA, and GSM8k, and achieve the state-of thethar-tt performance with an average score of 66.75. However, these improvements do not hold in subse- quent alignment iterations: DPO, IPO, and SPPO's performance declines after the first or second iterations. This limitation may be attributed to the "alignment tax" phenomenon (Askell et al., 2021], which suggests that aligning with human preferences (simulated by PairRM preference in our study) might not improve or hurt the general performance. Improving language model capabilities through alignment iterations remains a topic for future research, and we posit that in- corporating high-quality SFT annotations (Chen et al. 2024) could play significant role in this endeavor.
5 CONCLUSIONS
This paper introduced Self-Play Preference Optimization (SPPO), an approach to fine-tuning Large Language Models (LLMs) from Human/Al Feedback. SPPO has demonstrated significant improve- ments over existing methods such DPO and IPO across multiple benchmarks, including AlpacaE- val 2.0, MT-Bench, Arena-Hard, and the Open LLM Leaderboard. By integrating a preference model and employing new optimization objective, SPPO can align LLMs more closely with hu- man preferences.
Limitations Theoretically, approximating the optimal policy update via regression relies on the as- sumption that the model class is expressive enough and the generated data well cover the input space. Approximating the fog-partition factor with constant can help reduce variance only when it is close to the soft value function. The experiments are run on one dataset UltraFeedback and the models are tested few benchmarks due to limited computational resources, but the proposed methods can be further validated on more models, datasets, and benchmarks to have a holistic evaluation given more resources.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank the anonymous reviewers and area chair for their helpful comments. YW, KJ and QG are supported in part by the NSF grants IIS-2008981, DMSS-3231133, CPS-2312094, IIS-2403400 and Sloan Research Fellowship. YW is also supported by UCLA Dissertation Year Fellowship. The views and conclusions contained in this paper are those of the authors and should not be interpreted as representing any funding agencies.
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Ustün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimiza- tion for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume Long Papers), pp. 12248-1222 12267, Bangkok, Thailand, August URL
2024. Association for Computational Linguistics. doi: 10.
https::/aclaat /ttpps://claantholoog... org/ 2024. a0l.oo/ooo...ooo 662
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as laboratory for alignment. urXiv preprini arXiv:2 2t::::2122.0000 12.00661, 2021.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm to understand learning from human preferences. arXiv preprint 3rXiv:2310.12033 2023.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Ra- jani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open I1m leaderboard. Hugging Face, 2023a.
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Ra- jani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard. https:
11m leaderbos rd 2023b.
Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, 39(3/4):324-355, 1952. ISSN 0006-3444. doi: 10.2307/2334029.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, al. Human alignment of large language models through online preference optimisation. arXiv preprint 20XV:22403.086355 2024.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint a3X.3.2401.01335, 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- tems, 30, 2017
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1rx.v:1803.05557 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 10X 2021.
Ganqu Cui, Lifan Yuan. Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Utrafeedback: Boosting language models with high high-quality feedback. arXiv preprint ar 33.0022310.0137,, 2023.
11

---

Published as a conference paper at ICLR 2025
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsumori B Hashimoto. Length-controlled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint 2xXi:22444004475 2024a.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Perey Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024b.
Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Con- textual dueling bandits. In Conference Learning Theory, pp. 563-587. PMLR, 2015.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.011306 2024.
Yoav Freund and Robert E Schapire. A application to boosting. Journal of computer and system sciences, 55(1):119-1) 1997.
generalization of on-line learning and an
Yoav Freund and Robert E Schapire. Adaptive game playing using multiplicatiive weights. Games and Economic Behavior, 29(1-2):79-103, 1999.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Zhaolin Gao, Jonathan D Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Joachims, Andrew Bagnell, Jason Lee, and Wen Sun. Rebel: Reinforcement learning via regressing relative rewards. arXiv preprini arXiv:2404. 167677 2024.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- ence on machine learning, pp. 1861-1870. PMLR, 2018
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style embedding sharing, 2021.
pre-training with
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint SKKV:22009003300, 2020.
Jiwoo Hong, Noah Lee, and James Thome. Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXhv:2403.07691, 2024.
Kaixuan Ji, Jiafan He, and Quanquan Gu. Reinforcement learning from human feedback with active queries. arXiv preprint urXiv:2-02.09440, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825 2023a.
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm- bln-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint rrXiv:2300.02561, 2023b.
Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. Generative judge for evaluating alignment. arXiv preprint arXiv:2310.05470, 2023a.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gon- zalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.1 11939, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follovinn models. https: //gi thub
[abbalpaca_eeaal 2023b.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprini arXiv:2109.07958 2021.
12

---

Published as a conference paper at ICLR 2025
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023.
Hao Lou, Tao Jin, Yue Wu, Pan Xu, Quanquan Gu, and Farzad Farnoud. Active ranking without Strong stochastic transitivity, Advances in neural information processing systems, 2022.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances in Neural Information Processing Systems (NeurIPS), 2024.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint urXiv:2312.00886 2023.
Andrew Y Ng. Daishi Harada, and Stuart Russell, Policy invariance under reward transformations: Theory and application reward shaping. In leml, volume 99, pp. 278-287, 1999.
Josh OpenAl, Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 17730-277444 2022.
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.
Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From to qt: Your language model is secretly a q- q-function. arXiv preprint arXiv:2404.12358 2024a.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.0377 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585 2023.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056 2024.
Xiaohang Tang, Sangwoong Yoon, Scongho Son, Huizhuo Yuan, Quanquan Gu, and Ilija Bo- gunovic. Game-theoretic regularized self-play alignment of large language models, 2025. URL
http:/iww.rr/a/b///////// 00030
Amos Tversky. Intransitivity of preferences. Psychological review, 76(1):31, 1969.
13

---

Published as a conference paper at ICLR 2025
Yuanhao Wang, Qinghua Liu, and Chi Jin. Is rlhf more difficult than standard rI? theoretical perspective. Advances in Neural Information Processing Systems, 36, 2024.
Yue Wu, Tao Jin, Qiwei Di, Hao Lou, Farzad Farnoud, and Quanquan Gu. Borda regret minimization for generalized linear dueling bandits. In ICML 2023 Workshop The Many Facets of Preference- Based Learning, 2023.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sam- pling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint arXiv:2312.11456 2023.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023.
Chenlu Ye, Wei Xiong, Yuheng Zhang, Nan Jiang, and Tong Zhang. A theoretical analysis of nash learning from human feedback under general kl-regularized preference. arXiv preprint arXiv:2402.07314, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXi::240.110200, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint arXiv: /905.0090000000 000000 2019.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023.
Liammin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mn-bennh and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Banghua Zhu, Jiantao Jiao, and Michael I Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. arXiv preprint 1rX1v:2301.11270, 2023.
A RELATED WORK
RLHF with Explicit/Implict Reward Model Originally, reinforcement learning from human feedback (RLHF) was proposed by CCristiano et al. (2017) methodology that first learns a re- ward model reflecting human preferences and then uses reinforcement learning algorithms to max- imize the reward. This methodology is applied by Ouyang et al. (2022) to fine-tune instruction- following large language models and leads to the popular ChatGPT.
The reward model in the works mentioned above assumes a parametric model such as the Bradley- Terry model (Bradley & Terry] 1952), which assigns "score" representing how preferred given response is. More recently, Rafailov et al. (024bb proposed to instead directly solve the closed- form solution of such score implied by the Bradley-Terry model. The Direct Policy Optimization (DPO) method is claimed to be more efficient and stable, yet, still implicitly assumes such a reward model that specifies the "score". In a similar spirit. Zhao al. (2023) proposed to calibrate the score so that the score of the winner in comparison has a margin over the score of the loser, and induces a different SLic loss. Similarly, Etthayarahh et al.] (2024) derived a different loss function (called KTO) from the Kahneman- Taannn--Tersse Tversky human utility function, which implicitly denotes score of the given response. Liu et al., (0023) proposed Rejection Sampling Optimization (RSO) which utilizes preference model to generate preference pairs with candidates sampled from the optimal policy; then preference optimization is applied on the sampled preference pairs. Hong et al. (2024) proposed Odds Ratio Preference Optimization (ORPO) algorithm that can perform supervised fine-tuning and preference alignment in one training session without maintaining an intermediate reference policy. Motivated by the length bias in the preference dataset, Meng et al. (2024) model human preference
14

---

Published as a conference paper at ICLR 2025
based the average-teward Bradley-Terry model, and propose Simple Preference Optimization (SimPO) which maximizes the gap between the average log-likelihood of the winner and loser. As discussed in Section 3.3 SPPO can be seen as a semi-online REINFORCE-style algorithm. The effectiveness of REINFOR OCCE, rather than PPO with value function and gradient clipping. has been shown by Ahmadian et al. 2024), where they argue that REINFORCE should suffice for RLHF tasks. Our work also corroborates this observation.
RLHF with General Preference Model Often, the human preference is not strictly transitive, and cannot be sufficiently represented by a single numerical score. |Azar et al. (2023) proposed general preference optimization objective based on the preference probability between pair of responses instead of score of single response. They further propose learning objective based on identity mapping of the preference probability called IPO (Preference Optimization with Identity mapping). which aims to maximize the current policy's expected winning probability over given reference policy. Munos et al. (2023) formulated the RI.FF problem with general preference as a two-player, constant-sum game, where each player is one policy that aims to maximize the probability of its response being preferred against its opponent. They aim to identify the Nash equilibrium policy of this game and propose mirror-descent algorithm that guarantees the last-iterate convergence of a policy with tabular representations" policy for multi-step MDPs when a general preference model is present and shows that the problem can be reduced to two-player zero-sum Markov game.
[1 Wang et al. (2024) proposed to identify the Nash equilibrium
Theory of RLHF There is also a line of research to analyze RLHF and provide its theoretical guar- antees. Zhu et al. (023) studied the standard RLHF with separate reward-learning and model-tuning and proposed a pessimistic reward- reward-learning process that provably learns a linear reward model. Wang et al. (2024) proposed framework to reduce any RLHF problem with a reward model to reward-based standard RL problem. Additionally, they proposed to identify the Nash equilibrium policy when a general preference model is present and show that the problem can be reduced to two-player zero-sum Markov game. [iiong et al. (2023) studied the reverse- KL regularized con- textual bandit for RLHF in different settings and proposed efficient algorithms with fnite-sample theoretical guarantees. Ye et al.] (2024) studied the theoretical learnability of the KL-rrgularized Nash-Learning from Human Feedback (NLHF) by considering both offline and online settings and proposed provably efficient algorithms. Ji et al. (2024) proposed an active-query-basee proximal policy optimization algorithm with regret bounds and query complexity based on the problem di- mension and the sub-optimality gap.
Self-Play Fine- Fini-n Most works mentioned above (Rafailov et al. 2024b Zhao et al. 2023
Azar et al. 2023] Ethayarajh et al. (022)) consider one single optimization procedure starting from some reference policy. The same procedure may be applied repeatedly for multiple iterations in self-play manner. In each iteration, new data are generated by the policy obtained in the last iteration; these new data are then used for training a new policy that can outperform the old policy.
The self-play fine-tuning can be applied to both scenarios with or without human preference data. For example. Singh et al. (2023) proposed an Exppetation-MMxxiizzaioo (EM) framework where in each iteration, new data are generated and annotated with a reward score; the new policy is obtained by fine-tuning the policy on the data with a high reward. Chen et al. (2024) proposed a self-play framework to fine-tune the model in a supervised way. In each iteration, new preference pairs are synthesized by labeling the policy-generated responses as losers and the benan-generated responses as winners. Then DPO is applied in each iteration to fine-tune another policy based on these synthesized preference data. Yuan et al. (2024) proposed Self-Rewarding Language Models, where the language model itself is used to annotate preference on its own responses. Iterative DPO is applied to fine-tune language models these annotated data. These works show iterative fine- tuning can significantly improve the performance.
Munos et al. (2023) are among the first to introduce self-play algorithms for aligning large language models with human preferences by computing Nash equilibria of preference model rather than optimizing a reward model. Their proposed method, Nash-MD, based on mirror descent, provides a sequence of policies with the final iteration converging to the regularized Nash equilibrium. How- ever, Nash-MD requires exact sampling from the mixture distribution that is only feasible in the
'Due to the tabular representation, computing the normalizing factor is prohibitive and the algorithm is approximately executed by sampling one token instead of full response.
15

---

Published as a conference paper at ICLR 2025
bandit setting, while sequential generation necessitates approximation that may impact theoreti- cal guarantees.
Swamy et al. (204)) considered a more general multi-step Markov Decision Process (MDP) setting and proposed Self-play Preference Optimization (SPO), an RLHF framework that can utilize any no-regret online learning algorithm for preference-based policy optimization. They then instanti- idealized variant of their algorithm, which reduces to the exponential weight update rule (3.1) when constrained to the bandit setting. The main difference is that they focus on the multi-round Markov decision process (MDP) in robotic and game tasks rather than on fine-ture mee-tuning large language models and approximating the update using policy optimization methods such as PPO.
ated their framework with Soft Policy Iteration
Concurrent our work, Rosset et al. (2024) proposed the Direct Nash Optimization (DNO) al- gorithm based on the cross-entropy between the true and predicted win rate gaps, and provided theoretical guarantees on the error of anite-sample approximation. However, their practical version still utilizes the iterativeDDPO framework in Xu et al. (202)) with the DPO loss instead of their derived DNO loss. Notably, in their experiments, they added the GPT-4 generated responses as their "gold sample" into their fine-tuning data, and used GPT-4
judge to assign a numerical score to each response for preference pair construction. In sharp contrast, our work does not require the use of any strong external supervision besides small-sized reward model. Another concurrent work (Gao et al. 2024) proposed REBEL, an iterative fine-tuning framework via regressing the relative reward. When applied to the preference setting, it results similar algorithm to algorithm SPPO, excett that SPPO approximates the log-partition factor log Zx, (x) with constant 7//2 while REBEL regresses on the win rate difference (so that log Za, (x) is canceled). Additionally, Ca- landriello et al. (2024) pointed out that optimizing the IPO loss (Azar et al., 2023) iteratively with self-play generated data equivalent to finding the Nash equilibrium of the two-player game, and they proposed the IPO-MD algorithm based on this observation, which generates data with a mixture policy similar to the Nash-MD algorithm.
B COMPARISON WITH DPO, IPO, AND KTO
In practice we utilize mini-batches of more than 2 responses to estimate the win rate of given response, while the DPO and IPO loss focus on single pair of responses. When only a pair of responses Yw and is available, we have the pair-wise symmetric loss based on the preference triplet (x, Yw,Yi Yr) defined as:
(x, Y., Tre)) = (10g mo(ymllx) 7 (P(( xxxx) -11x))11)²
log To(y) ) (P(yu - y1(x) Trer(y((x))
1 2) )
(B.1)
where P(y.. > yrlx) can be either soft probability within [o, 1] or a hard label indicating Yw yı.
We now compare the SPPO loss to other baselines assuming a hard label y > y: given. For the ease of comparison, let (3 -1 1)=
To(Yw|x) ) bb -log Trer
To(x)|x)) TTrr(y: c c [Tol (Tee)
Blog
then we have
(rro(Yw, x) = logo - b),
(B.2)
(I X), = 1aa b) 1]2
(B.3)
(KTO(y X) - + c) + a(b c) (simplified),
(B.4)
where (2x) e² ((e" 1) and the SPPO loss can be written
ESppo w.
Yi, x) w. (a 1/2)² + (b 1 1/2)2.
It can be seen that SPPO not only pushes the gap between and to be 1, but also attempts to push value a to be close to 1/2 and the value of to be close to - 1/2 such that T (y., To (x) > Ter(y x)
and To (yi[x) < xx). We believe this is particularly important: when there plenty of
16

---

Published as a conference paper at ICLR 2025
preference pairs, DPO and IPO can ensure the policy will converge to the target policy, but when the preference pairs are scarce (e.g., one pair for each prompt), there is no guarantee that the estimated reward of the winner will increase and the estimated reward of the loser b will decrease. Instead, only the reward gap between the winner and the loser (i.e., a b) will increase. This phenomenon is
observed by Pal al. (2024) that DPO only drives the loser's likelihood to be small, but the winner's
likelihood barely changes. We believe that fitting ,3 log
m=+: (ylx) directly to PPyy > TT|x) 1/2
is more effective than IPO which attempts to fit /3 log
to P(y w > (x) Ply: > Tx In addition, SPPO shares similar spirit as КTO. The KTO loss pushes
TTes (Xwlx)
3 log
Teas (x:l)) x. [y(
(Y [C)
to be large by minimizing a (-a + c) and pushes b to be small by minimizing o (b c). In contrast,
SPPO pushes to be as large as 1/2 and b to be as small as -1/2.
On the other hand, we would like to comment that although DPO and KTO can be extended to their iterative variants, they are not by nature iterative algorithms and do not have provable guarantees that they can reach the Nash equilibrium. In contrast, SPPO and IPO are by design capable to solve the Nash equilibrium iteratively. SPPO is superior to IPO because its design explicitly alleviates the data sparsity issue, as discussed above and detailed in Pal et al. 2024).
C FULL EXPERIMENT SETUP
Base Model and Datasets We follow the experimental setup of Snorke model that utilizes iterative DPO to achieve state-of-the-arr performance on AlpacaEval benchmarks. Specifically, we use Mistral-7B- 11777--InttuuttO... as our base mode[] Hissral-7B-Instruct--o is an instruction fine- tuned version of Mistral-7B-v0.2 model (Jiang et al., 2023a). We also adopt Ultrafeedback (Cuil eet al. 2023) our source of prompts which includes around 60k prompts from diverse resources. During generation, we follow the standard chat template of Mistral-7B. To avoid overfitting during the fine-tuning, we split the dataset into three portions and used only one portion per iteration. These settings were also adopted by training the model Snorkel-Mistral Paaall-aPii---PGG (Snorkel), We follow the splitting in Snorkel for a fair comparison. Additionally, we use Mlama---B-IInstue as stronger base model along with the same preference dataset and data splitting.
Preference Model We employ ParRRM (Jiang et al., 20236). an efficient pair-wise preference model of size 0.4B, PairRM is based on DeBERTA-V3 (He et al., 2021) and trained on high-qualite human-prehe prefeeepeeference datasets. Results benchmarks like Auto-J Pairwise dataset (Li et al. 2023a) show that it outperforms most of the mogguageemodllle rably with larger reward models like UltraRM-13B (Cui et al. (023). We refer the readers to the homepage on Huggingface"") TM) for detailed benchmark results. We therefore keep PairRM as our ranking model following Snorkel for balance between accuracy and efficiency.
reward models and performs compa-
Specifically, PairRM will output a "relative reward" y'; that reflects the strength difference between y and y', i.e..
exp(s(y, y')))
P(y > y'(x) 1 expppp((yy y';))
Unlike the Bradley-Terry-bbsed reward model, PairRM only assigns the relative reward which is not guaranteed to be transitive (i.c.. s(y1, x2;x) s(y2, Y3; # y3;)), So it indeed models the general preference.
Response Generation and Selection During the generation phase in each iteration, we use top p 1.0 and temperature 1.0 to sample from the current policy. We sample with different random seeds to get K = 5 different responses for each prompt. Previous works utilizing Iterative DPO choose 2 responses to form a pair for each prompt. For a fair comparison, we do not include all K = 5 responses in the preference data but choose two responses among them. Following Snorkel, we choose the winner yw and loser y: to be the response with the highest and lowest PairRM score,
17

---

Published as a conference paper at ICLR 2025
which is defined for each response y,
1
SPaiRR (y;;
yx:
K
Probability Estimation We then estimate the win rate over the distribution by the average win rate over all the sampled responses explained in 3.5
Plys > m1(x)
Ply: yb(x), Vi [K].
K
Hyperparameter Tuning The experiments are conducted on 8 x Nvidia A100 GPUs. For SPPO, we trained three iterations in total. In each iteration, we selected the model trained on the first epoch of the 20k prompts from UltraFeedback to proceed to the next iteration. For both Mistral-7B- Instruct-v0.2 and Llama-3-8B-Instie
the global training batch size is set to 64, and is set to le3. The learning rate schedule is determined by the following hyperparameters: learning rate=5.0e- number of total training epochs=18, warmup 1a linear schedule. In practice, early stopping after the first epoch yields the best test performance. The best hyper-parameters for each model are hold-out subset of Ultrafeedback as
selected by the average win rate (judged by PairRM-0.4BB the metric. For more details on the win-rate comparison using PairRM as a judge, please refer to Section an2 Figure 2
We evaluate the following base models as well as baseline methods for fine-tuning LLMs:
Baselines
Mistral-7B-Instiee IB-IIntruct--O.22 2::: Mistral-7B- 7rr---B-I-nttttOii is an instruction fine-tuned version of Mistral- 7B-v0.2 model (Jiang et al. 2023a). the starting point of our algorithm.
Snorkel MMistral-PaiRRMM. -PD)): We directly evaluate the uploaded checkpoint HuggingFacd This model is obtained by three iterations of iterative DPO from Mistral- In-ttrruct--O.2.
We also implement the iterative DPO algorithm by ourselves. The experimental settings and model selection schemes align with those used for SPPO, except for the adoption of the DPO loss function as defined in (B.2). Hyperparameters are optimized maximize the average win-rate assessed by PairRM at each iteration. Note that the practical algorithm in Rosset ct al. (2024) essentially the same iterative DPO.
(Iterative) DPO:
(Iterative) IPO: We implement the iterative IPO algorithm by ourselves. The experimental setting and the model selection scheme the same as iterative DPO, except that the loss function the IPO loss B.3). For fair comparison, hyperparameters for IPO is also selected by evaluation using the average PairRM win-rate on the hold-out subset of Ultrafcedback.
Self-rewarding LM: Yuan al. (2024) proposed prompt the LLM itself a preference judge to construct new preference pairs and iteratively fine-tune the LLM with the DPO algorithm. We use the AlpacaEval 2.0 win rate reported by Yuan et al. (2024) for comparison. Note that Self- rewarding LM is a trained from Llama 2 70B.
Llama-3-8B-Instree Llama-3-8B-Instruu is an instruction-tuned model optimized for dialogue cases and outperforms many of the available open-source chat models on common industry benchmarks.
Benchmarks Following previous works, we use AlpacaEval 2.0 (Dubois et al. 2024a), Arena- Hard et al. (2024). MT-Bench (Zheng et al. 2024). and Open LLM Leaderboard (Beeching al. 2023b) as our evaluation benchmarks.
AlpacaEval 2.0 is an LLM-based automatic evaluation benchmark. It employs AlpacaFarm (Dubois et al. 2024b) its prompts set composed of general human instructions. The model responses and the reference response generated by GPT-4-Turbo are fed into a GPT-4-Turbo- based annotator to be judged. We follow the standard approach and report the win rate over the reference responses.
Arena-Hard (Li et al. 2024) is a high-quality benchmark that claims to harder and has the highest correlation and separability to Chatbot Arena among popular open-ended LLM bench- marks including AlpacaEval 2.0. We evaluate our models Mistral-Paii Paiill-PPPiM-PPPPP and the baseline models.
" "htn:://tugginnffcccoooooooottto http://ww......a..oo//////o//ana
18

---

Published as a conference paper at ICLR 2025
35
30
Snorkel (Mistral PairralPPiRMM.OcM OPO)
LC. Win Rate (%6
30
Mistral-7B.OD 7FODFO
Win Rate (1)
25
Mistral-7B.SPPP
20
25
Shariell (Mistral-PairRk.Dor) PPO)
15
Self RelarBenn 700
20
Mistral-7B-
Mistral-70-HH
10
mistral-79-SPPO
Itell
Iter2
Iter3
Iterl
Iter2
Iter3
(b)
Figure 1: Win Rate against GPT-4-Turbo with (a) and without (b) Length Controlling (LC)
AlpacaEval 2.0. SPPO demonstrates steady improvements on both LC and raw win rates.
MT-BBench (Zheng al. 2024) is collection of 80 high-quality multi-turn open-ended questions. The questions cover topics like writing, role-playing, math, coding, etc.. The generated answer is judged by GPT-4 and given a score directly without pairwise comparison.
Open LLM Leaderboard (Beeching et al. 2023b) consists of six datasets, each of which fo- cuses on facet of language model evaluation. In detail, the evaluation rubric includes math problem-solving. language understanding, human falsehood mimicking, and reasoning. We fol- low the standard evaluation process and use in-context learning to prompt the language model and compute the average score over six datasets to measure the performance.
D ADDITIONAL EXPERIMENT RESULTS
D.1 ADDITIONAL. TABLES AND PLOTS
In Figure I we plot the win rate against GPT-4-Turbo on AlpacaEval 2.0 different RLHF algo- rithms. We see that the performance gains of SPPO over previous iterations are 7.69% (Mistral- 7B-Instruct + Iterl), 2010 (Iterl + Iter2), and 1.64% (Iter2 - Iter3), respectively, indicating steady improvements across iterations.
D.2 EVALUATION USING PAIRRM AS A JUDGE
As SPPO identifies the von Neumann winner (see 2.3) in two-player constant-sum game, we examine the pairwise preferences among SPPO models and other baselines. The pairwise win rates, We observe that in all spprithmssnamel DPO, IPO, and SPPO-ttie the newer model iterations surpass the previous ones. For example, SPPO iteration 3 outperforms SPPO iteration Both SPPO and IPO consistently outperform DPO across all itera- tions. While SPPO is superior to IPO in the first two iterations, IPO exceeds SPPO in performance during the final iteration. Considering the superior performance SPPO in standard benchmarks evaluated by GPT-4 against ground-truth answers (e.g., AlpacaEval 2.0, MT--Bench, and Open LLM Leaderboard), along with IPO's tendency to produce longer sequence outputs (see Avg. Len Table D. we believe this is due IPO exploiting the length bias in PairRM that favors longer quences. Conversely, SPPO models benefit from a more robust regularization within a multiplicative weight update framework.
measured by PairRM, are depicted in Figure 2
19

---

Published as a conference paper at ICLR 2025
IPO Iter3
0.636
0.664
0.809
Snorkel Pacitral-PaiiRM--MP (PO)
0.428
0.480
0.496
0.588
0.500
0.609
0.659
0.652
0.673
0.750
DPO Iter3 0.423
0.477
0.504
0.552
0.577
0.608
0.633
0.631
0.667
0.750
SPPO Iter2 0.369
0.423
0.432
0.448
0.517
0.551
0.595
0. 0.598
0.631
0.722
Iter2
0.344
0.530
0.613
0.707
0.670
SPPO Iterl
0.281
0.324
0.341
0.364
0:405
0.426
0.450
0.514
0.540
0.651
DPO Iterl
0.287
0.318
0.348
0.369
0.402
0.423
0.438
0.486
0.526
00663
IPO Iter1
0.259
0.294
0.327
0.333
0.369
0.387
0.413
0.460
0.474
0.613
Mistral- 78-ntrrucctt
0.191
0.219
0.278 0.293
0.330
0.387
IPO Iter3
SPPO Itera
Snorkel (Mistral-Paii (Mastrl-PairRMM..o Parr------POO DPO)
DPO Iter3
SPPO Iter2
IPO Iter?
DPO Iter?
SPPO iterl
DRO iterl
IPO iter
Mistral 18-Instruct--0.2
Figure 2: Pairwise win rates among base model (Mistral-77 -errall--BIIsmseee IB-IIstruct-vO.22 DPO models, IPO mod- els, and SPPO models using PairRM-0.4B judge, which may favor models with longer outputs. On benchmarks with more powerful judge models (e.g., GPT-4), such as AlpacaEval 2.0 and MT- Bench, SPPO outperforms other baseline algorithms by large margin.
D.3 ABLATION STUDY
28
AlpacaEval
2.0
Mini-Batch Size
Iteration
Win Rate
Avg. Len
LC. Win Rate (%6)
26
LC.
Raw
(chars)
24
Iterl
23.85
23.53
1948
K =
Iter2
26.91
27.24
1999
22
Iter3
28.26
28.22
1961
Snorkel
Iterl
24.79
23.51
1855
20
Mistral-7Binnttructtv022
SPPO (K=2)
K = 5
Iter2
26.89
27.62
2019
18
SPPO (K=5)
Iter3
28.53
31.02
2163
Iter1
Iter2
Iter3
Figure 3: AlpacaEval 2.0 evaluation on SPPO of different mini-bated size in terms both normal and length- cogtthcoontoolled (LC) win rates in percentage (%). K = 2, denote different mini-batch sizes when estimating the win rate P(y > xx)
We study the effect of mini-batch size when estimating the win rate P(y > Tx(x). Specifically, for each prompt, we still generate 5 responses and choose the winner yu and loser yı according to the PairRM score. When estimating the probability, we vary the batch size to be K = 2,3, 5. For K 2, we estimate (P > zx(x) with only samples y and yr:
P x(xx= Ply. > x + P(y > vílx) 1/2 + P(y > y1lx)
and Ply: > n, lx)x similarly. = 5 indicates the original setting we use.
We compare the results on AlpacaEval 2.0, as shown in Figure B We find that the performance
of SPPO is robust to the noise in estimating Ply xx(x). While K initially outperforms
20

---

Published as a conference paper at ICLR 2025
K = 2 in the first iteration, the difference in their performance diminishes in subsequent iterations. Additionally, we observe that = 2 exhibits reduced tendency to increase output length.
D.4 TRADE-OFF BETWEEN KI DIVERGENCE AND ALIGNMENT
KL Divengence Win Fate Methoo and Hyperporometer
Hethods and tiperpaenctrr
SPC bel-aal
JMC
Figure 4: Win rate on AlpacaEval 2.0 length-controlled win rate against KL divergence to the Ceatral-7B-Insruuct-vii base model for various model checkpoints. Each method (DPO, IPO, SPPO) is plotted across three iterations (left to right, as KL divergence increases).
We estimated the KL divergence on the 1000 prompts from the UltraFeedback test set. The KL divergence estimate obtained from the average log-ratio of the responses generated from these prompts.
As shown in Figure 4 the steady improvement in win rate for SPPO demonstrates its efficiency in leveraging the KL budget. Notably, SPPO(8=1e-3) achieves higher alignment metric compared to iterative DPO by the Snorkel team (the purple point) while maintaining lower KL divergence for the final iteration. We also find that 3=1e-2 gives a better trade-off between KL and win rate, with significantly smaller KL divergence and similar final win rate.
E APPROXIMATING THE NORMALIZING FACTOR
As discussed before, we replace the log-partition factor with a constant to avoid either estimating predicting the log-partition factor. In hindsight, the approximation of the normalizing factor serves as a baseline for variance reduction, and does not need to be exact. Here we discuss the implicit assumptions and how obtained approximation based on different assumptions human preference behaviour.
We first consider the case where we have responses and then calculate the limit of 7 (x) when
K + co. We have two extreme cases:
1. The most "disordered" case: any preference is a fair coin flip
2. The most "ordered" case: there is a strict ordering among all responses.
The most "disordered" case Specifically, we have K different responses Y1, .2...., for the given prompt x. Since we consider the general preference setting, we assume that the preference probability between and y, (i < j) we observe is a fair coin toss:
[1, w.p. 1/2,
Ply: y,(x) (0, w.p. 1/2.
Note that for simplicity, we assumed that the preference probability follows the Bernoulli distri- bution, not the preference feedback. The preference feedback is deterministic since the preference probability either or 1. Assuming Ply: > {x)x) follows any other 1/2-mean distribution will yield the same constant.
21

---

Published as a conference paper at ICLR 2025
We define the random variable Pt.j 2y y2(x) 1 for convenience. In total, have ((K - 1 independent Rademacher random variable for all < j, and then we have Pj. = P >k for all i >>>> For J. Pi.j O. We also define X; "5 Pi./K. K.
Given the setting and notations above, we have
K Σ Ply. v,lx)/K X,.
Ply: 7,
Further,
K
ZEp (x)
Σ exp x))KK
e/12 Σ nxx KK.
For any fixed i, we have the expectation as follows:
K
K
(en/K ///KK
+ t-n// n/K
K-1
E
II
E
2
f=1
where the last equation comes from the definition of Ps.j (note that Pi, 0). The variance is:
]arle"X:]= = B|e - Elenxx12 ( (22n// K + t--2n/K k-1
een/K + e-n/K 2K-2
2
2
Additionally, the covariance between eaX; and en (i + j) is:
Cov(e ,)
- E(e B[enXitnX,]]
]
exp Σ Pi./K + Σ Pha/K
1=1
E +e-n/K
2K-4
e/K + te-i/K
2K-4
ee/k + te-n/K KK 2K-2
2
where the third line holds because Pt, Pi.j = 0. Pt.j + Рj. 0, and the rest terms are i.i.d. One can check that when K - oc, we have [x]e + Varle"X. + O, and Co enx;) By Chebyshev's inequality, ΣK enxi /K will converge to in probability. So we have
K Xx (KXi/K- KK 1/2
Zzn (x) = 1/2
and we can approximate log Z (x) with 1
The most "ordered" case We assume there is an ordering ((.) among the K different responses Y1. V2. .Yk for the given prompt x. The preference probability between y, and y, (i j) is:
1, if (1) < <j)),
Ply: > y5lx) o. if (i)) o(j).
Again, the preference feedback is deterministic: as long as y; is ranked higher than y,. Yr will always be preferred over y,- The same responses still tie: P(y: > yi(x) = 1/2.
Without loss generality, we can assume y1 x y2 Y3 K Yk. Given the setting and notations above, we have
K
1-11+11 1/2
Ply. > #K xx)
Σ
P
K
because for yi, there are responses that are strictly worse, and y, ties with itself.
22

---

Published as a conference paper at ICLR 2025
For the normalizing factor, have
K
log 2 (x) = log
Σ
exp(npPy > मे x))KKK
1=1
17 i 11-1/2
log
Σ exp
/K]
K
+ log
exp(nz)dd
e"
log
7
where the third line (limiting) can be obtained by the squeeze theorem.
For L, log e"-1 x 0.547. For large z 1e3 as we used in the experiments, we have log e"-1 RC 7)
Choice of Depending on how "disordered" the preference is, 1) can vary between n/2 and r). As this paper is partially motivated by human intransitive and irrational preference behavior, we chose to use 1/22 to approximate log z (x). Fine-tuning the coefficient of this constant as hyperparameter also an option and can help improve performance on given dataset.
F PROOF OF THEOREM 3.1
Proof of Theorem 3./
Suppose the optimization problem is realizable, have exactly that (y/x() & (y(xx) exp(nP(y > गe [x)x for = 1,2.....
(F.1)
To prove that the exponential weight update can induce the optimal policy, we directly invoke restated version of Theorem in Freund & Schapire (1999):
Lemma F1 (Theorem in Freund & Schapire (1999), restated). For any oracle F and for any sequence of mixed policies #1, ... the sequence of policies IT .2...., ... #T produced by F.I satisfies:
P( 4 pc) + KL(T|
Σ P(T x
Mc)
min
1 1-
1
1=1
By setting j we have that
T
T/
KL(T||TT)) +
min
c- PT x कT)
2
1 c-
where the LHS comes from that PT(T; nt) 1/2 and the RHS comes from that ET P(r K ne) = (rr R.). Now rearranging terms gives
1-e-"
min P(T mT) (L(T||TT) T/
2m
Note that To an autoregressive model that is fully supported on finite vocabulary (To |x, Yi.)) has non-zero probability for every token). Because its support is large but finite set, log [-)) is bounded from above. So we can naively bound the KL-divergence KL(T||TT) S (large) constant.
log (o(-)) which can be
By choosing 77
log la have
VT
log To (o) + +O(T s min [P(T x का]] +
log To To T
2
4VT
O(7²). Notice that 1/2 the LHS is already the value of the symmetric two-player constant-sum game. This shows that for appropriately chosen 17 and T, the mixture policy कy is close to the minimax optimal policy (Nash equilibrium).
where the LHS comes from Taylor's expansion -- -
23

---

Published as a conference paper at ICLR 2025
The optimality gap is thus bounded by
max [T > .τ) min [IP(T x おr)
max
[] - PT π おा) min [P(TT .r)
min [((T मn)
1
G RESPONSE EXAMPLES IN DIFFERENT ITERATIONS
Table 5: Generation example of our fine-tuned model by SPPO at different iterations.
Prompt
You will be given definition of task first, then some input of the task. You are given a dialog between 2 or more individuals. The dialog will include information as to the relationship between 2 of the participants. At the end, you will be asked to provide an answer as to the relationship of those 2 participants. In this scenario, '0" will represent spousal relationship, '1' will represent siblings, and '2' will represent child/parent relationship. Speaker I: I'll be right with you. Okay? Thanks, Wendy. Speaker 2: Oh my God! How cute is the on-call doctor? Speaker 3: Ooh, so cute, that I'm thinking about jamming this pen in my eye. Speaker no-no-no, not Dr. Burke. Dr. Burke is out of town. The-the on-call doctor will see me now. Speaker 4: Dr. Richard Burke out of town. Dr. Timothy Burke, his son, will see you now. Speaker 5: Ready? What is the relationship between Speaker and Richard Burke? Output:
Dr. Burke will see you know. Speaker 2: Oh
Mistral-78
The relationship between Speaker 5 and Richard Burke is that of a parent and child, Richard Burke is identified as the son of another character in the dialog.
Iteration 1
The relationship between Speaker and Richard Burke is that of father and child. Based on the dialogue, it was revealed that Dr. Timothy Burke is the on-call doctor and the son of Dr. Richard Burke.
Iteration 2
The relationship between Speaker and Richard Burke is that of father and child. The conversation reveals that Dr. Timothy Burke is the on-call doctor and he is identified as the son of Dr. Richard Burke.
Iteration 3
Speaker is the individual asking about the relationship, so they are not directly related to Dr. Richard Burke. The information provided in the dialog reveals that Dr. Timothy Burke is the on-call doctor and the son of Dr. Richard Burke. Therefore, the answer is: The relationship between Speaker 5 and Dr. Richard Burke is that of a child (Dr. Timothy Burke) and parent. Speaker 5 is inquiring about the relationship. but they relationship codes (0 for spousal, 1 for sibling, 2 for child/parent) align with this description: Speaker (0, 1, or and Dr. Richard Burke (2) connect parent-child duo. Output: 2 (child/parent relationship) In this scenario, Speaker 5's role is to determine the relationship between herself and Dr. Richard Burke, but since they are not related (she is an outside inquirer), the correct answer pertains only to the relationship identified within the dialog: Dr. Richard Burke (father) and Dr. Timothy Burke (son).
themselves part of it. The specified
24

---

Published as a conference paper at ICLR 2025
Table Another generation example our fine-tuned model by SPPO at different iterations.
Prompt
Q: Context: Roman Brady is fictional character from the long running NBC soap opera, Days of Our Lives. The role was originated in 1981 by Wayne Northrop who played the role until 1984, was filled by Drake Hogestyn from 1986 1991, again by Northrop from 1991 1994, and is currently being portrayed by series veteran Josh Taylor, who stepped into the role in 1997. Question: who is roman brady on days of our lives? Let's give stream of consciousness below
Mistral-77 -BB
Roman Brady is a fictional character on the NBC soap opera "Days of Our Lives." He was first portrayed by Wayne Northrop from 1981 to 1984. Later, Drake Hogestyn took over the role from 1986 to 1991. Northrop returned to the role from 1991 1994. Since 1997, the character being portrayed by Josh Taylor. Roman Brady is significant character in the show's long-running storylines.
Iteration 1
Roman Brady is a fictional character on the NBC soap opera "Days of Our Lives." He was first portrayed by Wayne Northrop from 1981 to 1984. Drake Hogestyn took over the role from 1986 to 1991. Northrop returned to play Roman again from 1991 to 1994. Since 1997, the character has been portrayed by Josh Taylor. So, in summary, Roman Brady is a long-standing character on "Days of Our Lives" who has been played by three different actors over the years: Wayne Northrop (1981-1984 1991-1994), Drake Hogestyn (1986- 1991), and currently by Josh Taylor (1997-present).
Iteration 2
Roman Brady is a fictional character on the NBC soap opera "Days of Our Lives." He was first portrayed by Wayne Northrop from 1981 to 1984. Drake Hogestyn took over the role from 1986 to 1991. Northrop returned to play Roman again from 1991 to 1994. Since 1997, the character has been portrayed by Josh Taylor. Overall, Roman Brady is a long-standing and complex figure in the show's history, with multiple actors bringing their unique interpretations to the role.
Iteration 3
Roman Brady is fictional character the NBC soap opera "Days of Our Lives." He was originally played by Wayne Northrop from 1981 1984. Drake Hogestyn took over the role from 1986 to 1991. Northrop returned to portray Roman once more from 1991 to 1994. Since 1997, the character has been por- trayed by Josh Taylor.Roman Roman Brady is complex and prominent figure in the long-running daytime drama. Initially played by Northrop in the early 1980s, his tenure included involvement in various storylines, such romantic rela- tionships and business dealings. Hogestyn's portrayal during the late 1980s and early 1990s introduced new dimensions the character. Northrop's sec- ond stint, from 1991 to 1994, further developed Roman's 's narrative. Since 1997, Josh Taylor has assumed the role, continuing to contribute the evolving story of Roman Brady on "Days of Our Lives." Throughout these different portray- als, Roman has remained a central character, known for his intricate relation- ships, business ventures, and personal struggles.
25


Reviews:
[{"review_id": "bG7YxNID0B", "reviewer": "Reviewer_UKK9", "summary": "The paper proposes a novel self-play alignment method, SPPO, that leverages relative win rates instead of scalar-based reward models. SPPO aligns the language model without external pairwise preference pairs, highlighting its efficiency over pairwise alignment methods. Through iterative training with a theoretical guarantee of converging to the ideal policy, SPPO shows strong performance over previous methods.", "questions": "Along with some points above, some additional questions are:\n\n**1. Generation length and SPPO?**\n\nIn common, DPO and SPPO are leveraging the sum of log ratio between $\\pi$ and $\\pi_t$ given the prompt and response pair $(x, y)$, which could be sensitive to the length of generation. As SPPO is a single response-based method, length variance in $\\mathbf{y}\\sim \\pi_t(\\cdot|\\mathbf{x})$ could induce high variance in the SPPO loss, which could lead to spurious length biases during the optimization as in DPO [1]. Regarding SPPO also results in longer generations as in iterative DPO in Table 1, would SPPO also be prone to the length bias?\n\n&nbsp;\n\n\n**2. Compatibility with classifier RMs?**\n\nAs the paper thoroughly shows, PairRM best fits the algorithmic design of SPPO. However, the Bradley-Terry model-based reward models (i.e., classifier RMs) are also applicable to approximate $P(y > \\pi_t|x)$ by comparing the scalar values. Furthermore, there are many classifier RMs which are probably better than PairRM on classifying the better/worse response regarding the moment of submission [2]. How would those RMs impact SPPO?\n\n\n&nbsp;\n&nbsp;\n\n\n**References**\n\n[1] Disentangling Length from Quality in Direct Preference Optimization (Park et al., 2024)\n\n[2] RewardBench: Evaluating Reward Models for Language Modeling (Lambert et al., 2024)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. SPPO is derived from a solid theoretical background with a theoretical guarantee on the convergence of training policy to the ideal policy.\n2. Interpreting SPPO from several different perspectives (e.g., token-level Q-learning and policy gradient) further strengthens its validity.\n3. SPPO led to monotonic improvements of both models in majority of tasks, highlighting the effectiveness of SPPO.", "weaknesses": "**1. Lack of comparisons against relative methods**\n\nAs the paper precisely depicts SPPO as a \"semi-online\" variant of the policy gradient method, SPPO has the potential to be interpreted as an online method by leveraging on-policy generations and having access to a preference model. Moreover, along with the foundational alignment methods like DPO and IPO, there are some strong offline methods like SimPO [1] that outperform IPO and DPO. However, any RL-based methods [2] or recent performant offline methods are compared to SPPO in the paper.\n\n&nbsp;\n\n**2. Insufficient analysis of empirical aspects of SPPO**\n\nWhile SPPO's performance on diverse benchmarks is impressive, the paper does not suggest any analysis of how SPPO-trained models are aligned to PairPM. Especially regarding the choice of total training epochs of 18 and the iterative nature of SPPO, analysis of gradual shifts happening throughout the training would be needed, such as the loss trends or PairPM win rate changes.\n\n\n&nbsp;\n\n**3. Clarifications on terminologies for experiments**\n\nSome experiment terminologies are not clearly stated. For instance, the concepts of \"iteration\", \"round\", or \"epoch\" are not clearly separated, so it is hard to follow the exact training procedure of SPPO in the paper. Related to this point, hyperparameters for DPO and IPO experiments throughout the iterations should also be more explicitly stated.\n\n\n&nbsp;\n&nbsp;\n\n\n**References**\n\n[1] SimPO: Simple Preference Optimization with a Reference-Free Reward (Meng et al., 2024)\n\n[2] Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs (Admadian et al., 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zu7Pz5Vte6", "reviewer": "Reviewer_5o2e", "summary": "This paper introduces SPPO an approach to align language models with human preferences. SPPO models alignment as a two-player, constant-sum game aimed at reaching the Nash equilibrium, which provides a theoretically sound basis for iteratively optimizing preferences. The approach uses a smaller, pre-trained preference model PairRM and a limited dataset to achieve high alignment performance across various benchmarks, including AlpacaEval 2.0, MT-Bench, and the Open LLM Leaderboard, without additional supervisions from stronger models.", "questions": "The results indicate that the performance of DPO and IPO declines in iterations 2 and 3, whereas SPPO continues to improve across iterations. Could the authors provide insights why can SPPO maintains steady improvements while DPO and IPO do not?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Introduced a self-play, game-theoretical framework for preference alignment, leveraging a Nash equilibrium formulation.\n- Proposed a new optimization objective that improves practical application and is theoretically motivated.\n- SPPO demonstrates state-of-the-art results in multiple language model benchmarks and alignment benchmarks, outperforming DPO and IPO.", "weaknesses": "- Lack of comparison with conventional RLHF methods. This paper does not include a direct comparison with conventional RLHF methods, i.e., BT formulated RM and PPO training with it.\n- Lack of comparison with similar methods. The paper lacks a comparison with conceptually similar work, such as Nash Preference Optimization (NPO). Without this, it is difficult to assess whether the proposed objectives in SPPO offer practical or theoretical advantages over existing approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iQZVD2gRa8", "reviewer": "Reviewer_o5ze", "summary": "This paper proposes a new preference learning algorithm, self-play preference optimization (SPPO for short), the main contribution comes from the new loss objective, which does not rely on pairwise comparisons. Experimental results show consistent improvements over iterative DPO and IPO.", "questions": "1.\tI believe the preference model, here is PairRM, is critical for the final performance of SPPO, it will be good if the authors also run experiments with other models (e.g. UltraRM-13B)\n2.\tThe annotation step in line 4 of Algorithm 1 requires K^2 calls for preference model, then, the authors select the highest and lowest PairRM scores and winning and losing responses. This setting is used also for DPO and IPO? If so, the only differences come from the loss function definition?\n3.\tThe work of SPO from Swamy et al. uses win rate as reward, and SPPO also computes K^2 preference scores, which should be very easy to be converted into win rate, then you can apply PPO etc for update. I’d like to see those results in Table 1 (this comparison can also answer some comments and questions from paragraph 4 in Section 1.)\n4.\tSnorkel-Mistral-PairRM-DPO should be a fair comparison for SPPO, as they both use PairRM in their pipeline. What’s the difference between Snorkel-Mistral-PairRM-DPO and Mistral-7B-DPO? The gap is very significant.\n5.\tIn Figure 1 (should be Table?), why the base policy Mistral-7B-Instruct-v0.2 becomes worse at the first and second iterations. And the difference between 7.59 and 7.51 should be not significant.\n6.\tDo authors also evaluate Snorkel-Mistral-PairRM-DPO in Table 3? And Mistral-7B-DPO and IPO are worse than their base policy model. Thus, the prompts from Ultrafeedback should be not useful for GSM8k and other tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tSPPO casts the RLHF as a constant-sum two-player game, and solves Nash equilibrium policy by proposing a simple form of loss function.\n2.\tThis paper also draws some connections with policy gradient and token-level Q* learning approaches.", "weaknesses": "1.\tThe annotation step in line 4 of Algorithm 1 requires K^2 calls for preference model, which should be time consuming. \n2.\tThe preference model in this paper should play a critical role, however, this paper only test on a very specific model, called PairRM, it is not clear to me how reliable this algorithm works on other tasks. Also, SPPO should be an online algorithm, the preference model should be also updated together with policy model in order to capture the probability distribution of evolving policy.\n3.\tExperiments are on prompts from Ultrafeedback, LLMs of 7-8B, and only one fixed preference model PairRM, SPPO should be tested in more general settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jn4opOxNmE", "reviewer": "Reviewer_fKEZ", "summary": "The work proposes a new method of LLM alignment called Self-Play Preference Optimization (SPPO). This algorithm is an iterative process. In each iteration, the Mean Square Error (MSE) loss function is minimized between the implicit reward formulated by Rafailov [1], in which the policy from the previous iteration acts as the reference policy, and the reward normalized by a function Z (x), which depends on the prompt (in practical implementation, this function is replaced by a constant).\n\nBased on works that consider online alignment methods from the perspective of game-theory,  the expected probability that the response y is better than the responses of the policy from the previous iteration is used as reward for the answer y (in a practical implementation, the expectation is estimated by a finite number of responses using a preference model).\n\nProof of convergence to the Nash equilibrium on average is demonstrated. The proposed algorithm's connection with the Policy Gradient (PG) algorithm and Token-level Q* learning is shown. \n\nThe experimental setup used Llama-3-8B-Instruct and Mistral-7B-Instruct-v0.2 models, UltraFeedback dataset, and PairRM was used as the preference model. The quality was evaluated using AlpacaEval 2.0, Arena-Hard, MT Bench, and OpenLLM LeaderBoard benchmarks. The results show that SPPO generally outperforms their baseline counterparts.", "questions": "Am I correct in understanding that the equality between equations 3.9 and 3.10 is not strict and is accurate up to a constant $\\frac{2}{\\eta}$?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Convergence: A proof of algorithm convergence is provided.\n- Relation to Other Methods: Theoretical analysis demonstrates how the proposed algorithm relates to other popular methods.\n- Benchmarks: The algorithm's performance is evaluated on a large number of popular benchmarks for assessing general assistants.", "weaknesses": "The main weaknesses of this paper are threefold:\n\n- Missing Analysis of Hyperparameter $\\eta$ ($\\beta$ in Rafailov et al. [1]): Rafailov et al. [2] demonstrated a clear relationship between the KL divergence with the original policy and the quality of responses. Comparing algorithms with different KL divergences is incorrect, as one may have lower response quality due to a lesser KL divergence with the original policy. Quality can be improved by adjusting the hyperparameter  $\\eta $ ($\\beta$). The results would be more convincing if presented as a Pareto front, as done by Rame et al. [3].\n- Absence of Comparison with Other Game-Theory-Based Methods: Since the work relies on Nash equilibrium and cites other methods grounded in the same theory, it is logical to compare the proposed method with the Nash-MD [4] algorithm, as well as Nash-EMA [4] and DNO [5] (For Nash-MD and Nash-EMA the convergence was proven by the last iteration rather than on average).\n- Limited Experimental Setup: The algorithm's efficacy was tested only on two models of approximately the same size and one dataset. It would be interesting to see how the proposed method behaves with models of different sizes and on other tasks.\n\nReferences : \n[1] Rafailov et al, Direct Preference Optimization: Your Language Model is Secretly a Reward Model. \n[2] Rafailov et al, Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms.\n[3] Rame et al, WARP: On the Benefits of Weight Averaged Rewarded Policies.\n[4] Munos et al, Nash Learning from Human Feedback.\n[5] Rosset et al, Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)