Self-Improving Robust Preference Optimization
Authors: Eugene Choi
Arash Ahmadian
Matthieu Geist
Olivier Pietquin
Mohammad Gheshlaghi Azar
Venue: ICLR 2025 Poster (2025)
Keywords: Preference optimization
direct alignment
reinforcement learning from human feedback
Self-refinement
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Online and offline $\mathtt{RLHF}$ methods, such as $\mathtt{PPO}$ and $\mathtt{DPO}$, have been highly successful in aligning AI with human preferences. Despite their success, however, these methods suffer from fundamental limitations: $\mathbf{(a)}$ Models trained with $\mathtt{RLHF}$ can learn from mistakes or negative examples through RL mechanism or contrastive loss during training. However, at inference time, they lack an innate self-improvement mechanism for error corrections. $\mathbf{(b)}$ The optimal solution of existing methods is highly task-dependent, making it difficult for them to generalize to new tasks. To address these challenges, we propose Self-Improving Robust Preference Optimization ($\mathtt{SRPO}$), a practical and mathematically principled offline $\mathtt{RLHF}$ framework. The key idea behind $\mathtt{SRPO}$ is to cast the problem of learning from human preferences as a self-improvement process, mathematically formulated as a min-max objective that jointly optimizes a self-improvement policy and a generative policy in an adversarial fashion. Crucially, the solution for this optimization problem is independent of the training task, which makes it robust to its changes. We then show that this objective can be reformulated as a non-adversarial offline loss, which can be efficiently optimized using standard supervised learning techniques at scale. To demonstrate $\mathtt{SRPO}$’s effectiveness, we evaluate it using AI Win-Rate (WR) against human (GOLD) completions. When tested on the XSum dataset, $\mathtt{SRPO}$ outperforms $\mathtt{DPO}$ by a margin of $\mathbf{15}$% after $5$ self-revisions, achieving an impressive $\mathbf{90}$% WR. Moreover, on the challenging Arena-Hard prompts, $\mathtt{SRPO}$ outperforms both $\mathtt{DPO}$ and $\mathtt{IPO}$ (by $\mathbf{4}$% without revision and $\mathbf{6}$% after a single revision), reaching a $\mathbf{56}$% WR against against $\mathtt{Llama-3.1-8B-Instruct}$.

Published as a conference paper at ICLR 2025
SELF-IMPROVING ROBUST PREFERENCE OPTIMIZA- TION
Eugene Choi Cohere
Arash Ahmadian Cohere for Al
Matthieu Geist Cohere
Olivier Pietquin" 1 Cohere
Mohammad Gheshlaghi Azar Cohere
ABSTRACT
Online and offline RLHE methods, such as PPO and DPO, have been highly success- ful in aligning AI with human preferences. Despite their success, however, these methods suffer from fundamental limitations: (a) Models trained with RLHF can learn from mistakes or negative examples through RL mechanism or contrastive loss during training. However, at inference time, they lack an innate self- self-improvement mechanism for error corrections. (b) The optimal solution of existing methods is highly task-dependent, making it difficult for them generalize to tasks. To address these challenges, we propose Self- lelf-lmpooving Robust Preference Optimiza- tion (SRPO), practical and mathematically principled offline RLHF framework. The key idea behind SRPO is to cast the problem of learning from human prefer- ences self-improvement process, mathematically formulated objective that jointly optimizes self-improvement policy and generative policy an adversarial fashion. Crucially, the solution for this optimization problem is independent of the training task, which makes it robust to its changes. We then show that this objective can be reformulated as a adversrarlli offline loss, which can be efficiently optimized using standard supervised learning techniques at scale. To demonstrate SRPO 's effectiveness, we evaluate using Al Win-Rate (WR) against human (GOLD) completions. When tested on the XSum dataset, SRPO outperforms DPO by a margin of 15% after 5 self self-revisions, achieving an impressive 90% WR. Moreover, outperforms both DPO and IPO (by 4% without revision and 6% after a single revision), reaching 56% WR against against 1.lam-- 3.. 1-88--8BB-- Inst ruct.
min-max
the challenging Arena-Hard prompts, SRPO
INTRODUCTION
Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) has quickly become a standard method to align large language models (LLMMs). key advantage of RL.F that allows the model to learn from its mistakes, allowing to not only learn from good experiences but also from had experiences and mistakes. Despite being able learn from mistakes, the standard atneettued models are not equipped with innate mechanism to self-improve and correct their mistakes at the time of inference (Kumar et al., 2024). a model makes a critical mistake or omits crucial fact during inference, it struggles to recover. Another practical issue that all the prominent RLHF methods (offline or online) (Ouyang et al., 2022; Rafailov et al., 2023; Azar et al., 2023; Zhao et al., 2023b; Ahmadian al., 2024) encounter that their optimal solution heavily depends terms of the distribution used to generate preference data (behavior policy) (Munos et al., 2023; Azar al., 2023). This makes existing RL.F methods vulnerable the cases in which the evaluation distribution of completions is significantly different from that of the behavior policy (Li et al., 2024b; Kirk al., 2024).
the training task in
To address these challenges, we introduce an alternative approach for aligning LLMs from human preferences based on more principled and robust foundations. Our goal to find solution that is
"Conresonding Authors: (eugene, moh mohannad @cohere.com 1 Research done Cohere. Now Earth Species Project.

---

Published as a conference paper at ICLR 2025
SRPO XSum Test (OOD) Results
(WRom +15.2)
(WRoro + 14.9)
(RRopo+ +446)
Win Rate (%)
88
133.3)
(WR 13.0)
(WRoro++ 12.1)
87
86
(WRooo+ +005)
1
2
3
4
5
6
N-revision
Figure 1: Self-improvement by SRPO in terms win rates against human (WR). We demonstrate robustness by training on TL;DR and evaluating on XSum. Gains on Direct Preference Optimization (DPO) are reported text captions.
robust to the changes in the preference dataset, meaning that changes in the distribution from which the completions are sampled do not affect the final outcome of learning significantly. To achieve this goal, we exploit the concept of self-improving (Huang et al., 2022; Bai et al., 2022) language models. By self-improving LLM we refer model capable of enhancing its outputs recursively with each inference iteration. Our Self-Improving Robust Preference Optimization (SRPO) consists of two back-to-back optimization processes:
(Step 1) In-Context Self- Improvipg Preference Optimization: The core idea is to learn an in-context self- improvipg model ,ę: given an in-context completion y and a context z, the self- improvement model, nf. outputs an improved completion y' with probability ग [y/"ly x)) from which sampled completions are most preferred to completion y according to the human preference model As explained later, it turns out that this problem, in its KL-regularized form, can be expressed as a well-defined preference optimization problem and solved analytically. Furthermore, the solution can be estimated through supervised direct preference optimization scheme similar to the approach used by Rafailov et al. (2023) and Azar et al. (2023).
(Step 2) Robust Preference Optimization of Generative Model: The next step is to exploit the self-impree cell-improvement policy learned in the previous step to learn generative LLM, т. The key idea here that the best generative policy can be identified as a policy that generates completions requiring minimal improvement using the optimal self-improvemmet policy т derived in step 1. This goal can be achieved by minimizing the objective of step 1 with respect to the generative policy for in-context completions, Similarly to step 1, this problem, in its KL-regularized form, can also be solved analytically in terms of the optimal improvement policy ग and the optimal generative policy m. More significantly, we show that the solution for steps 1 and 2 can be estimated jointly through single supervised direct preference optimization scheme using only a dataset of annotated pair- wise completions. Thus, one solve both for the self-imm inffimmproeeentt policy π; and æ by minimizing the supervised learning objective of SRPO. Unlike existing RLHF methods, this solution is independent of the behavior policy and is therefore robust its changes.
As using the self
model in SRPO is a significant departure from the existing paradigm for RLHF, we provide high-level motivation for it in Sec. 2. We then formalize our objective for SRPO in Sec. 3, allowing for the joint optimization of both ग, and π by optimizing an adversarial min-max objective. Sec. 4 we present our main algorithmic/mathematicl that the preference probability can be expressed in terms of the likelloos of the optimal self-imprrvemmen policy лद and the log-likelisoods of the optimal robust generative policy л². This
contribution: we prove
From now on, generative L.M will be considered equivalent to distribution policy from which sample completions y with probability (y][r), where x the context or prompt.
2

---

Published as a conference paper at ICLR 2025
theoretical finding is the key result for SRPO: solving this system of equations through ceastesquares regression provides us with the practical supervised SRPO objective that solves for both policy and robust generative policy through single supervised objective without any need for reward model or online inference. Our key theoretical finding is similar to the main result of DPO (Rafailov et al., 2023) in that both express preference prohabilities in terms of the optimal policy. However, DPO result only holds when preference probabilities conform to Bradly-Terry model (Bradley & Terry, 1952), whereas our key result is general as it holds across all preference models. In Sec. we further illustrate our argument on the robustness of SRPO by providing an in-depth analysis of the solution of SRPO and other direct preference optimization methods. We also showcase/analyze the robustness of SRPO on simple synthetic example. Finally in Sec. 6 we conduct large- scale experiments on training LLMs with SRPO both on in-distribution and OOD summarization tasks and we compare the results with those of standard baselines.
2 LEARNING SELF-IMPROVEMEN POLICY THROUGH PREFERENCE OPTIMIZATION
The goal of this section is motivate why learning self-improvement models through preference optimization can be useful for training a GPT-style language model. One of the main challenges that affect the performance of sequence-to-sqqeene from failure at the time of inference. This problem, which is known as exposure bias in the literature on sequence-to-See -o-suuueeee prediction (Xu et al., 2020; Arora et al., 2023; Bengio et al., 2015), is because at the time of inference, GPT-style models generate tokens step by step based on their own previous outputs, not the ground truth. This can lead to error accumulation: if the model generates an incorrect token early in the sequence (or misleading token that may lead to error later down the line), it may propagate errors since its subsequent predictions are based on the earlier incorrect (or misleading) token (Arora et al., 2023).
GPT-style models is that they are hard recover
To address this shortcoming of GPT-style models, one may fine-tune the model such that it can revise its incorrect generations through seff-improvementssfff seeffeieettstrrreeeeeeettt process (Madaan et al., 2024; Hu et al., 2024; Huang et al., 2022; Bai et al., 2022). The simplest way to train such a self-improvement model is through supervised fine tuning (SFT) (Bai el al., 2022; Wei et al., 2023). To train self- improvement model through SFT, one first need to build a supervised dataset of (I, y, v²) in which some initial completion than can be correct or incorrect and y" the corrected version of y. The improvement model can then be fine-tuned using SFT to predict y" from the pair (I, y).
However, training a SFT self sif-imprrvemmnt model a challenging task for the following reasons.
1. Creating a supervised dataset of (x, y, y²) is resource intensive and hard to scale (Huang et al., 2022). Also, the existing standard SFT datasets are not created in this way.
2. Moreover, the SFT training pipeline for self-improvement can simply overfit to predict the correct answer y" only from the context x and ignore the initial answer altogether over-fit to specific refinement pattern (Li et al., 2024a). Thus it may not learn properly how to improve an incorrect y to the correct y"
Alternatively, one might ask whether one can generalize the direct preference optimization methods (c.g., Rafailov et al., 2023; Azar et al., 2023) for learning self-imprrvemmnt models using the standard human preference datasets. To answer this question we first try to answer a more fundamental question:
What the best use of human preference data?
To address this question, we observe that human preferences inherently encode valuable information about the relationship between more-preferred and less-preferred completions. This relational information can be leveraged to refine less- preferred completions, guiding them closer to those that better align with human preferences. In essence, this process involves learning a model of alignment mechanics-prinnine
or "rules" that dictate how completions can be iteratively improved in terms of satisfying human preferences. This approach represents a more natural and intuitive learning task compared to directly predicting the tughest-prefente RLHF methods. Directly learning the optimal completion can be a challenging task, particularly when the completion space encompasses the full complexity of human language. Furthermore, the
completion, which is the focus of standard
3

---

Published as a conference paper at ICLR 2025
highest-prefeneet answer is unlikely to exist in the dataset, especially when completions generated by large language models (LLMs) that inherently fall short of human-level quality.
Instead, it is more effective to model the improvement process itself: given query and an initial complction y, the goal is to predict an improved completion y'. This shift in perspective transforms the task into learning how to enhance subpar outputs from an LLM, rather than aiming to directly generate perfect responses. By capturing the underlying principles of human preference, the model can iteratively refine suboptimal completions y, progressively steering them towards the ideal outcome y² This framework not only aligns with the iterative nature of human reasoning but also facilitates continuous self improveereeettt making it robust and scalable approach for enhancing the performance of LLMs. In the following, we show how the self improvement policy, Ty, is trained alongside the generative policy, n, using pair-wise preferences.
SRPO OBJECTIVE
We start by introducing some notations required to establish our theoretical results. Let æ and y denote context and completion drawn from the space of all possible contexts X' and all possible completions V. respectively. The large language model (LLM) represented by the probability distribution (policy) ı where π [z)22) denotes the probability of completion y given the context x. In the remainder of this article, we consider three variants of this base LLM, the trainable model Tiain (for which we use the shorthand r²), the reference model Tref and the behavior model from which the completions in pairwise preference dataset is sampled.
We also introduce self-improvemmet T (s/lly x) as model that using context and in-context completion (thought) aims at improving y to better completion y'. Similarly to the base LLM, we can define reference model ( x)) also for the self- self-improvement model. Let D {(x, y, y')} be a dataset of contexts and completions where y and y' are drawn independently of p(-[z)) We then present every pair y: y to human annotators who express preferences for one of the completions, denoted as You > Yi where Yor and y denote the preferred and dis-preferred actions amongst (v. y' respectively. We then write the true human preference p(y' > y|ır) the probability that y/ is preferred to y knowing the context x. The probability comes from the randomness of the choice of the human we ask for their preference. So p(y' y/æ) E, [1 (h prefers y' to y given 2}}, where the expectation is over humans h.
Consider a reference policy Tet and a real positive regularization parameter B E R Then, we define the Self-Improvin Robust Preference Optimization objective (SRPO) for every context as
J (x) = min max
ply' > y|æ) - BD (TT (Terll))) + + (x((
(1)
(.(z))
Corry (-ly.z)
with the KL-regularization terms are defined as: DKL. ( and ( [ = (L(x(-[x)] (=)))
XL(T++(-ll. |)T[ x))
In nutshell, this objective aims at (i) finding the best seff-improvement policy कं that improves every y optimally w.r.t. the preference distribution p, i.e., the improved policy is most preferred to y, while keeping गदद close to the reference policy Trefs (ii) minimizing the same objective to find the best (robust) policy for which the generated completions can be only minimally improved by the optimal sef-iimprovement model पद. The min-max nature of this objective makes self-improvement effective for all policies close to Tea as we optimize ny in the worst-case scenario.
4 OFFLINE SOLUTION FOR OPTIMIZING SRPO OBJECTIVE
In this section, we show that the min-max objective of Eq. (1) can be transformed to non-adversarial offline supervised loss. Thus it can be optimized at scale using standard optimization techniques.
4.1
MAIN RESULT
The optimization problem of Eq. (1) is a non-trivial optimization problem that often requires solving two-stage adversarial optimization problem through the game-theoretic approaches, which are often
4

---

Published as a conference paper at ICLR 2025
challenging and difficult to scale up, (see e.g., Munos et al., 2023; Rosset al., 2024; Calandriello et al., 2024, for how we can use game-theoretic approaches/objectives to train LLMs). Here, inspired aim at casting this complex optimization objective as
by Rafailov et al. (2023); Azar et al. (2023), a standard supervised learning problem that can be solved at scale given an offline pairwise preference dataset. The main theoretical result of our work is then follows
Theorem 1. Given context z and the behavior policy u(-lz)) let ply(æ) and scalars [0. 11, > O we have that the solution of the min-max objective of Eq. (1) is obtained by minimizing the following loss
La(T, =(1) (1) mT) aL)(
(2)
where L(T, T²4 and (x+ ) are defined respectively as follows
T+ TT(''lU, log
T (4(1,,
]]
(TT))
E M. mu(tta) xmpp
piu' - v/z)
2 - log
Trr((""),,))
TTy (yly. x)
(3)
and
B
T (2'11:
T (g)la))
L TT)
E v.y/ mp))- z<pp
ply' y/z)
log
+ log
(4)
2
Tier (vrl,
TTr (r)
T , x
T (z) Taylys (x:)
log
log
4.2 PROOF OF MAIN RESULT
To prove the main result we first notice that the inner- inner-maximizttonn in the objective function of Eq. (1) is an instantiation of KL-regularized RL (Todorov, 2006). Thus it can be solved in analytical form and its solution is given by
exp
ply'r yla)) 1
Trer( (w,
क| (1/b.zz)
(5)
Z Z (yy, x)
where zyy is the normalization factor. One can easily show that by plugging कद in the objective function of Eq. (1) we obtain:
(x)x) min (.))) [B(log(2" (v, DEL (TT) [Te [))]
(6)
Now by solving Eq. (5) with respect to p(y' > u| we obtain
ply' u| ((log(T] (v'ly,x)) Log(T y, x)) + (g,(z((),))))
4.2.1 OPTIMIZING THE SELF-IMPROVEMENT POLICY I
We now prove that minimizing the objective of Eq. (3) gives us the optimal self eeff-imprvvement policy (solution to maximization in the objective of Eq. (1)). We first notice that using the convention
ply > y/ = 2 Eq. (7) implies
= (log(( (wly, x)) log(meer Tier (yle,, + 3 log(Z" x)).
(8)
Now by subtracting Eq. (7) from Eq. (8) we derive
क :F(U'l..)) (0u'l..2) (rerry 2x)
(7 x)
p(y' > v/ = 3 log
log
(9)
Teerl (1) y,
This is our first key result that expresses preference p(y' y|æ) in terms of the optimal self- improvement policy πद. So we enforce this equation for all y and y through minimizing & loss of Eq. (3) which concludes the proof for the self-improvement policy.
5

---

Published as a conference paper at ICLR 2025
4.2.2 JOINT OPTIMIZATION OF THE ROBUST GENERATIVE POLICY AND THE IMPROVEMENT POLICY Tt
In this section, we show that to optimize the min-max objective of Eq. (1) (i.e., solving for ा and TT) one can optimize the loss of Eq. (4). We start by collecting terms in Eq. (8) which implies (1g (y, (log (ee (y) y, x)) log(Tī ( (yly, z))) 2.2 Thus, the objective of Eq. (6) can
be expressed in terms of log( q (vly. x)) (up to an additive and multiplicative constant) as follows:
(x) x min E (1
log
+ DKL (x||
(10)
Solving this objective with respect to л we obtain:
Merr( 7 (vly. x)
Fnd (x)
ा (yyla) (1) =
(11)
(x))
where Z (xx) is the normalization factor. Again by taking the logarithm from both side obtain
Log(TT((v/ (v(()) = log
Tie (vll))))))) # (vlv,, - (x)(())))) Now by collecting terms in Eq. (7) and
solving for log (r (v/) (lly, (1, x)) we obtain
log(#; (v' b. x)) = p(y' > yla)
log (y, (y, - log (Teer (y' x))
(12)
Now by plugging Eq. (5) into Eq. (11) we deduce ~" (y|x) expl- log 2 (2, (av((le))
Solving this
(x)
equation with respect to log( 7 (y, x)) implies
log(Z ("' (y, x)) = (og(Tee((y)a:)) - log(T" log(T"(vl=)) - (x))..
(13)
Combining Eq. (12) and Eq. (13) for any y and y' we have p(y'>ul: [)xz)
x (v/ (y,.))
log
(Me [yx=]
log
[ud( (vl loogooZ I (z)),, and also 2(w>y'|m) - log (vle/ = log <ealvlss
Tv (v/[x)) (ar) (s)
loo (x)).
x
Subtracting these two equations and collecting terms leads our key result in which we express the preference in terms of the self-improvement policy oद and the robust policy n².
1
x (ulor)
plu > y/2)
=
2
+
2
log
TTer(y) (w,x)
+ log
(14)
-(10g log
क. (ull'', 2))
π (y/ (z)
+ log
Tree (v' (x))
One may notice the similarity of this result and Eq. 6 of DPO paper (Rafailov et al., 2023). Both results express p(y' > y| in terms of the optimal policy π². However, the result of DPO only holds under the assumption that p conforms the Bradly-Terry model, whereas our result is general and holds for all p.
Remark 2.
To optimize for and using Eq. (14) we enforce this equation for all and y/ through minimizing the expected Eg loss of Eq. (4) which concludes the proof on joint optimization of the generative policy π and the self self-imm infrooerroetttt policy through Eq. (4).
4.2.3 PUTTING ALL. TOGETHER IN SINGLE COMBINATION LOSS
We observe that the losses defined in Eq. (4) and Eq. (3) are inherently aligned, as both aim to optimize the same overall objective given in Eq. (1). This alignment allows us to construct a combined loss function for SRPO by using a convex combination of these two losses. Specifically, for any a E [0, 1). minimizing the convex combination of the losses in Eq. (4) and Eq. (3) is equivalent to directly optimizing the objective in Eq. (1). Mathematically, the full loss of SRPO can be expressed as:
LaReo = a LL2 + +1 (1- - a) Lyelt-imrooe.
where LLa corresponds to the loss in Eq. (4) and Leell- corresponds to the loss in Eq. (3). By minimizing this combined loss, we ensure that the optimization process aligns with the original objective in Eq. (1) regardless of the value of
This formulation not only unifies the two loss functions within a single framework but also provides flexibility in adjusting their relative contributions during training. This completes the proof.

---

Published as a conference paper at ICLR 2025
4.3 DERIVING THE SAMPLE LOSS FOR SRPO
Calculaing/optimizing the expected loss of Eq. (17) is often not practical as we often have no direct access the behavior policy μs. Moreover, the space of completions y is often very large (even infinite). So calculating the full expectation is often impractical. Instead one can estimate the expected loss La of Eq. (17) using a preference dataset 'D with a sample loss La and optimize this loss instead. We now focus deriving the sample loss La for SRPO. Using the standard properties of & 2--momm to replace ply' > y/æ) with 1 > y/æ)). as ply' > y|æ) = Efj > y|z)], in the objective of Eq. (4) allows to derive the following sample loss for the improvement model:
TT T+ Bit x))
Tt (Hu
[T+ -E Mar Mo.. DD
B log
log
(15)
[rer(3/u Mee [21t xz)
Teer(1/ (yt,
(y (Bhus 2:)
mh (30) (B0l x))
2 - ,3 log
log
(Ber (ur x)
Teer(1) ( x)
Using the standard properties of f2-norm to replace p(y' > y y|ar) with 1
y/z) as
ply > y/z) Bf > y/2=)], in the loss of Eq. (4) allows us to derive the following sample loss for both the generative policy and the improvement model:
ST (T (311)) + log
π [y (z) Trer( xx)
[(Z, T4) -B Bw. log
(16)
-(105 (1 (1rl (eho;
(gr yer ( xx) ) log
(v)lar) )) (
looseness=-1 Furthermore, one can use single LLM (denoted by σ) to represent both and Ty by exploiting the in-context learning power of LLMs (Brown et al., 2020) such that my (y/ /l,y x) = π (t ly, æ). So for we define the full sample loss of SRPO as the convex combination of Eq. (16) and Eq. (15):
-
x)
a = =).
(17)
The following pseudo-code can be used to train the LLM policy using SRPO objective:
Algorithm Sampled SRPO
Require: Dataset D prompts, preferred and dis-preferred generations x, Yo and y/, respectively. A reference policy and a training policy no. regularization coefficient and combination coefficient a.
1: Initialize TT Tee
2: while true do
3: Sample minibatch E
4: Estimate Va (TO) from Eq. (17) using minibatch B the dataset
5: Update ne using TaLa(m)) using a standard optimizer
6: end while
7: return Te
ROBUSTNESS OF SRPO
We provide a comparison between SRPO and prior work on direct preference optimization in terms of their robustness to the behavior policy μ.. In particular, we consider as a point of reference DPO (PPO?) and IPO for which we have a good understanding of the underlying mathematical foundation.
In the case of both IPO and DPO the analytical solution already well-established and analyzed for both algorithms (Azar et al., 2023; Rafailov et al., 2023; Tang et al., 2024). In particular, the optimal
As it is shown by Azar et al. (2023) the optimal solutions of DPO and PPO are identical. So in the remainder of this section focus on DPO.
7

---

Published as a conference paper at ICLR 2025
solution for both IPO and DPO can be expressed explicitly in terms of the soft-mat of the expected preference as follows (Azar et al., 2023):
x (yl exp (B E ( [p(( 3 [)]]) (1)()))
(18)
(=)x)
with the choice of V = I(-) and VI W (.) for IPO and DPO, respectively, where a_! denotes the inverse-sigmoid (logit function). Thus, based on Eq. (18). we can see that the solution for both IPO and DPO has strong dependency on in the form of expected preference under the distribution Thus it may not be robust changes in j.. This dependency on f can be especially problematic when we evaluate the model on out out-of-distribution tasks where the desired behavior is very different from and the expected preference under the distribution μ is not good measure of performance. SRPO so- lution on the other hand has no dependency on the behavior policy μu: from Eq. (5) we observe that the optimal self- inff-improveentt policy a is independent of μ and, unlike DPO and IPO cases, is expressed in terms of softmax of p(y' > y u/ for any pair of completions (y. 1/'). Also the O-revision policy x is also completely independent of μı as it is evident from Eq. (11) (i.e., it is proportional to x² *"yly,, 2) which itself is independent of p)) Thus, from a mathematical point of view, SRPO provides a robust so- lution for the problem of direct preference optimization that does not depend on the behavior policy μ. To illustrate the differences between SRPO and DPO/IPO regarding robustness to J4, we consider simple bandit example. For simplicity, we assume there is no context x. Consider the simple case where we have 3 completions yo. y1, and 1/2, for which the preference model is
0.5
0.99
0.3
p=
0.01
0.5
0.25
0.7
0.75
0.5
To test this hypothesis we consider two synthetic dataset of actions generated from distributions No and #1: We set μo to be a uniform behavior policy (Ho(3o) = (0( = { = 11/3) and [i skewed towards y1 (91) = 0.7. ni1 (Wo) = n41 (3/2) = 0.15). We then generate a dataset of 10, 000 pairs from (An and jr and rate them according to the preference model p (for any pair (v. v/) we assign the preference by sampling from p(y > y²). that is y is preferred to y' with probability p(y > y²)). This provides us with two dataset of rated completions Do and Di for pn and μu, We then use these two datasets to train the policy π using SRPO, DPO and IPC using a simple Adam optimizer. In the case of IPO and DPO we optimize only the 0-revision policy I (t)) where for SRPO we also optimize the self- self-immpovemmnt policy π (gyly') as well. We set the regularization constant 33 for all methods to 1. We consider uniform distribution Tyer(y) 1/3 for all algorithms and all ys. In the case of SRPO we set the self-improvement reference policy mer(yly') = 1/3 for all y and g'. Also for SRPO we set the combination coefficient a = 0 for simplicity.
We observe that in the case of using uniform oo as a behavior policy, all methods perform as expected, converging solutions where 1/2 dominates y and yo (Fig. 2a). However, when the behavior policy N4 is skewed toward y/. both DPO and IPO converge to solution where yo dominates y11 and y2. while the policy of SRPO remains unchanged (Fig.2b). Notably, the SRPO policy differs slightly between the two cases, which is expected given the finite data setting, where the sampling distribution influences the empirical preference model." " However, when we use the behavior policy f4, which is skewed towards V/. both DPO and TPO converge to a solution where Yo dominates y11 and y2- while the policy of SRPO remains unchanged (Fig.2b). Notably, the SRPO policy differs slightly between the two cases, which is expected given the finite data setting, where the sampling distribution influences the empirical preference model."
to solution in which Yuu dominates y/1 and y2, while the policy of SRPO remains intact (Fig. 2b). Notice that the SRPO policy is slightly different in both cases. This is to be expected, we are in a finite data setting, and the sampling distribution will have some influence on the empirical preference model.
50PD
020
2370
IP0
(a) SRPO vs IPO and DPO for uniform Poo (b) SRPO vs. IPO and DPO for skewed p1.
Figure 2: Learned action probabilities for the synthetic example. SRPO always chooses the correct arm regardless of skew in while both TPO and DPO are effected by the skew (Fig. (2b)).
8

---

Published as a conference paper at ICLR 2025
6 EXPERIMENTS
Setup. In our experiments, we consider the offline direct preference optimization setup to learn from human preferences (Rafailov et al., 2023). In the offline setting, the goal is train the LLM policy directly from dataset D of pairwise completions (yt, sampled from a behavior policy i and annotated by human raters without using a reward model or online inference/RL. We empirically test the effectiveness of SRPO against two prominent offline preference learning methods, DPO (Rafailov et al., 2023) and IPO (Azar et al., 2023) as baselines. We make this choice since both these baselines, have been widely used in solving different language tasks (Tunstall et al., 2023; Wallace et al., 2023; Yuan et al., 2024a; Pang et al., 2024a; Lin et al., 2023).
Implementation details. SRPO trains simultaneously both the standard generative policy and the self-improvement policy क, used for revising the completions of models through a single optimization process. As explained earlier only use single LLM to represent both and и (denoted simply by T). To get the best completions from SRPO we first generate completions in the O-revision (0-rev.) model and then we improve these completions with the self-impprvemmnt model. We call the revised outputs I-revision (1-rev.) completions. We can iterate on the improvement process N times to get N-revision (N-rev.) completions. We report results from O-rev, 5-rev. cases. For IPO and DPO we also report results on O-rev. and 1-rev. For revising the completions we use IPO and DPO in in-context learning mode with the O-rev. completions used as contexts. In the case of DPO we use the same loss and
used by (Rafailov et al., 2023). For IPO since the original paper hasn" 't we used a set of parperrrrreeters (i.e., learning rate and regularization constant 3) from the range of hyper-parametert that was working. Furthermore we noticed that the performance of IPO was not affected significantly by the choice of these hyper-parameters. So no significant gain is expected by arper-paraettrr tuning.
provided the syper-parameete
Datasets. We use the Reddit TL;DR Summarization dataset (Stiennon et al., 2020) as the main dataset for our experiments" For training, there are 116k human written instruction following examples with reference completions (SFT split) while there are 93k -anaata-d preference pairs (Preference split). We also use the XSum dataset test split* (Narayan et al., 2018), which contains 11.5k total test examples to measure Out-of-Distribution (OOD) generalization.
Model Setup. We use LLaMA-7B base model (Touvron al., 2023) and single 8 x NVIDIA H100 node to conduct all LLLaM-bassdd experiments. We first SFT the model on the SFT split of the TL;DR dataset, before preference training and use the same 7e for all preference training experiment. Below are details on the recipe for the SFT and preference training stages.
Supervised-fing Tuning. In the SFT stage, we train for epochs, using the AdamW optimizer (Loshchilov & Hutter, 2019), with 3n = 0.9 and B3 = 0.999, and 0.1 weight-decay. We use a cosine decay learning rate (Loshchilov & Hutter, 2017) with a peak value of 10 and 3% of all steps being warm-up up steps. We use an effective batch-size of 64.
Preference Training. We use our SFT model as गाe and we initialize ' with Tet. All models were trained for 5 epochs on the TL;DR preference split using the same optimization setting of the AdamW optimizer as in the SFT stage with 150 warmup steps, and an effective batch-size of 128. To fine-tune the models, we use the default PEFT settings in the TRL library', using LoRA (Hu et al., 2022) with a rank of 16 and an alpha of 32. For SRPO and IPO, we used 3 = 0.01 with a learning rate of 2 x 10- For DPO following Rafailov et al. (2023), we used the common 3 0.1 with a learning rate of x 10 6 and a constant learning rate schedule.
Evaluation. We use win rates computed by pppt4-0611 (OpenAI, 2023) using the Alpacafarm framework (Dubois et al., 2024), as the main means for evaluation (See Sec. C.3 for more details). both in-distribution and OOD examples at test time in the following
We measure performance manner: For the former, we compute win rates against gold reference completions from the test set of the TL:DR SFT split. For the latter, we measure win rates against gold completions from the test set of the XSum dataset. In both settings, we use the first 1,024 samples from each of the test sets. To estimate the win rate more accurately with confidence intervals, we bootstrap 20 times with replacement from the 1,024 samples, each time using sample size of 512. To sample from the
'https: h//g:////i// thub. wtw openai iateenaiissmmmaa wwt.sssuuaaeeeeeeeeeeeeea fron- feedback 'nttps:/ http://hhgggggaaegee co/ dat.sattaeess//eaeaaoaa wetttttsets//eeebueeette wttpsstnnbeeceeeeeecceect hpspueetepppplmmmmmm sum 'https: thub. com wtt./coooocoocccccccc com/huggingfaac//tn
9

---

Published as a conference paper at ICLR 2025
self- celf-improvement policy, first sample y from ((-]2)). Then, using the same policy, condition on y to sample from the impfovmmreee policy, that is y ~ I (-11y.2) x). We refer to generations from ((+(a) O-rev. (0-rev.) and generations from 3 (-1 g,, 2) as I-rev. (1-rev.) (Bai et al., 2022). For N-revision, apply the same procedure, conditioning on the sample from the N-1
TL;DR Results. We test our models on the test set split of the TL;DR dataset in Fig 3 (left panel). For every model, we generate O-rev. and then use these generations revise our completions recursively from 1-rev. to 5-rev. using the self-improvement model, and measure the models' win rate against the human-written gold reference summaries.
We observe that in the case of in-distribution TL;DR SRPO 4-rev. generates high-quality summaries with the highest win rate against the gold summaries, compared the win-rates of of the baseline methods, as well as other variants of SRPO O-rev. Furthermore, we observe that SRPO eeff-improvement process manages to consistently improve upon SRPC O-rev. However, DPO and IPO fail to generate improved sample through the self-improvement step.
nut-of-distribution (OOD) Results. To assess robustness in an OOD setting, we test SRPO models trained with TL:DR preference dataset on the XSum test split in Fig. 3 (right panel) (Narayan al., 2018). As in the TL;DR case, we observe that self-improvement is effective in improving the performance of SRPO SRPO 5-rev. generates the highest win rate against the gold summaries, compared to all revisions of the baseline methods, as well as prior revision SRPO. We also observe that the gap in performance between SRPO and the baselines is significantly higher in OOD case.
T1:DR (In Distribution)
XSkm (DOD)
86
84
Win Rate (%)
82
82.5
SAPO
80
80.0
DRO
71.5
N-revision
Figure We present the win rates of SRPO, IPO, and DPO against -umttn-written summaries (GOLD) as function of N-revisions for both in-distribution (TL;DR) and dustif-dist (XSum) settings. The curves represent the mean win rates, with shaded areas indicating the st.dev. across 20 bootstrap evaluations. Notably, DPO and IPO show no improvements in their generations, whereas SRPO shows significant improvements with each iteration.
7 DISCUSSION AND LIMITATIONS
In this paper we have developed Self- Belf-improving Robust Preference Optimization (SRPO), a robust of- fline approach for learning from human preferences. We have proven mathematically and empirically, that unlike other offline methods like DPO and IPO, the solution of SRPO is completely independent of the behavior policy j and thus SRPO completely robust changes in μ.
Summary of results. We have tested SRPO on standard summarization tasks both on in in-distribution and cittof-ddistrbuttio (OOD) regimes. We have observed that in the OOD case SRPO outperforms both TPO and particularly the celebrated DPO by clear margin in terms of win-rate against gold completions, while the in-distribution case there is less difference between SRPO and the baselines. This is expected behavior since in-distribution less. We have observed that although 0-revision generation SRPC performs well, we have observed boost across the board by revising the generation through the self- reff-improvement model.
the robustness aspect of the algorithm matters
Future work and Limitations. In our work we used standard and relatively simple language tasks. In the future we would like apply SRPO to more challenging multi-task benchmarks in which the existing RLHF methods often specialize to a specific set of tasks more represented in the dataset, whereas SRPO should be more resilient due to its robustness to behavior policy p.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Ustün, and Sara Hooker. Back Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, 2024.
Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Chi Kit Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation, 2023.
URL https: /att://arxae www.iaivvvoooooo org/ abs/ 2204, 01171.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer. Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawtor Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022.
Ben Mann, Dario Amodei,
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. In Advances in Neural Information Processing Systems, volume 28, 2015.
Ralph Allan Bradley and Milton Terry. Rank analysis of incomplete block designs: I. of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
The method
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Daniele Calandriello, Daniel Guo, Remi Munos, Mark Rowland, Yunhao Tang, Bernardo Avila Pires, Pierre Harvey Richemond, Charline Le Lan, Michal Valko, Tianqi Liu, Rishabh Joshi, Zeyu Zheng, and Bilal Piot. Human alignment of large language models through online preference optimisation, 2024.
Yiding Chen, Yihe Dong, and Tengyu Ma. Provably mitigating overoptimization in rlhf: Your sft loss is secretly regularizing, arXiv preprint arXiv:2405.16436, 2024. URL http::::://axxi.rrrro
2.bs/2405.r 16436.
Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao. Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedack:: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.0137,, 2023. URL https: /tt////aavvv///////e//eeeee.aev/ www.oos//////.oooooooooooooo /2310 01377.
Abhimanyu Dubey and 532 others. The llama 3 herd of models. arXiv preprint arXiv:440.221783. 2024. URL https::t//.. //t////aivv..////////a//......aa www./or///ooo abs/ .2183.
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback, 2024.
Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, and Matthieu Geist. Averaging log-likelihoods in direct alignment. arXiv preprint arXiv:2406.1918 2024. URL https: / a rxiv. org/ abs /20 1988.

---

Published as a conference paper at ICLR 2025
Shangmin Guo, Biao Zhang. Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, and Mathieu Blondel. Direct language model alignment from online ai feedback, 2024. URL https: /ttp///axvvvvvv.///iv//e.///...l org/ 04792.
htttpp://axxd...og//bb..a 2240 2402.
Chi Hu et al. Teaching language models to self-improve by learning from language feedback. arXiv preprint arXiv:2406.077 066, 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: : /oopnreviiew net/ forum? id=nZevKeeFYfB,
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve, 2022.
Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects rlhf on Ilm generalisation and diversity, 2024.
Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq lqbal, Colton Bishop, Rebecca Roelofs, Lei M Zhang, Kay McKinney, Disha Shrivastava, Cosmin Paduraru, George Tucker, Doina Precup, Feryal Behbahani, and Aleksandra Faust. Training language models to self-correct via reinforcement learning, 2024. URL https: //arxiv. org/ abs 2409. 12917.
Zhuorong Li, Daiwei Yu, Lina Wei, Canghong Jin, Yun Zhang, and Sixian Chan. Soften to defend: Towards adversarial robustness via self-guided label refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2024a. Ziniu Li, Tian Xu, and Yang Yu. Policy optimization in rlhf: The impact of out-of-preference data, 2024b.
Yong Lin, Lu Tan, Hangyu Lin, Zeming Zheng, Renjie Pi, Jipeng Zhang, Shizhe Diao, Haoxiang Wang, Han Zhao, Yuan Yao, et al. Speciality vs generality: An empirical study on catastrophic forgetting in fine-tuning foundation models. arXiv preprint arXiv:2309.062555 2023.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Chain of hindsight aligns language models with feedback, 2023.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts, 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with reference- free reward. arXiv preprint PrXiv:2405.14755 2024. URL http::::////xx....rro.. abs 2405. 14734.
Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback, 2023.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.186 1/Dll/////1-- 1206. URL https::/acet antho -
12

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller amd Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization, 2024a.
Richard Yuanzhe Pang.
Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston, Iterative reasoning preference optimization, 2024b. URL https://axii..com
abs/2404.25 19733.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. arXiv, 2023.
Paul Roit, Johan Ferret, Lior Shani, Rose Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, Léonard Hussenot, Orgad Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00/88 2023.
Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models self-improve with general preferences, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv, 2017.
Yuda Song. Gokul Swamy, Aarti Singh, J. Andrew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage, 2024. URL /RLptttts////xxx.ooor abs abs/2406. 01 612.2.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feed- back. In Larochelle, M. Ranzato, R. Hadsell, M.F. Balean, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Asso- ciates, Inc., 2020. URL http:tttpsp/pppcceceeeeeeeeeeeee neurips oc cc/ httpp/ssceeepppeeeeeeeeee//essee willesppapet pdf.
Fahim Tajwar, Anikait Singh, Archit Sharma, Rafael Rafailov, Jeff Schneider, Tengyang Xie, Stefano Ermon, Chelsea Finn, and Aviral Kumar. Preference fine-tuning of Ilms should leverage suboptimal, on-policy data, 2024. URL
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires. and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. 2024.
Emanuel Todorov. Linearly-solvabl markov decision problems. Advances in neural information processing systems, 19, 2006.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023.
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. Zephyr: Direct distillation alignment, 2023.
13

---

Published as a conference paper at ICLR 2025
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:2311.12908 2023.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Gen- eralizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309. 16240, 2023.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models, 2023.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang. Chao Yu, and Yi Wu. Is dpo superior to ppo for Ilm alignment? a comprehensive study, 2024. URL https: /arxivvvvvv/./////vi//aa/////viv org/ alss 2404 10719.
Yifan Xu, Kening Zhang, Haoyu Dong, Yuezhou Sun, Wenlong Zhao, and Zhuowen Tu. Rethinking exposure bias in language modeling, 2020. URL https: aaa.aaiiivv.......a.a.aaa..... org/ aass//b///// aaaaab/s/19/ /1910,190,,.00 11235.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024b. URL https://arxiv.orr /bs/
2401.10020.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023. URL /attts:://axiii: org/ abs//3044 05302.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. SLiC-HE: Sequence likelihood calibration with human feedback. arXiv, 2023a.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv, 2023b.
14

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
Our work lies in offline preference optimization, a vivid area of research since the introduction of DPC (Rafailov et al., 2023). Some of the core concepts of this research topic was generalized and formalized by Azar et al. (2023). In particular they characterized the underlying optimal solution for a generie preference optimization objective and introduced IPO for addressing some of the related shortcomings of DPO. HL (Zhao et al., 2023a) was introduced around the same time, from a less RL-centric point of view. All these approaches have been abstracted later by Tang al. (2024), the general recipe being to build a contrastive loss function from convex classification function and to make use of the analytical solution of the RI. problem to learn directly the policy. A common underlying assumption is that the related RL problem is KL.regularized, This has been generalized to more general fddivergences by Wang et al. (2023). These are just a few among many works on direct alignment (Tajwar et al., 2024; Xu et al., 2024; Guo et al., 2024; Yuan et al., 2024b; Song et al., 2024). However, they all share the fact of not considering self-improvement policies. contrary to SRPO. online RLHF for learning self-correcting model.
Concurent to this work Kumar et al. (2024) However unlike our method which aims at maximizing the preference of improved completion with respect to the in-context completion their work focuses on maximizing the reward of final answer. The issue with this approach is that the model can simply ignore the in-context completion and try to optimize the final completion without paying attention in-context completions.
Offline preference optimization was introduced as an alternative to more classic RLHF approaches, such PPO (Schulman al., 2017: Ouyang et al., 2022) more generally saaicc-gradient--ee approaches (Roit et al., 2023; Ahmadian et al., 2024). These methods require training a reward model on a preference dataset, usually with a Bradley-Terie model is then used to fine-tune the LLM via online RI, requiring many generations from the model. This reward model shares the common issue of DPO and other direct preference alignment methods, it is dependent on the sampling distribution μ used for constructing the preference dataset, contrary to SRPO. Moreover, classic RLHF is online, while SRPO is offline and thus more easily scalable.
model (Bradley & Terry, 1952). The reward
Some similarities also exist between SRPO and Mash-MD (Munos et al., 2023). Indeed, if in I we replace the self-improvement policy 3y (-ly,2) by a classic policy =(|[x), then we obtain the saddle-pon poiite--ointtooo optimization problem that Nash-MD solves. However, considering seff-improvement policy is a core contribution of our work, and it is not anecdotal. From a technical viewpoint, this is critical for simplifying the minimax problem of Eq. (1) and obtaining a simple offline optimization problem. NashMD on the other hand adapts algorithms from the game-theory literature and can only be solved online with all the stability issues of online methods and large inference costs. From practical point of view self-improvement provides a boost in performance by refining the original generations of LLM. The feature that Nash-MD is missing. Finally, even though the Nash equilibrium of Nash-MD MD does not depend on the sampling distribution μ., it relies on learned reward function, with the possible associated caveats mentioned earlier, which is not the case of SRPO.
Our work is also obviously related to the concept of chain of thoughts (Wei et al., 2023; Yao et al., 2024), aeff-improvement (Huang et al., 2022) and self-refining LLMSs (Madaan et al., 2024). However, it is very often used as a way of prompting model to obtain better results, and less often as a component of a learning paradigm (Liu et al., 2023; Huang et al., 2022). A notable exception is the recent work of (Pang et al., 2024b) that generalizes DPO to incorporate chain of thoughts. However the main focus of this work is mostly on improving the chain of thoughts reasoning and not on self-improvement. To our best knowledge, we propose the first approach that combines training self-improvement LLMs and offline preference optimization through a single supervised objective, moreover in theoretically grounded manner and showing the robustness to J.
B ABLATION: THE EFFECT OF COMBINATION COEFFICIENT a ON SRPO PERFORMANCE
SRPO loss of Eq. (17) is a convex combination of two losses L and Ż, via the combination coefficient a. To understand how both terms affects the loss we plot the win rates both in in-distribution case and OOD case as a function of a in Fig. 4. We observe that the term that contributes most to the performance of SRPO is Lg as in the case of a = 1 when we only use the loss for improvement model 7. we almost match the best performance. On the other hand using only 7. (i.e., a -0) is not
15

---

Published as a conference paper at ICLR 2025
enough to achieve top performance. We also observe combining both losses seems to provide some boost in performance especially OOD case.
T1,OR iin Distribction
XSum (OOD)
87
Win Rate (56
82
81
82
N-revision
Figure 4: We present the win rates of SRPO against human-written summaries (GOLD) as function of N-revision iterations at different values. We report their mean (curve) + st.dev. (shaded area), across 20 bootstrap evaluations, as described the Evaluation section. We observe that SRPO achieves meaningful iterative improvements capability as the value of increases.
C EXPERIMENTAL DETAILS
We provide the prompt templates used for training and evaluations in section 6.
C.I PROMPT TEMPLATES
C.1.1 TL:DR
O-revision:
Below is reddit POST and the corresponding SUBREDDIT and TITLE. Write a both precise and concise sunmary the contents the POST.
SUbREDdIT: ((subreddit)
TITLE: $(title)
POST: (post)
TL;DR:
N-revision:
Below is a reddit POST and the corresponding SUBREDDIT, TITLE, and EXAMPLE SUMMARY. Write both precise and concise summary of the contents of the POST.
SUBREDDIT: $ (subredditt)
TITLE: $(title)
POST: (post)
EXAMPLE SUMMARY: S (N-1) th_ (h_eaaa__ l__smmrrr
TL;
C.1.2 XSUM
0-revision:
Below is news ARTICLE and the corresponding ID and TITLE. Write both precise and concise summary of the contents of the ARTICLE.
16

---

Published as a conference paper at ICLR 2025
ID: (iid) TITLE: ((title) ARTICLE: s(article)
DL;
N-revision:
Below is a news ARTICLE and the corresponding ID, TITLE, and an EXAMPLE SUMMARY. Write a both precise and concise summary of the contents of the ARTICLE.
ID: ((d))
TITLE: $ (tiitee)
ARTICLE: S(article)
EXAMPLE SUMMARY: 5 (N-1) th
tt_eeamplleeee Lee_eexmple_sssee le_mmarryy)
TL, DR:
17

---

Published as a conference paper at ICLR 2025
C.2 EXAMPLE SUMMARIES
C.2.1 TL:DR
Post
I have horrible caffeine addiction, and I don't ton't like sacrificing any of my daily calories for coffee. I used to drink 5-6 Diet Dr. Peppers a day, but have switched to almost exclusively drinking only water most days. I do have Diet Dr. Pepper occasionally. and know that techmically it's okay to probably have one a day, but I honestly just feel better all around when I drink only water. But if I don't have some caffeine in the morning, 1 will be writhing in the floor in pain with skull-poudding headache by mid-day. I know. Ridiculous, right?
So, went the vitamin aisle the other day in search of supplement with a little caffeine and the only thing found was One A Day Adult Vitacraves Gummies with Energy Support. Here's the website with [nutritional info] One serving 20 calories and has 90 mg of caffeine. There's 's also some vitamin B6, B12, folic acid in there that supposedly helps turn food into energy. have no idea if Ihat's true. I've been laking them couple days, and so far do feel more energized and I haven') I had any headaches. The issue is that one bottle is around 89 and there are only 60 in bottle. You have to take 4 (4!) day, which is very intriguing considering the brand name and all. So I'll cost me around $20 a month. Which is admittedly cheaper than what I used spend on Diet Dr. Peppers, but still. I was just curious if anyone else had a better option for caffeine in supplement or vitamin?
GOLD
The vitamin with caffeine I'm taking is on the expensive side, and I'm self- proclaimed tightwad. Anyone have better options?
SRPO 1-
I have a caffeine addiction and need a little bit of caffeine in my day. I'm taking One A Day Adult Vitacraves Gummies with Energy Support, but it's expensive ($20/month). Any other suggestions for a caffeine supplement or vitamin?
rev.
SRPO 0- rev.
I have a terrible caffeine addiction and need a little caffeine in my day. found a vitamin with caffeine in it, but it's pretty expensive. Any other suggestions for a caffeine supplement or vitamin?
IPO 1-rev.
I have caffeine addiction and need some in my system every day function, but I don't 't like sacrificing calories for coffee. I've found a supplement with caffeine, but it's expensive. Anyone have better option?
IPO O-rev.
I have a caffeine addiction and need some in my system every day function, but don't 't like sacrificing calories for coffee. I've found a supplement with caffeine, but it's expensive. Anyone have a better option?
DPO 1-rev.
I have a caffeine addiction and need a supplement or vitamin that has caffeine in it to take couple times week that don't have headache all day. Any recommendations?
DPO O-rev.
I have caffeine addiction and need supplement vitamin that has caffeine in it to take a couple times a week so that don't have a headache all day. Any recommendations?
18

---

Published as a conference paper at ICLR 2025
C.2.2 XSUM
Article
A paper detailing its proposals focuses on the need to avoid a hard border. The government stresses there should be physical infrastructure, such customs posts, at the border, which has almost 300 crossing points. Critics say the proposals lack credible detail. with Labour deriding the plans for the border as "a fantasy frontier". The government's 's paper does not envisage CCTV cameras or number plate recognition technology at the border, or set back from it. Instead, the government is arguing for wide-ranging exemption under which small and medium-sized busi- nesses will not have to comply with any new customs tariffs. Northern Ireland is the only part of the UK that will share a land border with an EU state post-Brexit. The future management of that border is a highly sensitive issue and is one of three main priorities in UK-EU Brexit negotiations. the customs union do not impose tariffs- taxes on imports on each other's 's goods. Every country inside the union levies the same tariffss imports from abroad. So, for example, a 10% tariff is imposed on some cars imported from outside the customs union, while 7.5% is imposed on roasted coffee. Other goods such as soap slate have no tariffs. The UK has said is leaving the EU's customs union because as a member is unable to strike trade deals with other countries.
What is the customs union? Countries in
GOLD
The government has said there must be an "unprecedented solution" for the border between Northern Ireland and the Republic after Brexit.
SRPO 1- rev.
The UK wants to avoid hard border between Northern Ireland and the Republic of Ireland after Brexit by proposing two customs models to the EU. The UK also dismisses the idea of customs border in the Irish Sea.
SRPO 0- rev.
Brexit Secretary David Davis wants limited transition period implement new customs arrangements, including consideration of the "unique circumstances" of Northern Ireland and the Republic of Ireland. The UK is trying to prevent the need for physical customs posts along the Irish border.
IPO 1-rev.
Theresa May wants 'scamless' border between Northern Ireland and the Republic of Ireland after Brexit - but the EU may not accept that as viable. The UK government has published position paper setting out its ideas the issue.
IPO O-rev.
Theresa May wants a 'seamless' border between the Republic of Ireland and Northern Ireland after Brexit but the EU may not accept that as viable. The UK government has published position paper setting out its ideas on the issue.
DPO 1-rev.
the UK is proposing a "new customs partnership" or a "highly streamlined customs arrangement" prevent hard border between Northern Ireland and the Republic of Ireland, in the event of no deal being reached over Brexit.
DPO O-rev.
the UK is proposing "new customs partnership" arrangement" to prevent hard border between Northern Ireland and the Republic of Ireland, in the event of no deal being reached over Brexit.
"highly streamlined customs
19

---

Published as a conference paper at ICLR 2025
Article
It follows a row over the removal of personal items from graves in Torfaen which were sent rubbish tips. Nearly 2,000 people signed petition calling for legislation on the issue. Public Services Minister Leighton Andrews said he was sympathetic but believed was a matter for each local authority "develop and justify their own approaches". Torfaen council said in June 2014 it had organised the clear-up because the over-personalliatioon of graves some cemeteries had prompted complaints and made maintenance difficult. The authority apologised for any upset, but said it had made efforts to inform people about the clear-up and had allowed them two months to collect any items they wanted to keep.
GOLD
Calls for Wales-wide law stating what tributes can be placed on children's graves have been rejected by ministers.
SRPO
Public Services Minister Leighton Andrews has rejected a petition calling for legisla- tion on the removal of personal items from graves, saying was a matter for each local authority to "develop and justify their own approaches".
(Irv..)-revv.
SRPO O-rev.
Public Services Minister Leighton Andrews has rejected petition calling for legisla- tion on the removal of personal items from graves.
IPO (),
Vicky Pryce wanted revenge on ex-MP Chris Huhne over him getting points on his licence, so she took the speeding points for him in 2003, a court heard.
rev.
DPO 5
Vicky Pryce told court she signed speeding points form for her husband Chris Huhne in revenge for him threatening their marriage over his speeding points.
C.3 EVALUATION DETAILS: CALCULATING WIN-RATE
Calculating the win-rate of AI models, especially in the context of comparisons against human performance other models, involves several parameters and methodologies. In particular to calculate the win-rate of Al model vs. human, we follow these steps:
1. We run the trained model on the same input data the test of supervised dataset and generate multiple completions.
2. We compare the completions using gpt 4 with the corresponding ground truth completions supervised dataset by assigning a point system based on outcomes (c.g.. for win, 0.5 for draw, 0 for loss).
3. We aggregate scores across rounds for each model and calculate the win rate.
In particular we use the following formula to calculate the win rate:
Win Rate Number of Wins by Model
Total Rounds
We also report confidence intervals indicate statistical reliability.
D EXPERIMENTS ON ULTRA FEEDBACK
Setup. In our experiments, we consider the offline direct preference optimization setup to learn from human preferences (Rafailov al., 2023). We empirically test the effectiveness of SRPO against two prominent offline preference learning methods, DPO (Rafailov et al., 2023) and IPC (Azar et al., 2023) as baselines. We also consider two popular extensions of DPO, namely, S SIMIPO (Meng et al., 2024) and RPO (Chen et al., 2024).
Implementation details. In the case of SRPO and all baselines we optimize the regularization coefficient 3 and the learning rate by syper-parameter sweep range of possible parameters. (In the case of RPO we also sweep over the cross- cross-entropy coefficient.) Also following on the recent line of work that shows using average log-likelihood is advantageous using sum log- likelihood
20

---

Published as a conference paper at ICLR 2025
in domains with varying completion length we use the average log- logllikei variant of IPO, DPO, RPO and SRPO (Grinsztajn et al., 2024; Yuan et al., 2023).
dataset for SFT, filtered version of the original dataset containing 53k prompt and completion pairs. For our preference training, we use the binarized version of the UltraFeddbcck (Cui al., 2023)?, which contains 64k pairwise preference data generated by various Al chatbots. For the win-rate evaluation, we use the Arena-Hard dataset", which contains 500 prompts, to test against LLaMA- 33. 1-8B Instruct (Dubey & others, 2024), an official post-trained version of LLaMA- 3 --8B Base model by the Meta LLaMA team. We generated completions using LLaMA-3. 3, 1-BB Instruct(Dubey others, 2024) and Arena-Hare prompts. which we later used to compute win-rates against.
Datasets. We the ShareGPT-
Model Setup. We use LLaMA- -..1--1311-- -3.1-83 83 Base as base model (Dubey & others. 2024) and Google cloud v5 -55 TPUs to conduct all MaaaM-bbasd training and evaluation. We first SFT the LLaMA-3. 1-8B Base on the ShareGPT dataset. Then, we used the SFT checkpoint to initialize all preference training experiments. Below are details on the recipe for the SFT and preference training stages.
Supervised-fig Tuning, In the SFT stage, we train for single epoch, using the Adam optimizer (Loshchilov Hutter, 2019), with 31 = 0.9 and 82 0.999. We use cosine decay learning rate CoshcullerLoohcchllvv & Hutter, 2017) with peak value 2.5 x 10-5, and decay it down to We use 10 warm-up steps and an effective batch-size 64.
1.25 x 10
Preference Training. For fair comparison across different methods, we initialize all models using the same SFT checkpoint and train them for epoch using the same Adam optimizer setting: we used learning rate of 1.25 x 10-S with cosine learning rate scheduler to decay it down to 1.25 10-? 7 with 128 warm-up steps, and an effective batch-size of 32. 31 and 82 values of the Adam optimizer the same as the ones used for the SFT stage. We used the following values of [:: Asaeo 1.3, $reo = 1.0, Spec = 10.0, BrEO = 10.0, and ΩΑIIIPP = 50.0. For RPO, we used 1.0 x 10 as the weight of cross entropy loss on the preferred completions, which is then scaled by B.
To fine-tune the models, we use a training pipeline based on OP TAX and FLAX libraries of JAX."
Evaluation. We win rates as computed by gpt-4o (OpenAI, 2023), the main means for evaluation (See Sec. C.3 for more details).
Arena-Hard Results. We test our models using the prompts from the Arena-Hard dataset in Fig 5. For every model, we generate O-rev, and then use these generations to revise our completions recursively from 1-rev. to 5-rev. using the self-improvement model, and measure the models' win rate against the generations of LLaMA-3 11..11.111.... 1 1- Instrucct model for the same prompt.
We observe that srpo 1-rev. produces the methods and other revisions of SRPO in terms of win rates by a significant margin. Notably, we also observe a significant drop the win rates for all models after just one revision.
completions, outperforming both baseline
Additionally, we present SRPO's head-to-heat win rate results against DPO, IPO, RPO, and SIMIPO O-rev. and 1-rev. We see that SRPO outperforms all other methods at both O-rev. and 1-rev.
Table 1: Ared head2head results SRPO's WR
vs.
DPO O-rev.
IPO O-rev.
RPO O-rev.
SIMPO O-rev.
SRPO O-rev.
53.67%
51.10%
51.30%
63.1%
vs.
DPC 1-rev.
IPO 1-rev.
RPO 1-rev.
SIMPO 1-rev.
SRPO 1-rev.
67.50%
57.50%
58.30%
74.90%
un unfiltered
'https://huggiii
htttps///iithub.comm//// hta
21

---

Published as a conference paper at ICLR 2025
Arena-Hard Results (vs Llama 3.1 8B Instruct)
55.0%
Win Rates (judge: GPT-4o)
50.0%
45.0%
40.0%
SRPO
35.0%
IPO
DPO
30.0%
RPO
SimPO
0
1
2
3
4
5
N-revision
Figure 5: We present the win rates of SRPO, IPO, DPO, SIMIPO and RPO against LLaMA-3.0 B.--MA--.1-BBII-- Instruttt I.stttt-ssttttute as a function of N-revisions for Arena-hard prompts setting. The curves represent the mean win rates, with shaded areas indicating the st.dev. across 20 bootstrap evaluations. Notably SRPO dominates the win-rates of all other methods across all revision steps.
22


Reviews:
[{"review_id": "pKSELG5C4m", "reviewer": "Reviewer_Nqte", "summary": "This paper studies the problem of reinforcement learning from human feedback (RLHF). It designs a novel RLHF algorithm called SRPO based on training a robust self-improving policy by solving a min-max problem. It explains the math in detail on how to find the solution to this min-max problem. The experiment results show that SRPO achieves a higher winning rate against the gold dataset than existing state-of-the-art RLHF algorithms like DPO and IPO.", "questions": "1. It can be beneficial to provide more explanation of the self-improving policy. This concept is not common in many RLHF literatures. Readers may be curious about how these policies work and how they are implemented. Providing more details can make this paper easier to follow.\n\n2. Can you provide more details about the evaluation? For example, what are the parameters, such as the accuracy compared to humans, of the AI evaluation platform? Exactly how do we measure the winning rate between any two models? Revealing such details can make this work much easier to reproduce.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper studies the critical problem of finding a more efficient RLHF algorithm. It could have great potential in many important real-world applications.\n\n2. The idea of using a self-improving policy in solving the RLHF problem is novel.\n\n3. The mathematical explanation of how to solve the min-max learning problem they propose is thorough.\n\n3. The SRPO algorithm aims at training a robust policy against the quality of the dataset, making it more trustworthy than many other RLHF algorithms.\n\n4. The experiment design is reasonable in general, and the results look positive.", "weaknesses": "1. There is no theoretical guarantee of the learning outcome. This makes the whole theoretical part weak. Is there a chance to provide any theoretical guarantee on the performance of the policy learned by SRPO under some assumptions?\n\n2. The design of SRPO algorithm is novel, but the description of its motivation can be improved. Currently, the motivation is that 'Instead, it is more natural to learn that given a query x and a completion y what would be the improved completion upon y'. However, in general, you can also say that to some other learning algorithms as long as they improve the quality of their policy to generate better completion after each training iteration. It is better to find some more unique motivation for the design of the SRPO algorithm.\n\n3. The SRPO algorithm's winning rate against other algorithms is not provided. This paper only measures the winning rate against a gold-standard dataset. A policy that has a higher winning rate against a fixed dataset than that of another policy does not necessarily mean this policy is better than that policy. To verify that the policy learned by SRPO is better than the policies learned by other algorithms, it is necessary to compare these policies against each other directly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xm5TX8duFo", "reviewer": "Reviewer_Pvzi", "summary": "This paper introduces a novel method, Self-Improving Robust Preference Optimization (SRPO), which aims to address two primary limitations in current preference optimization techniques. First, existing methods generally lack an inherent self-improvement mechanism at inference time, which limits adaptability. Second, they often rely heavily on the training task and distribution used to generate preference data, which affects the robustness of solutions. To address these issues, SRPO is designed as an offline preference optimization method that leverages a min-max formulation to learn a robust generative policy. This policy generates completions that require minimal adjustment when optimized through a self-improvement policy. The authors apply a DPO/IPO-inspired derivation to show that this min-max objective can be effectively optimized through standard supervised learning techniques. Experimental results on the TL;DR Summarization dataset indicate that SRPO achieves better in-distribution (ID) and out-of-distribution (OOD) performance compared to DPO and IPO baselines.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written and easy to follow, with each step of the SRPO algorithm systematically derived.\n\nThe derivations provided in the paper appear rigorous, demonstrating a well-founded approach to preference optimization. The use of a min-max objective with a focus on self-improvement mechanisms is an interesting contribution.", "weaknesses": "The evaluation is currently limited to a single dataset (TL;DR Summarization) and only compares SRPO against two baselines, DPO and IPO. Conducting experiments on additional datasets would strengthen the empirical claims related to robustness.\n\nThe paper lacks (empirical) comparisons to more recent preference optimization methods, such as SimPO (Meng et al., 2024) and RPO (Liu et al., 2024), which integrate recent advancements over DPO. \n\nIn Figure 3, SRPO achieves improved performance with revisions, likely due to its self-improvement mechanism. However, unlike DPO and IPO, which lack this self-correction feature, SRPO appears to incur additional inference costs to achieve its performance benefits.\n\n(Meng et al., 2024) SimPO: Simple Preference Optimization with a Reference-Free Reward\n\n(Liu et al., 2024) Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L4P8qHRyj1", "reviewer": "Reviewer_hrBf", "summary": "This paper proposes Self Improving Robust Preference Optimization (SRPO), a new method of LLM fine tuning. The authors propose using in-context learning to iteratively generate higher quality completions while being robust to the underlying offline dataset. The paper proposes a min-max objective which aims to maximize the quality of the improvement (of the improvement model) and minimize the amount by which you can be improved (for another language model). The paper then analyzes optimizations for each of the models independently and derive a convex combination of these losses. Finally, the authors experimentally verify their theory with experiments on TL;DR and XSum.", "questions": "- DPO and IPO both have \"revisions\"? Does this mean they received the same in context prompt as SRPO?\n- The primary thesis of this work is that current preference optimization methods should not expect the ideal completion to be written and PPO/DPO/IPO are solving a fundamentally different task. In that vein, how does PPO compare under these same conditions? Is the lack of robustness also observed?\n- Overoptimization has been observed in DPO and other direct alignment algorithms. Is this observed with SRPO as well? Why/Why not? [0]\n\n\n[0] https://arxiv.org/abs/2406.02900", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The observations made by this paper are well supported empirically. In particular, they use a nice synthetic bandit example to show robustness. Further, they experimentally verify on the TL;DR dataset and XSum on out of distribution examples. This paper further does a good job elucidating the derivation of the objective.", "weaknesses": "- The proposed method empirically performs similar to other methods under the same amount of inference compute.\n- The paper does not provide much theoretical justification of the combination loss.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wYrMT7o3cd", "reviewer": "Reviewer_LbBR", "summary": "The authors propose Self-Improving Robust Preference Optimization (SRPO) to address two of the main drawbacks of existing RLHF methods. That is, (a) models trained with RLHF can learn from mistakes or negative examples through RL mechanism or contrastive loss at the time of training. However at the time of inference they are not equipped with an innate mechanism to correct mistakes by self-improvement. (b) The optimal solution of existing methods is highly task-dependent and thus it is difficult for them to generalize to new tasks. SRPO overcomes these shortcomings, and its effectiveness is shown also empirically.", "questions": "Is the problem studied by the authors related to continual learning under specific trade-offs [1]?\n\n---\n\n[1] https://arxiv.org/abs/2305.14782", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Although this is not my field of research, the paper was sufficiently clear. Its motivations are well-stated, and the results easy to grasp. They seem relevant to the RLHF literature.", "weaknesses": "Probably due to my unfamiliarity with this field, I am not able to find major weaknesses to the paper. Some minor considerations follow.\n\nLine 62: \"at the time of inference It would be very\" should be changed to \"at the time of inference, it would be very\".\n\nWhat is $\\mathbb{R}^*_+$ in line 162?\n\nLine 195: \"Eq. equation 2\" should be \"equation 2\". This typo is repeated throughout the manuscript.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)