Meta-Learning Objectives for Preference Optimization
Authors: Carlo Alfano
Silvia Sapora
Jakob Nicolaus Foerster
Patrick Rebeschini
Yee Whye Teh
Venue: NeurIPS 2025 poster ()
Keywords: Preference Optimisation
Evolution Strategies
Meta-Learning
Mirror Maps
Primary area: reinforcement_learning
TL;DR: Using a new suite of MuJoCo tasks for systematic evaluation, we develop specialized mirror descent-based preference optimization algorithms that outperform existing methods in both MuJoCo and LLM alignment tasks.

Abstract: Evaluating preference optimization (PO) algorithms on LLM alignment is a challenging task that presents prohibitive costs, noise, and several variables like model size and hyper-parameters. In this work, we show that it is possible to gain insights on the efficacy of PO algorithm on simpler benchmarks. We design a diagnostic suite of MuJoCo tasks and datasets, which we use to systematically evaluate PO algorithms, establishing a more controlled and cheaper benchmark. We then propose a novel family of PO algorithms based on mirror descent, which we call Mirror Preference Optimization (MPO). Through evolutionary strategies, we search this class to discover algorithms specialized to specific properties of preference datasets, such as mixed-quality or noisy data. We demonstrate that our discovered PO algorithms outperform all known algorithms in the targeted MuJoCo settings. Finally, based on the insights gained from our MuJoCo experiments, we design a PO algorithm that significantly outperform existing baselines in an LLM alignment task.

Meta-Learning Objectives for Preference Optimization
Carlo Alfano*t Department Statistics University of Oxford
Silvia Sapora"
Jakob N. Foerster Department of Engineering University of Oxford
Department of Statisties University of Oxford
Patrick Rebeschini
Yee Whye Teh Department of Statistics University of Oxford
Department of Statistics University of Oxford
Abstract
Evaluating preference optimization (PO) algorithms on LLM alignment is a chal- lenging task that presents prohibitive costs, noise, and several variables like model size and hyper-parameters. In this work, we show that it is possible to gain insights on the efficacy PO algorithm on simpler benchmarks. We design a diagnostic suite of MuJoCo tasks and datasets, which we use to systematically evaluate PO al- gorithms, establishing more controlled and cheaper benchmark. We then propose a novel family of PO algorithms based on mirror descent, which we call Mirror Pref- erence Optimization (MPO). Through evolutionary strategies, we search this class to discover algorithms specialized to specific properties of preference datasets, such as mixed-quality or noisy data. We demonstrate that our discovered PO algorithms outperform all known algorithms in the targeted MuloCo settings. Finally, based on the insights gained from our MuJoCo experiments, we design PO algorithm that significantly outperform existing baselines in an L.M alignment task.
1 Introduction
Learning from human preferences (Christiano et al., 2017) is a paradigm which enables the alignment of machine learning systems to relative human preferences, without requiring access to absolute rewards. While the framework was developed for robotic and games applications with experiments on MuJoCo simulations and Atari (Akrour et al., 2012; Biyik & Sadigh, 2018; Ibarz al., 2018), this paradigm has been successfully applied to Large Language Models (TTem et al., 2023; Achiam et al., 2023). In particular, fine-tuning pre-trained LLMs with human preferences has become a popular strategy to adapt them to specific tasks and to improve their safety and helpfulness.
Within this framework, Reinforcement Learning from Human Feedback (RLHF) is one of the most popular methods. It consists in learning a reward function using a preference dataset and then optimizing the estimated reward using Reinforcement Learning methods such as Proximal Policy Optimization (PPO) (Schulman et al., 2017). However, the training pipeline of RLHF is quite complex, which is why implicit approaches such as Direct Preference Optimisation (DPO) (Schulman et al., 2017) have gained traction thanks to their simplicity. These methods do not learn a reward model but estimate it implicitly using the policy of the agent. Many follow ups to DPO have been proposed (Yuan et al., 2023; Zhao et al., 2023; Azar et al., 2024; Xu et al., 2024a; Hong et al., 2024; Park et al., 2024; Meng al., 2024), but comparing their performance in LLM alignment is complex task that incurs high costs, noise, and the inherent difficulty in judging a response better than another.
In this work, provide a comprehensive analysis of PO algorithms, examining their behavior automatically generated preference datasets. We return to the roots of RLHF by performing this
"Equal contribution, order decided by coin flip.
Now at Amazon.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

analysis in MuJoCo environments and datasets, where the underlying ground-truth reward structure well defined and offers clear performance metric compare agents. In particular, we design a task where a pre-trained agent has to adhere to new stylistic constraint, emulating the typical conditions of L.LM fine-tuning. Our findings indicate that many PO algorithms present distinct failure modes when applied to specific mixed-quality or noisy datasets.
Moreover, we introduce a framework for finding PO algorithms. Specifically, we define class of PO algorithms based on mirror descent (Nemirovski & Yudin, 1983), which generalizes DPO and ORPO for particular choices of the mirror map. We then show that this class can be easily parametrized and searched using evolutionary strategies (ES), optimizing for the final performance of the trained policy, as measured by the ground truth reward.
For each setting we consider, we discover an algorithm that significantly outperforms all baselines. Analyzing the discovered algorithms, we find that the main difference between them and the baselines is that they keep optimizing the policy of the agent well after the probability of generating the chosen trajectory has surpassed the probability of generating the rejected one. We use this insight to design new PO algorithm, Temporally-Awe Aware Mirror Preference Optimization (TeMPO), which demonstrate promising results in an LLM alignment task. We summarize our contributions below.
1. We perform systematic evaluation of eight existing PO algorithms on automatically generated preference datasets with varying levels of data quality, noise levels and initial policy. We see that most existing algorithms struggle when dealing with noise and mixed -uxaitty data.
2. We introduce a novel family of offline PO algorithms using mirror descent, named Mirror Preference Optimization (MPO), which can be easily parameterized and explored via ES.
3. For both noisy and mixed-quality settings, we find and describe PO algorithm within our framework that largely outperforms all the considered baselines in our MuJoCo benchmark.
4. We demonstrate that takeaways from our analysis on the MuJoCo setting, as well as the characteristies of the discovered PO algorithms, can be successfully transferred onto LLM tasks. In particular, we show that our TeMPO algorithm significantly improves upon the baselines.
2 Preliminaries
Let M = (S. μ)) denote an episodic Markov Decision Process, where S and A are respectively the state and action spaces, P(s') 8, 8,a) the transition probability from state s to s when taking action a, r(x,a) a) E [0, 1] is the reward function, T is the maximum episode length, and ju is a starting state distribution. policy क E (A)))) where A ((A) is the probability simplex over A, represents the behavior of an agent according to the probability distribution 7(- 8). Let { az) az) )T= denote trajectory of length T and, with a slight overload of notation, let (T)) = I x (aa (s) and r(r) ΣT= r(st, Lastly, let π (-I7) be distribution over (A()))² defined T--[so) x... ((-1sN-1).
MDP, whereby state E S the agents takes actions
Our objective is to find a policy x² that maximizes the expected cumulative reward of an episode, that is
n²" E Brg
= argmax E Bep
Σ ((s),a))
t=
where ot (-10s) and 8141 ~ P(- at). Let D {(s)) Th, 1 be a preference dataset, where each tuple (8o, Tw, T) consists of starting state and two trajectories with starting state 8g. Each pair of trajectories is ranked by a judge, who determines a chosen trajectory Tu ("win"") and rejected trajectory 7. ("lose"), based on the cumulative rewards r(Tw)) and r(n)). Most settings assume the judge ranks trajectories according to the Bradley-Terry model (Bradley & Terry, 1952), whereby the probability of choosing Tw over is defined as
exp(r(Tw))
PT > m))= exp(r(rw)) + exp(r(n)) = -(il))-r(n))
(2)
where a is the sigmoid function. In this work, we consider an offline training setting, where the agent aim to solve the optimization problem in (1) but only has access to the the dataset D and cannot collect further data. We also assume the agent does not have access to either the transition probability P. the reward function or the MDP M.
2

---

2.1 Alignment to preference feedback
There are several algorithms in the literature to optimize the objective in (1) using a preference dataset D. We describe supervised fine-tuning (SFT), DPO and ORPO, they among the most popular and as many methods can be seen as a variation of one of these algorithms.
SFT SFT is an initial alignment phase, where the policy To is trained to imitate high-quality demon- stration data. The starting policy wo is updated to minimize the cross-entropy loss E(T, (so, Twr 7i)) log i (Tw)). We call reference policy Teet the policy obtained at the end of this procedure.
Direct Preference Optimization (DPO) consists in solving a maximum likelihood estimation problem and policy optimization problem in single step. The maximum likelihood estimation problem is the one to find an estimate the true reward function that governs how the preferences are expressed, that is
DPO
E argmax E B(oo -o ro(r))),
for parametrized reward class {re E the policy optimization problem is the one to maximize the expected reward, that is
T - (er (1)(
n E argmax (i
-
(4)
where DRc represents the KL-divergence and is introduced to prevent the policy from moving too far away from the dataset distribution.
DPO merges these two problems by using the agent itself to implicitly represent the reward model. It consists in optimizing the objective
π²' E argmax E ( logo (5(108
(10g (1 (Tw)) Tref (Tar)
(π))
log
(5)
Tref (Ti)
π
which is obtained by plugging the theoretical solution of (4) in the maximum likelihood problem in (3). Refer to Appendix C for details. Thanks to its simplicity, DPO has been widely adopted to fine-tune LLMs (Yuan et al., 2024; Jiang et 2024).
known issue of DPO is that pushes probability mass away from the preference dataset and to unseen responses (Xu et al., 2024b), which can cause the final policy to deviate significantly from reference policy, even when the reference policy aligns well with human preferences. To mitigate this risk, DPO is usually applied for few epochs.
ORPO ORPO further simplifies the training pipeline and addresses the distribution shift issue present in DPO. It merges the SFT and DPO steps into one, optimizing the unified objective
x² argmax E ((00 log n (T.)) +A log a (log (odds= log (odds (ri))
(6)
preference optimization
where ( (()) (1)x()))) ORPO gets rid of the need for reference model by adding an SFT term to the preference optimization objective function, and uses this term to prevent the optimized policy from moving too far away from the dataset distribution. Additionally, the SFT term prevents pushing probability mass away from the preference dataset, addressing the distribution shift issue present DPO.
Research on preference optimization has been very active and many methods have been proposed. We present a summary some among the most popular algorithms in Table 4 and brief discussion in Appendix A. Beyond these implicit algorithms, there are several other methods that explicitly solve the maximum likelihood problem in (3) and use the learned reward model to optimize the objective in (4) with an RI algorithm. Overall, RLHF is superior approach and the industry standard, but more computationally expensive and complex to implement. For a detailed discussion and comparison between DPO-like methods and PPO, refer to Appendix H.

---

2.2 Mirror Maps
We review the concept of mirror map, which will be needed when describing our methodology. For convex set X' C TRl-A, mirror map h X - R is defined strictly convex, continuously differentiable and essentially smooth function" function that satisfies Vh(x)) R Al, Essentially, mirror map is a function whose gradient allows bijective mapping between the primal space and the dual space RIAI, The specific class of mirror maps that we are going to use is the w-potential mirror map class, to which most mirror maps considered in the literature belong.
Definition 2.1 (w- (w--otential mirror map Krichene al. (2015)), For E (-o0, tool, w S 0, an w-potential is defined as an increasing C' -diffeomoopphism Ø (-00, u) + (w, too) such that
lim <(x) w. lim ((x) = +00,, / Φ o I 00.
For any w-potential Φ, the associated mirror map is he EnEA 0-
(2)
When <(x) e= we recover the negative entropy mirror map, while we recover the f--norm when ((x) 2x (refer to Appendix F). Mirror maps in this class are simple to implement in practice, where A is often large, as they can be parametrized by scalar function instead of a multi-dimentional one. Additionally, the same w-potential ø can be used to generate mirror maps for different action spaces, allowing the insights obtained for one action space to easily generalize w-potential mirror map his induces a Bregman divergence (Bregman, 1967), which is defined as
others. An
Dno Duo (T(-1s), a' (-1s)) = (T((s-)))) ho( - (Vhs (+(s+)),, x(1ss) - T (-(ss)),
where Dro KL-diivegeenc, while we recover the Euclidean distance when ((x) = 2x (refer to Appendix F). When the Bregman divergence is employed the mirror map allows us to control the geometry of the updates of the parameters to be optimized, determining when to take large small updates based on the current value of the parameters.
2 for all y E D. When 6(x) = c=- Dr. is equivalent to the
regularization term in optimization problems, tuning
2.3 Evolution Strategies
OpenAI-ES (Salimans et al., 2017) a popular method be able to optimize non-iifferentiable functions and it has been widely used to meta-learn objectives (Lu et al., 2022; Jackson et al., 2024), as it obtains an unbiased estimate of the gradient (unlike second order gradient methods). The gradient V VCF(C) is estimated using:
E ( (((S-
Ee-A(0.1a))
20
where (0, La) is the multivariate normal distribution, is the number of parameters, F an estimate of F. and >> 0 is hyperparameter regulating the variance of the perturbations.
3 Mirror Preference Optimization
We introduce Mirror Preference Optimization (MPO), a new framework for preference optimization that generalizes DPO and ORPO. We start by replacing the KL-divergence penalty term in the objective in (4) with Bregman divergence and aim solve the problem
' argmax Be
F (e., a) (DD(
(7)
where Dj, the Bregman divergence induced by a mirror map h. This new objective allows to enforce different types of regularization, which, as we show later in the paper. can be tailored to account for specific properties of the preference dataset. Following the same intuition used obtain the DPO objective, we have the following result.
Theorem 3.1. Let his be a O-potential mirror map and nt" be solution the optimization problem in (7). If (als) > 0 for all & E S, E A, we have that
r(T) = 36-
36 (T)e((TT) + c(so),
(8)
where c(su) is a normalization constant that depends only on sp-
"A function h is essentially smooth if lim---xx (Vh())) +o0, where 0X denotes the boundary of X.

---

We provide a proof for Theorem 3.1 in Appendix C. The next step is to model the reward using classification problem based on the reward difference rather than the maximum likelihood problem in (3), as suggested by Tang et al. (2024). That is, our aim is to solve the optimization problem ro(n)))
F E argmax,
where is an increasing function. We give further details on this interpretation of reward modeling in Appendix D. By plugging (8) in the optimization problem in (9), we obtain the objective:
x² E argmax, Ep (T((6_(((TT)) o- (TTTTTt x (T(()) + +o +oo " (Tree( (10)
where ED is equivalent (e
Two-step MPO (2S-MPO) We can use the objective (10) to define a class of two-step PO algorithms, which consist of a preliminary SFT phase to obtain the reference policy Tref and PO phase which optimizes (10). When Φ = e", the (10) is equivalent to (5) and we recover DPO.
One-step MPO (IS-MPO) By adding an SFT term to (10) and by setting the Tref be the uniform distribution, we obtain class of one-step PO algorithms. These algorithms consists in a single phase, where we optimize the objective
x. argmax (= (TT(T)) + Ag (T-'))) - (=T(()))
(11)
where s an w-potential. The term o- (Treet (TI) - uniform. We note that setting Tref to be the uniform distribution is equivalent to replacing the Bregman divergence penalty in (7) with the mirror map h( (( (-)). which enforces a form of entropy regulariza- tion. When (z) = log(z) and o (z) = log (z/ (1 - z)), (11) recovers the ORPO objective in (6).
- - (TTrrr(TT)) (Tw)) has canceled out due to Tee being
Temporally-Awan MPO (TeMPO) Lastly we design variation of MPO that gradually switches from SFT to PO. TeMPO consists of single-phase algorithms that optimize the objective
T argmax,
(IT)) a(t)g ( (I(Tw)) o (z(n))) ], (12)
where a: [0, 1] + [0, 1] is an increasing function of the percentage of training progress.
The objectives in (10), (11), and (12) allow us to implement variety of preference optimization algorithms, while benefiting from a theoretical justification. In the following, we will show that it is possible to parametrize and optimize g, va and Φ to obtain new algorithms that outperform baselines. In Appendix E, we discuss why we chose Bregman divergences rather than f-divergences.
3.1 Meta Learning PO objectives
To search the space of PO algorithms we have defined, we employ a neural network parametrization for g. 2. and o which we optimize using evolutionary strategies (Salimans et al., 2017).
Similarly to Alfano et al. (2024), we parameterize g, Y and Φ-1 as a one layer neural network with 126 hidden units and non-negative kernels, where the activation functions are equally split among:
x, (z) x², (2 (4) where (x)++ (a, 0) and clip(z) = max(min(ar, 1), 0). The non-negative kernels and the increasing activation functions guarantee the monotonicity of g. vi and Φ-1, while the several different activation functions facilitate expressing complex functions. To ensure that we are able to recover the DPO and ORPO objectives, we add a log(z), log(z) and c clog(z/(1 - x)) to the final outputs of g. &b and Φ respectively, where a, b, O.
log((z)+), e",, tanh(z), log(clip(z)/(1 - (llpp((
To search for the best g. s and h-1 within this class, we employ the OpenAI-ES strategy. Denote by Ç the parameters of g, e and o- and by x the final policy obtained optimizing the objective in (11) when using the parametrized s and -1 Lastly, let F()) be the expected cumulative reward of T5, i.e. = = P ())) We then use Adam (Kingma & Ba, 2015) to update the parameters ( using the estimated gradient. In practice, to compute (19), we sample 128 values of e to obtain 256 perturbed objective functions. We then train 256 agents with the perturbed objective functions on an a preference
dataset. To measure the value of each agent, iee. F(C') for all perturbed ç', we sample 100 trajectories on the target environment for each agent, and take the average cumulative reward as estimate for the value of the agent. Refer to Appendix G for further discussion and details the ES methodology.
We consider both the case where we fix g = log a and learn 1a and 111, and the case where we learn all three functions. We perform the evolution on both the two-step and one-step MPO classes.
5

---

starting state
base
noisy
mixed-quality
T1 original agent
rare agent
original agent target agent
random agent
random agent
BT judge
BT judge
BT judge
flip with prob
(Tur Ti)
Figure 1: Data generation for MuJoCo experiments. For cach pair of trajectories, which share the same starting state, two agents are chosen based on the data generating strategy. The trajectories are thenjuugek by Bradley-Terry Judge and, in the noisy setting, the labels are flipped with probability
4 MuJoCo Experiments
Our first set of experiments is carried out on continuous RL tasks in MuloCo. In particular, we show the performance of all the algorithms presented in Table 4 across several settings and we compare it with the performance of our discovered objectives. To maximize computational efficiency, all our MuJoCo experiments are implemented in JAX (Bradbury et al., 2018) using the brax (Freeman et al., 2021) and evosax (Lange, 2022) libraries. We provide an implementation of our methodology here and report hyper- hyper-parament parameters in Appendix K.
4.1 Tasks
To reproduce the typical conditions of LLM fine-tuning, which involve a pre-trained model, we consider setting where the task is to adapt pre-trained agent to meet the original objective while ad- hering to an additional stylistic constraint. Specifically. in the Ant environment, we start from an agent that has been pre- pre-trained on the standard Ant goal of moving forward and enforce the objective of avoiding the use of one of its legs. This is accomplished by introducing the Three-legged antt (TLA) environment, modified version of Ant where utilizing the fourth leg results in significant penalties.
The offline preference optimization task is defined as follows. We train one agent (the original agent) in the original Ant environment, achieving a reward of 6000, and another (the target agent) in the TLA environment, which achieves a reward 3900. For comparison, the original agent achieves a reward of 7700 in the TLA environment. We then generate preference dataset of 1280 rows, each with two trajectories of length 1000 starting from the same state. Each trajectory is generated by either the original the target agent, depending the current setting. A Bradley- Terry judge ranks each pair of trajectories and declares a winner, based on their true cumulative reward. We consider three variations of the preference dataset, each meant to represent common issue of real world data.
Base dataset: for each pair of trajectories, one is generated by the original agent and one by the target agent.
Noisy dataset: same the base dataset but each chosen/rejected pair of labels given by the judge is flipped with probability E.
Mixed Mixed-quality dataset:
each trajectory the dataset is generated by an agent selected at random between the original and the target one. The resulting dataset will consist of, approximately. 25% comparisons between two trajectories from the target agent, 50% comparisons between trajectories of different agents, and 25% comparisons between two trajectories of the original agent.
We also consider training a randomly initialized agent on the Hopper environment. This setting addresses the case where a behavior has be learned from the preference dataset and there no prior knowledge of the task available. We report the results of the experiments for this task in Appendix I.3.

---

Table I: Three Legged Ant (TLA). Performance existing and discovered MPO algorithms on TLA, for various dataset settings, For each syporthm-ddttssee combination, we report the average value and standard error of 25 trained agents. For each discovered MPO algorithm, we specify on which setting it was discovered and report its performance across all settings (with fixed hyperparameters). We underline the highest (or two highest if their confidence interval overlaps) average performance among the noman-designes algorithm and report in bold the overall highest, for each setting.
Base
Mixed Quality
Noisy (8 = 0.1)
Noisy (6 = 0.3)
RRHF (Yuan et al.. 2023)
2789 + 285
2245 = 134
1730 +442
330 = 552
SLiC-HF (Zhao et al., 2023)
3255 66
2478 54
2329 +289
1135 = 224
DPO (Rafailov al., 2024)
3528 + 58
2766 + 89
3082 +80
1519 +14
IPO (Azar et al., 2024)
3618 44
2937 85
3162 66
1133 = 115
CPO (Xu et al., 2024a)
3450 + 55
2322 208
2967 = 58
2000 =35
ORPO (Hong et al., 2024)
3087 322
2500 71
2841 1
1953 3:
R-DPO (Park et al., 2024)
2606 65
2107 + 40
2099 50
1667 + 24
SimPO (Meng et al., 2024)
3683 478
3117 185
2314 + 752
-3828 341
SFT
3287 + 62
2344 + 40
2733 +45
2049 +33
Our algorithms
LPO
3774 + 102
2841 46
3617 69
1569 + 156
With g = logo
IS-MPO (mixed-qualitt)
3206 + 330
3153 274
1319 + 714
-3967 382
1S-MPO (noisy, E 0.1)
3789 60
3210 + 60
3813 + 47
3279 + 83
2S-MPO (mixed-qualitt)
3595 + 57
2785 - 78
3197 58
1687 + 58
2S-MPO (noisy, E 0.1)
3551 + 58
2784 - 63
3190 62
1569 = 122
With parametrized g
IS-MPO (mixed-quality)
3560 + 333
3627 79
3371 + 410
2681 + 251
2S-MPO (mixed-quality)
3736 51
3202 + 64
3488 + 73
2253
+ 125
IS-MPO (noisy, 8 = 0.1)
3861 79
3075 + 87
3724 59
1771
107
2S-MPO (noisy, E 0.1)
3701 + 52
3178
+ 59
3490
+ 95
2074
+ 136
IS-MPO (noisy, E = 0.3)
3931 + 69
3244 55
3834 82
3417 82
Temporally-awen
TeMPO (1)
3577 + 45
2730 + 63
3106 + 62
1971 + 35
TeMPO (2)
3625 49
3088 69
3443 53
1988 39
TeMPO (3)
3352 + 57
2256 + 44
2725 + 46
1923 + 27
4.2
Results
We provide the results of our experiments for TLA in Table 1, which reports the performance of several PO algorithm and of our discovered objectives. We performed hyperparameter search for each algorithm-dataset combination and only report the performance of the best hyperparameters. All algorithms
for 12 epochs over the preference dataset, with the exception of DPO, IPO, SimPO and R-DPO, which are run for 2 epochs after epochs of SFT. We provide an additional noisy setting (s 0.2) and performance for other existing algorithms in Table in Appendix I.1.
We notice that none the human-designed algorithms manages to recover the performance of the tar- get agent and that most of them experience a drop in performance in mixed-quality and noisy settings.
Importance of SFT The first group within Table shows that SFT plays a key role in the perfor- mance across different settings. SimPO, which does not contain an SFT term nor SFT step, is at the top the leaderboard on the base and the mixed-quality setting but performs poorly on all noisy settings. IPO and DPO, which do not contain an SFT term but have an SFT step, are among the top performers on the base, mixed-quality and low noise settings. Their performance finally drops when the noise level reaches 0.3. Lastly, the algorithms that present SFT term in their objectives, e.g. CPO, ORPO and, obviously, SFT, exhibit a subotpimal performance the base and mixed-quality settings but are much more robust to noise than the other algorithms.
7

---

logits
Gradient wut. logits rejected
Gradient logits chesen
logits rejected
logit rejected
Training progress
logit rejected
Train prognss
Oradlent
Gradien
logit chosen
logt chosen
logit chosen
logt, chosen
Figure 2: SimPO
Figure 3: ORPO
Gradient Gradient
Gradient
Gradient rejected
logit rejected
Gra Gradienn Value
Training progress
lagit rejected
Training progross
Gradder
logit
logt choson
logit choson
logi. choson
Figure 4: Discovered IS-MPO (shuffled)
Figure Discovered IS-MPO (noisy)
Figure 6: Absolute value of the gradient of SimPO. ORPO, IS-MPO (shuffled)) and 1S-MPO (noisy.
E = 0.3). The dots are sampled datapoints from the training distribution.
Keep optimizing part of the objective be clipped when (Tr)) > <nn) + 6, rather than when (T.)) (T)).. This modification leads to a higher performance on all tasks, demonstrating that is beneficial to keep optimizing the policy even if (T.)) > (T)). To further stress this point, consider the objective
Furthermore, while RRHF and SIiC-HF are very similar, SliC-HF allows the PO
n" E argmax, (w))>) [} log fog (x) log To [z)]]
To(
which we call Linear Preference Optimization (LPO). LPO corresponds to SLiC-HF with +o0 and obtains better results than most of the existing algorithms in Table 1, confirming that it is important to design objectives that do not flatten when (TT.) > T(T)..
Discovered objectives Table also reports the performance of our discovered objectives, for both the case where we the monotonic transformation g be the logarithmic function and where we parametrize and learn it. We learn a separate objective for each dataset setting and report the performance of each learned objective on settings. Differently from the human-designed algorithms, the discovered objectives recover the performance of the target agent in multiple instances and are more robust to the mixed-que mixed-quality and noisy settings. We note that allowing the evolution procedure to learn leads to better performance in all dataset settings. Additionally, we have that the objectives discovered within the two-step MPO class always have lower performance than those within the one-step MPO class. This is probably due to the ability to modify the SFT term in the one-step MPO class, which not present in the two-step class.
When exploring the one-step MPO class with log our discovery procedure always recovers variation of CPO. That is, we obtain an objective that can be approximated
π E argmax, [E(s..T.)))D (a log( (P((T(TT))) + A log a ( log(T(Tw)) (log(T())))), where the coefficients and depend on the setting. In particular, have low value for and a high value for 3 in the noisy settings, while we observe the opposite in the mixed-quality setting. These results confirm the observations made the hand-crafted objectives, whereby objectives with an SFT term are more robust to noise and objectives without are more robust to mixed-quality trajectories. When we search the two-step MPO class with g log σ, we recover DPO.
Figure 6 provides a visualization of the gradient of some the objectives discovered when we parametrize and meta-learn g. In particular, MPO class on the mixed-quality and noisy (s = 0.3) settings. For comparison, we provide the same plots for the ORPO and SimPO objectives. The band-crafted algorithms present larger gradient when π (T..) < (Tz) and a smaller one when T(T.)) > л ((T)), that is, they induce large updates when the data-point contradicts the current behaviour of the agent, and small otherwise.
show the objectives discovered within the one-step
The objective discovered on the noisy dataset has the opposite behavior, meaning that only optimizes the more robust-to-noise SFT term when ((Tw) < (T²). As log(IT(Tw) log(m(n))
8

---

Table 2: AlpacaEval LLM results. We report win-rates and standard error (length controlled win-rates and standard error in parenthesis) for three combinations of base model and preference dataset. We report in bold font the highest winrate (length-controlled winrate) for each column.
gemma-7b dpo-mix-7h 7k
gemma-7b, capybara-
mistral-7b!, dpo-mix-7k
CPO
28.94:1.6 (21.9-+.0)))
29.2::1.6 (25.3++0.3)
31.0+=1.6 (21.7:+0.3)
ORPO
27.4+1.6 (21.5+00.)
28.2+1166 (24.2++0.3)
28.0+1.66 (21.1+0.3)
DPO
30.7=1.6 (31.0.0.3)
37.9:1.77 (32.8+0.2)
32.8::1.6 (2..5=0..3
SimPO
32.5#1.7 (25.8=0.2)
27.31:66 (23.6:0.2)
28.1 (22.6:0.3)
TeMPO (1)
31.5+1.6 (25.00..)230 (25.0=0.2)
34.2 (30.0+10.2)
39.6:1.77 (30.4:0.2)
TeMPO (2)
27.8+1.6 (23.2=0.3)
33.4+1.77 (29.6+00.2
36.6#1.77 (27.7=0.2)
TeMPO (3)
35.4+1.7 (29.11011)
39.4+1177 033.8:+0.1)
41.6+:1.7 (30.6+10.2)
becomes larger, shifts toward increasingly large updates thanks to the PO term. similar pattern appears in the mixed-quality dataset, where the objective also increases its updates the difference log(nz / (TT() - log(=(n)) grows. The key distinction between these two losses is that the shuffled loss triggers high-gradient updates even when x (Tw) =(m)- We highlight once more that both discovered objectives advocate to keep optimizing the policy even when π (Tww) > (Tt).
4.3 Including temporal awareness
We build an algorithm within the TeMPO family using the insights gained in the previous section. In particular, we keep the standard SFT loss, which was rediscovered in all our experiments, and use SimPO for the PO component of TeMPO, given its high performance and its ability to continue optimizing the policy even when (Tw)) > (T)). That is, define the objective
π E argmax, Ep (1-a(t)) log Te (Tw (r) +a(t) log a (B To (Tw (x) -r) T (ollgg/@((aa))) (-]z))->)) (13)
We try three expressions for designed to put most of the weight on the SFT component of (13) at the start of training and switch to PO towards the end of training:
(1) (l) = 1; (2) al)) = 12; (3) a(l) = (20(x 0.75)).
Table shows promising results for cemporallyawaan PO algorithms, as the objective in (13) with the version (2) of matches or surpasses all human- -deigged algorithms in all settings.
5 Experiments: LLM transfer
We show that the insights obtained in the MuJoCo environments can be transferred to the LLM alignment setting. In particular, test the TeMPO algorithm in (13), which is designed continue to optimize the policy even when π (Tw)) >> Tπ1). as all our discovered algorithms do. We evaluate TeMPO on LLM alignment, comparing its performance against baselines for three combinations of base model and preference dataset, as shown in Table 2.
To tune the LLMs, modify the Alignment Handbook library (Tunstall et al.) to include the TeMPO objective in (13). We evaluate the tuned LLMs against GPT-4, using the AlpacaEval library (Li et al., 2023) and Llama-3.1 1-700-1ornaa Ins--nnnttuute as a judge. For all combinations of starting LLM, dataset, and PO algorithm, we perform 4 update epochs and set the learning rate to 5e-5 and 3 to 0.05. In the case of DPO and SimPO, performed 3 epochs of SFT, with learning rate 5e-5, and 1 epoch of DPO/SimPO, with learning rate 5e-7 and B = 0.05. Refer to Appendix I. for further details.
Table 2 shows the effectiveness of the TeMPO objective in (13), which presents high winrate for all schedules of a. particular, TeMPO with the sigmoid schedule for a has the highest winrate in all tasks and the highest length controlled winrate in out of three tasks. Additionally, TeMPO requires only one stage of training, while DPO and SimPO require two, ie. SFT and PO.
7klo--k-ii -a---------iiiiiee binarrized

---

data is sparse Table 3: We report win-rates and standard error on AlpacaEval (length controlled win-rates and standard error in parenthesis) for gemma-70 7b and two preference dataset.
Algorithm
dpo-mix-7k
capybara-7k
SimPO
30.50 (26.06)
27.52 (25.
SimPO
30.62 (25.35)
27.52 (25.54)
SimPO
32.50 (25.80)
27.95 (25.71)
SimPO
33.42 (28.22)
27.02 (25.07)
SimPO
10
33.60 (27.12)
27.39 (25.00)
SFT+SimPO
30.31 (26.55)
35.07 (28.25)
We also tested the static algorithms discovered in the MuJoCo environment. Those with g log which rediscovered CPO, exhibited performance similar to CPO itself. In contrast, algorithms with a parameterized function, which learned to greedily optimize the policy even when (Tv)) > (T)) performed poorly. We hypothesize that these algorithms overfit to the TLA task, where our datasets sufficiently cover the state-action space. On the other hand, in LLM tuning-wiin relative to the environment greedy methods are more susceptible to over-optimization. At the same time, the objectives discovered TLA only used to logits within -14 and -8, shown in Figure 6, and have less regular shape outside of this subset.
In Table 3, we report additional results that allow us to better understand the influence of different components of TeMPO. particular, we have tested SimPO with 7 = 0, SimPO with 7 > O, and SFT + SimPO with > O, where the last one consists of one-step objective made of the addition between the SFT and the SimPO loss. Table 3 shows that having large 7 which encourages large updates even when π (Tw)) > (T)). or adding an SFT term to the PO loss are both helpful in improving performance. The performance of TeMPO reported in Table 2 is still the highest, which means that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.
6 Conclusion
We have introduced novel framework for Preference Optimization algorithms, as well as methodol- ogy for the automatic discovery of PO algorithms using evolutionary strategies. Through a systematic evaluation across diverse settings in MuJoCo environments, have demonstrated that the per- formance of our discovered objectives consistently exceeds the performance of existing methods, particularly in noisy and mixed-quality datasets where many baselines underperform. Our analysis in MuJoCo also revealed a common shortcoming among current baselines: truncating the loss whenever (Tw)) TT)). Using this insight, we proposed temporally-awee such loss truncation and gradually switches from the SFT step to the PO step. We then tested this objective on an LLM fine-tuning task, achieving significant improvements over existing methods, thereby confirming the broader applicability of our approach.
algorithm, TeMPO, that avoids
Acknowledgments and Disclosure of Funding
Carlo Alfano and Silvia Sapora are supported by the Engineering and Physical Sciences Research Council EP/W524311/. Jakob Foerster is partially funded by the UKI grant EP//0224411/1 (originally selected for funding by the ERC). Jakob Foerster is also supported by the JPMC Research Award and the Amazon Research Award. Patrick Rebeschini is funded by UK Research and Innovation (UKRI) under the UK government's 's Horizon Europe funding guarantee [grant number EP/Y028333/1), Yee Whye Teh acknowledges support from the Ministry of Digital Development and Information (MDDI) under the Singapore Global AI Visiting Professorship Program (Award No. 21VP-2024-002)
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint X3Xx:2333008774 2023.
10

---

Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference-learning based reinforcement learning, 2012. URL https: org/ abs/ 1201 0984.
Carlo Alfano, Sebastian Towers, Silvia Sapora, Chris Lu, and Patrick Rebeschini. Meta-leamiig the mirror map in policy mirror descent. arXiv preprint arXiv:2402.05187, 2024.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. International Conference on Artificial Intelligence and Statisticy, 2024.
Erdem Biyik and Dorsa Sadigh. Batch active preference-bast based learning of reward functions. In Aude Billard, Anea Dragan, Jan Peters, and Jun Morimoto (eds.), Proceedings of The 2nd Conference on Robot Learning, volume 87 of Proceedings of Machine Learning Research, pp. 519-528. PMLR, 29-31 Oct 2018. URL https: / //proc/ee/ngs/gsss.... mlr wttppp//e//i//ill/////t html..
James Bradbury. Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL
bictp://githe wts.thuu.com//// jaz-m/jjxaaaass
Ralph Allan Bradley and Milton E Terry. Rank analysis incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
Lev M. Bregman. The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming, USSR Computational Mathematies and Mathematical Physics, 1967.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 2017.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint urXiv:2402.01306, 2024.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agmostic men---arrin for fast adaptation of deep networks, 2017. URL https: /ttp///aww...rga///////////
C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax differentiable physics engine for large scale rigid body simulation, 2021. URL http: //github. com/ homn/oogle//brar
Alexander David Goldie, Chris Lu, Matthew Thomas Jackson, Shimon Whiteson, and Jakob Nicolaus Foerster. Can learned optimization make reinforcement learning less difficult?, 2024. URL https://arxiv. org/ http////0.00.0 .7007.0782..
Jiwoo Hong, Noah Lee, and James Thome. Reference-fre free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.07691, 2024.
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D) Lee, Wen Sun, Akshay Krishnamurthy, and Dylan Foster. Correcting the mythos of kl-reuularization: Direct alignment without overparame- terization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399 2024.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari, 2018. URL https:://axxt org/ abs/1811.0655
Matthew Thomas Jackson, Chris Lu, Louis Kirsch, Robert Tjarko Lange, Shimon Whiteson, and Jakob Nicolaus Foerster. Discovering temporally-aware reinforcement learning algorithms. In International Conference on Learning Representations, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024.
Mixtral of experts. arXiv preprint

---

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Louis Kirsch, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Improving generalization in meta reinforcement learning using learned objectives, 2020. URL 04098.
Walid Krichene, Syrine Krichene, and Alexandre Bayen. Efficient bregman projections onto the simplex. In IEEE Conference on Decision and Control, 2015.
Robert Tjarko Lange. evosax: Jax-based evolution strategies. arXiv preprint arXiv:2212.0444 2022.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of cestructionfoolowiing models. https: stttps///i //gtthbb........ trub. com/tatsu-llea laaaalllaaaaaaa wawpsauaaaaaapaaaaaaaaaaaaaaaaaa eval, 2023.
Bo Liu, Xidong Feng, Jie Ren, Luo Mai, Rui Zhu, Haifeng Zhang, Jun Wang, and Yaodong Yang. theoretical understanding of gradient bias in meta-reinforcementt learning, 2022.
Chris Lu, Jakub Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, and Jakob Foerster. Discovered policy optimisation. Advances in Neural Information Processing Systems, 2022.
Chris Lu, Samuel Holt, Claudio Fanconi, Alex Chan, Jakob Foerster, Mihaela van der Schaar, and Robert Tjarko Lange. Discovering preference optimization algorithms with and for large language models. arXiv preprint arXiv:2406.0e 084 14, 2024.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward. arXiv preprint arXiv:2405. 14734, 2024.
Luke Metz, C. Daniel Freeman, Samuel S. Schoenholz, and Tal Kachman. Gradients are not all you need, 2022.
Arkadi Nemirovski and David B. Yudin. Problem Complexity and Method Efficiency in Optimization. Wiley Interscience, 1983.
Junhyuk Oh, Matteo Hessel, Wojciech M Czamecki, Zhongwen Xu, Hado P van Hasselt, Satinder Singh, and David Silver. Discovering reinforcement learning algorithms. Advances in Neural Information Processing Systems, 2020.
Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403. 19/59, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 2024.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. Evolution strategies as scalable alternative to reinforcement learning. arXiv preprint arXiv: 100300000 2017.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.077707.06347, 2017.
Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. In ICML 2024 Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024.
Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: unified approach offline alignment. Forty-first International Conference Machine Learning, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
12

---

Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook. URL
attps://github: com/ htw.a/gggiggaceaaaaaaaaaaaaea -ageenn-thhthhoooo handbook.
Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse kl: Generaliz- ing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference Learning Representations, 2023.
P.J. Werbos. Backpropagation through time: what it does and how to do it. Proceedings the IEEE, 18510): 1990. doi: 10.1 109/5558883.
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of I1m performance in machine translation. In Forty-first International Conference on Machine Learning. 2024a.
Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang. Chao Yu, and Yi Wu. Is dpo superior to ppo for IIm alignment? comprehensive study, 2024b. URL https://arae
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-ftrst International Conference Machine Learning, 2024.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback without tears. arXiv preprint arXiv:2304.05332, 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
13

---

A
Baseline PO Algorithms
Table 4: Overview of popular PO algorithms. The objective is to be maximized and (Twi 77) ~ D.
Method
Objective
To max (o. Fral log mo (Twl प log (m(())))
RRHF (Yuan et al., 2023)
SLiC-HF (Zhao et al., 2023)
Alg Mogma(Tulaa)) max(0, 6 log Te
(Tw (o
log (o)
DPO (Rafailov et al., 2024)
log (Blog mo(sula) (Ter(Tulz)
log =s(r212)
mur(ti
IPO (Azar et al., 2024)
(log Roor(Te (o(llla) log meer(Fila)
ma(7))))
CPO (Xu et al., 2024a)
log Te (t) + log o log na (Twla) B log To (m([a))
ORPO (Hong et al., 2024)
log T (Tw) + A Alog a (log (Twdax(Tw)) log (odds= (Ti))
R-DPO (Park et al., 2024)
log (olg Moinuln)
ma(ri=) 3 log Hecr(hiley + (a//wil almrl)
(eer(Fulz)
SimPO (Meng et al., 2024)
log o
[ log (T( (ar) Toel log (m( -7)
These methods include RRHF, which uses sength-normalizer log-likelihood, and SLiC-HF, which uses direct log-likelihood and incorporates SFT. The comparison also includes IPO, a theoretically- based approach that handles pairwise preferences differently than DPO. Another method, CPO, combines sequence likelihood as a reward with an SFT objective. Finally, R-DPO modifies the original DPO by adding regularization to prevent length exploitation.
B Related Work
Automatic Discovery of Preference Optimization Loss Functions Several works in the literature have shown that it is possible to discover machine learning algorithms that outperform algorithms manually designed by researchers (Oh et al., 2020; Lu et al., 2022; Jackson et al., 2024; Alfano et al., 2024). An approach particularly relevant to method is DiscoPOP by Lu al. (2024), which leverages an LLM to discover objective functions for LLM tuning. They consider a different space of objective functions from us, as they replace the log-sigmoid in (5) with a generic loss function, following the framework built by Tang et al. (2024). Additionally, instead of searching over a space of parametrized functions, they ask the LLM generate loss functions in code space. This distinction suggests that our approaches could be complementary, as the model discovered by DiscoPOP could be paired with our learned mirror map. Lastly, DiscoPOP optimizes its objective function directly on the final task, whereas we adopt two-stage paxvess-optmmzinng the loss function on separate task (MuJoCo) and later transferring the LLM setting. This transferability underscores the broader applicability of our approach.
Generalisations of DPO A generalization of DPO alternative to ours is f-PPO (Wang al., 2023), which consists in replacing the KL-divergence in (1) with an f-divergence and then apply the same heuristic DPO to obtain the final objective function. We note that the KL-divergence is the only f- divergence to be also a Bregman divergence, and vice-versa. They empirically demonstrate that differ- ent f f-divergences lead to different balances between alignment performance and generation diversity, highlighting the trade-off's inherent to this class of algorithms. Huang et al. (2024) further explore this class of PO algorithm and individuate an f-divergence for which f-DPO is robust to overoptimization.
C Proof of Theorem 3.1
We provide here a proof for our main result, i.c. Theorem 3.1. The proof to obtain the DPO objective in (5) follows by taking & = ex. Theorem C.1 (Theorem 3.1). Let hy be a O-potential mirror map and x² be a solution to the optimization problem in (7). If Trat (als) > 0 ffrrr all E S,a E A, we have that
r(r) o ((T((T)) o- (TTe((TT) + c(so),
(14)
14

---

for all trajectories T, where c(so) is a normalization constant that depends only on Proof. We use the KKT conditions solve (7), i.c.
n² E argmax Ba Bana -1a {zcr((17))
We use the stationarity condition to obtain the equation
E. Ea-z (1)) (a) (r)), a)] -Dx (Te) - - -
T(T)
T1:50Er'
r'is0Er!
r(r) - 36- (x(r)) + (
X (()) = =0
for all initial states so S and for all trajectories starting from sg. Rearranging, we obtain that (()) ((r(T) 36 (Tt A + a(T))/3))
Since 0 & dom の due to the definition of a O-p 0-potential, and Te (T)((T) > 0, we have that x(7) > 0 for all trajectories т. Invoking the complementary slackness condition, whereby ((T))(T)) for all trajectories т, we have that (T)) 0 for all trajectories т. Therefore, we have that
r(r)
BΦ (TT) + 36 (ore (T)) =
The theorem statement is obtained by rearranging the last equation and denoting c(5o) À
D Reward Modeling
In Equation (9), we utilize the interpretation of reward modeling as a binary classification problem
given by Tang al. (2024), which we summarize here. Let 2 (Ti, 72) be a pair of trajectories and
l E (-11, 1 be the associated label that states whether T1 is preferred to T2 (E = 1) or not (& = -1).
We want to find function {(2) E R such that sign sign({(z) ((()) is good estimate of f. For dataset
D {2, & 1 the classification loss (or 0-1 loss)
L(C, D) Ep sign (())) k)].
(15)
which is often approximated with a surrogate
L D) - Ep [1 (o(z)
(16)
for function f: F - R. This approximation is possible because, when f is decreasing (or convex), (15) and (16) have the same minimizer, as we prove in the following. Denote (z) P(e 1|2). then the conditional surrogate loss is
I x)( = p(())) (R(=) + (1 -p(())) (-R(=))-
(17)
The minimizer e- of (16) is such that (²)) minimizes (17). While the minimizer (17) might not be computable explicitly, we can show (2) > > (1)) > 1/2 when f is a decreasing function, meaning that é. is also the minimizer of (15). Firstly, we have that
Lf((,x) ((fx)-L((-B0) L,(- (1()) (e(=) + (1- pi()))f (11-1(()) (-é(z)
pI(f) (-((z) - (1-p)(z))) (((z)
(18)
(2p)(z) - 1) (1 ( --((t)))
-
Since f is decreasing, have for all ?(=) > 0 that (((=)) <s(-?(=)) Plugging this into (18), obtain that, if (1=) > 1/2, (2, x) < (-E, x) for all ?(z) > 0. Therefore, the minimizer E (=)
of (17) must be positive. The opposite can be proved in the same manner.
Equation (9) can be obtained by setting f = =-g. g. for an increasing function g. and
((2) f(T)) -((2).
15

---

E Bregman divergences vs f-divergences
this section, we discuss why we use Bregman divergences rather than f-divergences in (7). Firstly, we note that, if the reference policy Tef is set to be the uniform distribution, Bregman divergences and f-divergences generate equivalent families of algorithms. Let Φ be a 0-potential. Using a reasoning similar to that of Theorem 3.1, have the following cases:
Bregman divergence
r(Tw) r(n) (xt o (mrd(T))) o (m
+ + (
+aniform Tref
r(Tw) r(n) x= ( - o- 0 (TT (ni)),
f-divergence
r(Tw:) - -(()) (T (Tw)) x (T
+uniform Tyef
r(Tw) - -r() = (I7(( ((]7(((((Tw)))) - - (7
(t (T)()))
where is the set all trajectories and + The two resulting expressions are equivalent, meaning that our IS-MPO class also includes the of f-divergences. Our experiment on the 1S-MPO class therefore explore both Bregman divergences and f-divergences with ES and Table 1 reports the performance of the best divergence found within both divergence classes.
reference policy not uniform, then the two algorithmic classes have the KL divergence as the only intersection. In this setting, we have chosen to focus on Bregman divergences Wang al. (2023) have already shown that, among the f-divergences they considered, the KL-divergence typically offers superior alignment performance. Since we wanted to explore class of algorithms with the objective of finding better alternatives to the KL-divergence, we decided to consider a different generalization of the KL-divergence, i.c. Bregman divergences. In our experiments with g = log, we discovered that the KL-divergence is optimal also within Bregman divergences. On the other hand, found that significant improvements in performance come from modifying, rather than the KL-divergence. We have observed similar results in preliminary MuJoCo experiments where replaced the Bregman divergence with an f f-iieergence, that is that the KL-divergence is optimal among f-divergences.
F Further discussion of w-potentials
We show here two examples of Bregman divergence induced by an w-potential mirror map, that is when (xx) e and when (bx)) x. (fx)) = e²-1, the associated mirror map is defined
(als)
(als) (log(x) + 1)dr
hx(x(-1s))
Σ
(x) Σ
REA
acA
Σ
x(a | log(T(( s)) - ( + T(a
aEA
x(a 5) log( s)).
oEA
which is the negative entropy. Plugging this expression in the definition of Bregman divergence we obtain
Dn(a, h(x) h(w) (Vh(y), y)
Σ
La log(x.) No log(ua) (log(ya) (aa ya)
a
Σ
Ta loglooo////ll
acA
which is the definition of the KL-divergence. If ((x) = 2r, the associated mirror map is defined as
(a(as)
(als)
ho( = Σ
Φ (x)dx =
Σ
2xdr = Σ T(a s)²
aEA
aEA
aEA
16

---

which is the no-m. Plugging this expression in the definition of Bregman divergence we obtain Da(z, y) = h(z) - h(y) - (Vh(y). - y) = Σ
vi (2 - -ya) = Σ
(xa ya)
.EAA
acA
which is the definition of the Euclidean distance.
G Further discussion on Evolution Strategies
Evolution Strategies (ES) represent a powerful, backpropagation-feee method for optimizing complex functions, that has been particularly successful in the context of long-horizon, noisy, and bi-level optimization tasks such as RL and meta-RL. ES, and in particular the OpenAI-ES algorithm (Sal- imans et al., 2017), rely on perturbation-basee sampling to estimate gradients without requiring backpropagation through the entire computational graph. This feature makes ES well-suited for tasks with long computational graphs, for instance algorithms with many updates, where, due to memory constraints, traditional gradient-buse based methods have to resort to gradient truncation, introducing bias (Werbos, 1990; Metz et al., 2022; Liu et al., 2022).
In our setting, we use ES to search for the best &b and Φ within the parametrized class introduced in Section 3.1, so that an agent trained using the objective in (11) achieves the highest value. Denote by Ç the parameters of & and Φh and by T² the final policy obtained optimizing the objective in (11) when using the parametrized to and Φ-1. Lastly, let F(() be the expected cumulative reward (or value) of TT, i.e. F(G) = P)))). At each iteration, we estimate the gradient VcF(() as
2a (F(S+ac) - (iic)-F(ss-00 oc)]
(19)
where NNO Iu) is the multivariate normal distribution, d the number of parameters, F is an estimate of F, and o > 0 is a hyperparameter regulating the variance of the perturbations.
H Further discussion on Online vs Offline Methods
In the domain of RL and preference optimization, the choice between online and offline algorithms presents critical trade-off, influencing computational efficiency, data requirements, and generaliza- tion capabilities. Online methods, such as PPO, iteratively collect and incorporate new data during training. These inherently support exploration of the environment, enabling the discovery of novel strategies or behaviors that are not captured in pre-existing datasets. However, they need feedback for each generated "trajectory" (or response, in the LLM case), which might be expensive to obtain. Online methods are also more complex and particularly sensitive to hyperparameters, often requiring meticulous tuning for stability and efficiency.
Offline algorithms, such as DPO and its variants, rely entirely on pre-collected datasets. These methods are designed for efficiency and simplicity: they don't don't require any additional feedback from users and are therefore particularly effective in scenarios where feedback is delayed or unavailable. However, the reliance static datasets means offline methods may struggle generalize beyond the training data, particularly if the distribution shift between the training dataset and test time distribution is significant. Additionally, the performance of the algorithm is closely tied to the quality of the training dataset: noisy, biased, or corrupt datasets can severely degrade performance, as these methods cannot mitigate such issues through exploration resampling.
In summary, RLHF (i.c., online) is considered the superior approach, particularly when substantial amounts of online labels are accessible. This makes it the industry standard (Xu et al., 2024b). While DPO has been theoretically equated to optimizing using PPO and a reward model trained on an offline dataset, recent empirical research (Tang et al., 2024) has challenged this notion. These studies have demonstrated that online methods, such as PPO, consistently outperform offline methods like DPO. This superiority is attributed to the benefits of on-policy sampling.
While DPO has occasionally outperformed PPO, it's important to note that several studies (Xu et al., 2024b; Song et al., 2024) have consistently shown PPO's overall superiority. DPO's relative strength lies in its simpler training regime, which avoids the complexities associated with reward model inaccuracies. However, DPO's performance is significantly limited by its sensitivity to distribution
17

---

Table 5: Three Legged Ant (TLA). Performance of existing and discovered MPO algorithms on TLA. For each algorithm-datenn dataset combination, we report the average value and standard error of 25 trained agents. For each discovered MPO algorithm, we specify on which setting it was discovered and report its performance across all settings (with fixed hyperparameters), We report in bold the highest (or two highest if their confidence interval overlaps) average performance, for each setting.
Base
Noisy (e 0.1)
Noisy 0.2)
Noisy (6 0.3)
RRHF
2789 285
1730 1442
749 + 498
330 - 552
SLiC-HF
3255 + 66
2329 +289
1964 = 116
1135 = 224
DPO
3528 + 58
3082 =80
2530
1519 140
IPO
3618 + 44
3162 6 66
2392 #136
1133 115
CPO
3450 + 55
2967 58
2427 44
2000 =35
ORPO
3087 + 322
2841 50
2359 43
R-DPO
2606 65
2099 5:
1740 366
1667 +
SimPO
3683 +78
2314 + 752
118 715
-3828 + 341
SFT
3287 + 62
2733 1:45
2345 +
2049 +33
KTO
1534 + 34
1551 + 31
1531 + 49
1442 + 35
f-DPO (Jensen-Shannon)
3621 = 50
3192 + 76
2494 - 101
1633 = 123
Our algorithms
LPO
3774 + 102
3617 + 69
2705 + 370
1569 + 156
With g log
1S- IS-MPO (mixed-quality)
3206 + 330
1319 + 714
-1225 + 944
-3967 + 382
IS-MPO (moisy, E = 0.1)
3789 + 60
3813 + 47
3280 83
3279 + 83
2S 2S-MPO (mixed-quality)
3595 57
3197 + 58
2487 = 110
1687 = 58
2S- 2S-MPO (noisy, E = 0.1)
3551 = 58
3190 = 62
2552 94
1569 = 122
With parametrized
IS-MPO (mixed-quality)
3560 + 333
3371 + 410
3230 + 259
2681 + 251
2S-MPO (mixed-quality)
3736
3488 +
2992 87
2253 + 125
1S-MPO (noisy, 0.1)
3861 + 79
3724 + 59
2845 + 365
1771 + 107
2S-MPO (noisy, = 0.1)
3701 + 52
3490 + 95
2886 + 127
2074 + 136
IS-MPO (noisy, 0.3)
3931
+ 69
3834
+ 82
3735 + 84
3417 = 82
shift, especially when the offline preference data lacks diversity (Song et al., 2024). This limitation becomes particularly evident when querying the model with out-of-distribution data, a common challenge for methods relying solely offline data. To mitigate this issue, DPO-iter (Xu et al., 2024b), which incorporates online data, has been proposed as a potential solution.
MuJoCo Additional Results
L.1 TLA
display additional results for the TLA task Table 5. With respect to Table 1, we replace the mixed-quality setting with noisy setting where the noise parameter is set to 0.2. We also include the KTO (Ethayarajh et al., 2024) and f- f-PPO Censen-Shannon) (Wang et al., 2023) algorithms.
I.2 Simplified expression for discovered objectives
Below, report simplified version of the objectives discovered for the shuffled and noisy (c 0.3) settings, when is parametrized:
C(Tw, n) =0.82CsPT(7w) 1.7(log(Te(TT)) log(mT(()))
(l log( (.33))))))))))))) (T))))
C(TW, T) =0.82CerT(Twl
maxx(1399(ogg(Te log(me(())) 0.110))))... log(=o(7))))
18

---

Table 6: Hopper. Perfomance of existing MPO algorithms on the Hopper setting. The agent is
randomly initialised.
Base
Mixed Quality
Noisy (c = 0.1)
DPO
1796 +: 78
458 +: 82
693 + 131
IPO
2049 + 13
1606 + 91
739 + 118
CPO
2078 +: 12
1078 + 35
1813 + 33
ORPO
2022
+ 15
1039 = 20
1710 33
SimPO
2027 + 15
1460 = 94
1794 + 65
1.3 Hopper Tasks
We consider second set of simulations on MuJoCo, based on the Hopper environment. As for the previous sections, our experiments are implemented in JAX (Bradbury et al., 2018) using the brax (Freeman al., 2021) and evosax (Lange, 2022) libraries. We report our hyper-parameters in Appendix K.
Differently from the TLA task, we consider setting where the agent is randomly initialized and needs to learn a policy from scratch. On Hopper, we train an agent with an expected cumulative reward of 2100 (the expert agent) and an agent with expected cumulative reward of 900 (the bad agent). We generate the preference datasets in the same way we do for TLA, with the exceptions that the number of rows is 5120 and the trajectories are generated by either the expert or the bad agent. We consider the same three variations of the preference datasets used in TLA, where the expert agent corresponds the target agent, and the bad agent corresponds to the original agent. We include more data compared to the TLA setting as it takes more datapoints for the agent to learn from scratch rather than to adapt to a slightly different objective (like in the TLA case).
Our Hopper results confirm our conclusions the TLA setting. All algorithms but DPO come close SimPO
matching the performance of the expert agent (2100), with CPO being the best. We the only algorithm that significantly outperforms the bad agent (performance of 900) in the Mixed Quality setting (the for SimPO was set very high, 7 10, lower value significantly limited performance).
1.4 Further Mu.JoCo Analsys
In addition to the noisy dataset, we also considered bad judge setting, where the judge would be more likely to swap the label of a pair of trajectories their ground truth rewards were closer to each other. This is practically implemented an increase in the temperature of the Bradley-Terre judge. However, we did not notice significantly different results compared to the simple noisy setting, therefore detailed results are not reported.
J Further discussion on Meta-Learning Algorithms
Meta-learning, or "learning to learn", has been extensively employed to automate the design of algo- rithms that can either adapt rapidly with minimal data samples or generalize effectively to unseen data, tasks, environments. The development of broadly applicable algorithms is particularly critical in the context of preference optimization for LLMs. Here, L.LMs are fine-tuned on relatively small datasets of offline data but must generalize to virtually infinite range of potential user queries. Prior work in meta-learning has demonstrated success in developing generalizable optimization algorithms and loss functions (Lu et al., 2022; Jackson et al., 2024; Lu et al., 2024; Goldie et al., 2024; Kirsch et al., 2020).
At its core, meta-leaming is defined as a bilevel optimization problem with an inner and an outer loop. The inner loop consists in an iterative optimization algorithm that trains agents to solve predetermined task given set of meta- pattaaaprameerss.... The outer loop consists in evaluating the agents trained in the inner loop and update the meta petta-parmeterss accordingly. following some optimization method like second order gradient descent (Finn et al., 2017). The evaluation of the agents typically done on held-out dataset in supervised learning or by sampling trajectories on the environment simulator in RL (Lu et al., 2022; Jackson et al., 2024). In our setting, the inner loop is the offline preference optimization algorithm, while the outer loop is the agent evaluation on the environment (online) and the update of the meta-parameters Ç.
19

---

K MuJoCo Hyper-parameters
We give the prper-parameters for training. The hyper-parameters specific to each algorithm are tuned for each task-data type combination. All the experiments were conducted on 4 NVIDIA L40S GPUs.
Table 8: Hyper-parameter settings of OpenAI-ES.
Table 7: Hyper-parameter settings for PO,
Parameter
Hopper
TLA
Parameter
Value
Population Size
256
256
Number of epochs
12
Number of generations
128
256
Minibatch size
Sigma init
0.03
0.03
Learning rate
le-3
Sigma Decay
0.999
0.999
Max gradient norm
1.3
Learning rate
0.02
0.02
L LLM Hyper-parameters
We give the hyper-parameters we use for LLM training. All the experiments were conducted on NVIDIA L40S GPUs.
Table 9: Hyper-parameter settings for LLM Training.
Parameter
Value
Gradient Accumulation Step
32
Batch Size
Total Batch Size
64
LoRA
Yes
LoRA Rank
128
LoRA Alpha
256
Lora Dropout
0.05
Max length
2048
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Sections 2-5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
21

---

Justification: See Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Sections 4 and 5 and Appendix J.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
htaplc/guidee//odeeeeeee wwwpemissssoooooioiooooooooomoee Policyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //iips.coo. wttpccccpppccceeeeccecccccccccct /tte:////egiieeeeeeuuuuuiiiiii
cetpuummmsssoonnii wsiinnoll onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each quantity is reported with standard error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix J.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: /rl::///nenn /t///eeuris.... http:////cccpp.ci/..ccccteeeeee.
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper presents algorithms for LLM tuming. Misuse of these algorithms for training LLM produce undesirable, unethical, or harmful outputs could be possible by user, but it is not a direct impact of our work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Reference section.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https for what should or should not be described.
27


Reviews:
[{"review_id": "5gXHARyYEZ", "reviewer": "Reviewer_wh5w", "summary": "This paper proposes a new framework, Mirror Preference Optimization (MPO), for aligning models with human preferences. Instead of evaluating Preference Optimization (PO) methods directly on large language models (LLMs), which is costly and noisy, the authors construct a controlled benchmark using MuJoCo environments to simulate preference datasets. They formalize a general class of PO algorithms using mirror descent and employ evolutionary strategies to discover algorithms tailored to various dataset properties (e.g., noise, quality). These discovered algorithms outperform existing PO methods in MuJoCo settings. Leveraging insights from these experiments, they propose a temporally-aware MPO (TA-MPO) algorithm, which shows improved performance in real LLM alignment tasks.", "questions": "1. Can you explain the insights that allow transfer from MuJoCo to LLMs?\n2. Can you do more LLM experiments?\n3. Why do you use Bregman divergence instead of f-divergence?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nProposes a unified framework (MPO) that generalizes DPO/ORPO and allows algorithm discovery via mirror descent and ES.\n\nExperiments on MuJoCo are systematic and cover different noise and data quality settings.\n\nThe discovered algorithms outperform baselines in these synthetic benchmarks.\n\n**Weaknesses**\n\nRelated work is incomplete—missing citations like SPO and several recent LLM-specific PO methods.\n\nEvaluating PO methods on simple benchmarks is not novel and has been done before.\n\nThe claimed transfer from MuJoCo to LLM lacks clear insight or analysis showing what actually generalizes.\n\nLLM experiments are limited and only form a small part of the paper.", "was_revised": true, "final_justification": "While initially I have some concerns, they are addressed by the authors in the discussion period. Thus, I recommend acceptance.", "rebuttal": "We thank the reviewer for their valuable feedback. Please find a reply to your concerns below. \n\n**[Additional LLM experiments]** We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\\gamma = 0$, SimPO with $\\gamma>0$, and SFT + SimPO with $\\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper. We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis. We report the results below. \n| Algorithm   | $\\gamma$  | dpo-mix-7k       | capybara-7k     |\n|-------------|----|------------------|------------------|\n| SimPO       | 0  | 30.50 (26.06)    | 27.52 (25.12)    |\n| SimPO       | 1  | 30.62 (25.35)    | 27.52 (25.54)    |\n| SimPO       | 2  | 32.50 (25.80)    | 27.95 (25.71)    |\n| SimPO       | 5  | 33.42 (28.22)    | 27.02 (25.07)    |\n| SimPO       | 10 | 33.60 (27.12)    | 27.39 (25.00)    |\n| SFT+SimPO   | 1  | 30.31 (26.55)    | 35.07 (28.25)    |\n\nThese new results show that having a large $\\gamma$, which encourages large updates even when $\\pi(\\tau_w)>\\pi(\\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.\n\n**[Related works]** We provide discussions on related works in the Introduction, in Section 2.1, and in Appendices A and B. Additionally, we test eight existing preference optimization algorithms on our MuJoCo benchmark. Should the reviewer want us to test more algorithms, we would be glad to test any additional proposed algorithm. As to the proposed SPO method, we found four possible candidates in the literature, which we discuss below. Please let us know which one you would like us to include in our work.\n\n- Self-Play Preference Optimization [1]: the authors propose a method to tackle non-Markovian, intransitive, and stochastic preference models. While their method can be applied to the offline setting, i.e. ours, the authors do not perform any experiment in this setting. Additionally, their method involves estimating a preference function, which does not fall in the DPO-style losses we consider in our work.\n\n- Soft Preference Optimization [2]: the authors propose a DPO-style loss where the KL-divergence penalty between the current and the reference policy does not depend on the preference dataset. The KL-divergence is instead estimated by sampling trajectories from the current policy every $T$ policy updates, for a hyperparameter $T$. While the authors have not released code for their implementation, we can implement this algorithm should it be the one meant by the reviewer.\n\n- Self-supervised Preference Optimization [3]: the authors propose a method to generate synthetic data and to use it alongside the original preference dataset to train an LLM. Although their proposed loss is similar to the DPO loss, methods that involve synthetic data are beyond the scope of our work.\n\n- Step-by-step Preference Optimization [4]: the authors propose a method for post-training diffusion models using preference optimization. The method is tailored to the setting of diffusion models and cannot be straightforwardly applied to our setting.\n\n**[Objective of our MuJoCo benchmark]** Evaluating PO algorithms on simpler environment has been done before in the literature [1, 5]. However, we are not aware of a recent work that has performed a comprehensive comparison of PO algorithms in a simple benchmark, or that has used the simple benchmark to obtain insights transferrable to the LLM alignment setting.\n\n\n**[Bregman divergences vs $f$-divergences]** In our work, we choose to use Bregman divergences because $f$-divergences have already been explored by previous works in the literature [6], while the application of Bregman divergences to PO was missing from the literature.\n\n\n\n[1] Swamy, Gokul, et al. \"A Minimaximalist Approach to Reinforcement Learning from Human Feedback.\" International Conference on Machine Learning, 2024.\n\n[2] Sharifnassab, Arsalan, et al. \"Soft preference optimization: Aligning language models to expert distributions.\" arXiv preprint arXiv:2405.00747, 2024. \n\n[3] Li, Jian, et al. \"Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness.\" Findings of the Association for Computational Linguistics: EMNLP 2024.\n\n[4] Liang, Zhanhao, et al. \"Aesthetic post-training diffusion models from generic preferences with step-by-step preference optimization.\" Proceedings of the Computer Vision and Pattern Recognition Conference, 2025.\n\n[5] Xu, Shusheng, et al. \"Is dpo superior to ppo for llm alignment? a comprehensive study.\" arXiv preprint arXiv:2404.10719, 2024.\n\n[6] Wang, Chaoqi, et al. \"Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints\". International Conference on Learning Representations, 2023."}, {"review_id": "H8WPr1049b", "reviewer": "Reviewer_FmSK", "summary": "This paper studies preference optimization (PO) algorithms for aligning both reinforcement learning agents in simulated control environments and large language models (LLMs) using human feedback. Motivated by the high cost and instability of evaluating PO algorithms directly on LLMs, the authors design a MuJoCo-based diagnostic benchmark, simulating preference datasets with varying quality and noise. The authors empirically show that their discovered objectives outperform baselines on MuJoCo TLA tasks, particularly in noisy and mixed-quality datasets, and that a variant (TA-MPO) yields competitive or superior results in LLM alignment.", "questions": "Please refer to Strengths And Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The authors alleviate an important evaluation challenge in PO by introducing a controlled and interpretable MuJoCo benchmark to probe PO algorithms’ behaviors under varying data noise and quality, enabling better comparisons than LLMs alone.\n\n2. The MPO framework is interesting, extending and generalizing well-known PO algorithms (DPO, ORPO) via mirror maps and Bregman divergences. \n\n3. The approach shows robustness and improved performance in high-noise and mixed-quality data cases.\n\n4. This paper provides gradients for SimPO, ORPO, and discovered MPO losses, illustrating where these losses induce significant parameter updates; such visual inspection aids interpretability of the algorithms’ behavior.\n\n\nWeaknesses\n\n1. While the paper claims practical transferability to LLM alignment, the experiments in Table 2 only cover AlpacaEval. The degree of improvement, although sometimes statistically significant, is not overwhelming. It’s not clear how robust TA-MPO will be across other important alignment tasks, adversarial data, or human preferences beyond the datasets tested.\n\n2. As noted in Section 5, algorithms discovered with parameterized $g$ overfit the MuJoCo setting and transfer poorly to LLMs, raising concerns about generalizability and task-specific tuning rather than finding universal improvements.\n\n3. Table 2’s reliance on an automatic judge (Llama-3.1-70B-Instruct) rather than diverse human annotators or more comprehensive human-based preference collection somewhat weakens claims of improved LLM alignment in practice.", "was_revised": true, "final_justification": "Thank authors for the response, I will maintain my positive score. Moreover, my current assessment is that this is a borderline paper.", "rebuttal": "We thank the reviewer for their valuable feedback. Please find a reply to your concerns below.\n\n**[LLM benchmarks]** We remark that our LLM experiments have been performed in three different settings, for different combinations of language model (gemma-7b and mistral-7b) and preference dataset (dpo-mix-7k and capybara-7k). Only the evaluation was done on AlpacaEval, which is a standard benchmark for evaluating LLM performance (see for instance [1, 2]). We also highlight that automatic judges present a high level of agreement with human judges, as reported in the AlpacaEval repository. \n\n**[Transfer to LLM alignment of discovered objective]** While the meta-learned optimizers can overfit to particular tasks, our paper shows that we can still extract generalizable knowledge from them. We provide additional results regarding the transfer of the insights obtained from MuJoCo below.\n\n**[Additional LLM experiments]** We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\\gamma = 0$, SimPO with $\\gamma>0$, and SFT + SimPO with $\\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper.  We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis. We report the results below. \n| Algorithm   | $\\gamma$  | dpo-mix-7k       | capybara-7k     |\n|-------------|----|------------------|------------------|\n| SimPO       | 0  | 30.50 (26.06)    | 27.52 (25.12)    |\n| SimPO       | 1  | 30.62 (25.35)    | 27.52 (25.54)    |\n| SimPO       | 2  | 32.50 (25.80)    | 27.95 (25.71)    |\n| SimPO       | 5  | 33.42 (28.22)    | 27.02 (25.07)    |\n| SimPO       | 10 | 33.60 (27.12)    | 27.39 (25.00)    |\n| SFT+SimPO   | 1  | 30.31 (26.55)    | 35.07 (28.25)    |\n\nThese new results show that having a large $\\gamma$, which encourages large updates even when $\\pi(\\tau_w)>\\pi(\\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.\n\n[1] Lu, Chris, et al. \"Discovering preference optimization algorithms with and for large language models.\" Neural Information Processing Systems, 2024.\n\n[2] Meng, Yu, Mengzhou Xia, and Danqi Chen. \"Simpo: Simple preference optimization with a reference-free reward.\" Neural Information Processing Systems, 2024."}, {"review_id": "3LXIZr3QBD", "reviewer": "Reviewer_cgc5", "summary": "In this paper, the authors introduce a generalized class of direct preference alignment approaches that build on some theoretical techniques from information geometry. Specifically, in contrast to classical RLHF objective (Eq. 4), they replace the KL-divergence term with a Bregman divergence term (Eq. 7) based on \"mirror maps\" (the details of which are included in 2.2. While I wasn't familiar with these concepts, I found this part easy to follow). They key advantage of this formulation over a standard KL-divergence term is summarized starting on line 140: \"When the Bregman divergence is employed as a regularization term in optimization problems, tuning the mirror map allows us to control the geometry of the updated of the parameters to be optimized, determining when to take large or small updates...\"\n\nMost interestingly, many mirror maps can be defined that induce different Bregman divergences, each of which will enforce different types of regularization (this includes a temporal-aware variant, the learns to switch between different losses). They show that DPO and ORPO, two popular direct preference alignment approaches, employ a particular set of mirror maps (their full objective and derivation of DPO and ORPO importantly relies on Theorem 3, which follows a similar derivation to what's used in the original DPO). What's more, they propose learning new mirror maps using meta-learning via the OpenAI-ES method in order to discovery preference-tailored losses. \n\nTheir experiment setting is non-standard, and first involves what they call \"MuJoCO experiments\" and variations of the ant simulation (clarifying what MuJoCO is and stands for early on would be helpful for those who are not familiar). In brief summary, they use the MuJoCO environment to generate different ant simulation scenarios with specific properties (e.g., stylistic variations, systematically injected noise) and synthetically generate preference data from these scenarios. The intuition (as I understand) is use such data to demonstrate how their approach is able to adapt to the pecularities of each preference distribution and learn the corresponding mirror maps via OpenAI-ES. They find that their approach is able to effectively discover/meta-learn improved losses (Table 1). To show that their approach also works in a more conventional LLM preference optimization setting, they report small scale experiments in Table 2 where their approach is competitive with a comprehensive set of standard approaches (CPO, ORPO, DPO, SimPO).", "questions": "-- Why are the length controlled win-rates put in the parethenses in Table 2, shouldn't this be the main metric to report? Please clarify. \n\n-- Can you explain why you only use these 7k preference mixtures? Do you think this would be the same if you scaled this? Have you tried any scaled experiments?", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# strengths \n\n-- A new theoretical framework for direct preference optimization from information geometry (MPO) that broadens our understanding and the scope of current approaches. This approach not only provides an original view on existing losses, but provides a means for defining many new classes of losses. \n\n-- An empirical demonstration that such a framework can yield improved losses, especially through meta- and evolutionary learning. I could imagine that this part about learning the parameters of their MPO could influence many follow up studies, especially following up on their small scale \"LLM transfer\" study in Section 5.  \n\n# weaknesses \n\n-- (minor) Given the non-standard nature of their MuJoCO expeirments, a figure illustrating their data generation process and a running example would be very helpful, since the details are otherwise hard to follow. \n\n-- (minor) I also found figure 5 to be confusing, an intuition for why the absolute value of the gradient is meaninful to show seems to be missing. Please clarify this.", "was_revised": true, "final_justification": "I found that the rebuttal clarified the important questions I had. Given that my score was already high, I decided to keep it that way, also after considering the feedback of other reviewers and their rebuttals.", "rebuttal": "We thank the reviewer for their valuable feedback. Please find a reply to your concerns below.\n\n**[Objective of Figure 5]** Understanding the gradient of an objective function is fundamental when analyzing gradient-descent-based methods. We plot the gradient of the objective functions to provide a visual representation of their behavior and to understand where they induce large or small updates. We plot the absolute value as the gradient w.r.t. the chosen trajectory is always positive and the gradient w.r.t. the rejected trajectory is always negative.\n\n**[LLM benchmark metrics]** We agree with the reviewer, we will report the length controlled win rates as the main metric and report the win rates in parentheses.\n\n**[LLM benchmark]** We performed our experiments on datasets with 7k samples as our computational resources do not allow for a comprehensive comparison on a larger dataset like ultrafeedback [1]. We highlight that other works have performed their LLM experiments exclusively on small datasets (e.g. [2]) and that [3] have shown that training on a smaller but higher quality dataset can lead to a superior performance than training on a larger dataset. \n\n**[Data generation figure]** Unfortunately, this year's guidelines prevent us from sharing a figure in the rebuttal. We will include a figure explaining the data generation procedure in a camera ready version of this paper.\n\n**[Additional LLM experiments]** We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\\gamma = 0$, SimPO with $\\gamma>0$, and SFT + SimPO with $\\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper. We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis, for consistency. We report the results below. \n| Algorithm   | $\\gamma$  | dpo-mix-7k       | capybara-7k     |\n|-------------|----|------------------|------------------|\n| SimPO       | 0  | 30.50 (26.06)    | 27.52 (25.12)    |\n| SimPO       | 1  | 30.62 (25.35)    | 27.52 (25.54)    |\n| SimPO       | 2  | 32.50 (25.80)    | 27.95 (25.71)    |\n| SimPO       | 5  | 33.42 (28.22)    | 27.02 (25.07)    |\n| SimPO       | 10 | 33.60 (27.12)    | 27.39 (25.00)    |\n| SFT+SimPO   | 1  | 30.31 (26.55)    | 35.07 (28.25)    |\n\nThese new results show that having a large $\\gamma$, which encourages large updates even when $\\pi(\\tau_w)>\\pi(\\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.\n\n[1] HuggingFaceH4/ultrafeedback\\_binarized\n\n[2] Lu, Chris, et al. \"Discovering preference optimization algorithms with and for large language models.\" Neural Information Processing Systems, 2024.\n\n[3] Deng, Xun, et al. \"Less is more: Improving llm alignment via preference data selection.\" arXiv preprint arXiv:2502.14560, 2025."}, {"review_id": "1yGC5DxlNr", "reviewer": "Reviewer_B54D", "summary": "This paper proposes a unified formulation of the existing Preference Optimisation (PO) algorithm. Significantly, it replaces the original KL divergence with the Bregman divergence and introduces an evolutionary term to control the percentage of training progress from SFT to PO.  Based on the unified formulation (meta-learning objective, such as 1step/2-step MPO and Temporally-Aware MPO, TA-MPO), they evaluate existing PO algorithms in a synthetic experiment environment, MuJoCo, in which you can control the reward structure as the reward datasets are well-defined. Beyond the synthetic setting, the authors also show the results when transferring to LLM. Specifically,  they implement the learning objective (TA-MPO) and compare with existing PO algorithms, such as DPO, SimPO, etc. The experimental results show the effectiveness of TA-MPO.", "questions": "1. As far as I understand, this meta-objective (I) replaces KL with Bregman divergence (2), introducing an adaptive factor transition from SFT to PO, so what are the separate effects for the two designs?\n2. Why TA-MPO (1), TA-MPO (2)  perform very differently? any hypermeter suggestions in different practical settings?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper proposes a unified theoretical formulation of existing preference optimisation algorithms, which provide a comprehensive and fundamental perspective to compare and improve the PO algorithms.\n2. The meta-objective shows effectiveness in both synthetic and LLM PO settings.\n\n\nWeakness:\n1. The verification in LLM settings is rather limited compared with the synthetic settings, so it is unclear if the objective can work well and be generalisable in real LLM training. Also, the performances of TA-MPO (1), TA-MPO (2) vary a lot.", "was_revised": true, "final_justification": "I keep my original positive evaluation score, as it introduces an evolutionary term to control the percentage of training progress from SFT to PO, and empirical results in a controlled environment show its efficiency.", "rebuttal": "We thank the reviewer for their valuable feedback. Please find a reply to your concerns below.\n\n**[Separate effects of Bregman divergences and temporal awareness]** The objective in Equation (12) that we propose for LLM alignment takes advantage of the insights we developed on MuJoCo. The first is that for $g=\\mathrm{log}$ we rediscover the KL-divergence, so we set $\\phi^{-1}(x)= \\mathrm{log}(x)$, as in DPO or in SimPO. The second is that it is important to keep optimizing even when $\\pi(\\tau_w)>\\pi(\\tau_l)$, which is why set $g$ so that it introduces a constant $\\gamma$ as in SimPO. The third is that objectives with an SFT term are more robust to noise, which is why we add an SFT term in Equation (12). The fourth is that a gradual transition from SFT to PO helps performance. In the next paragraph, we show that each of the last three insights leads to improved performance in the LLM setting.\n\n**[Additional LLM experiments]** We have performed additional LLM experiments to better understand the influence of different components of TA-MPO. In particular, we have tested SimPO with $\\gamma = 0$, SimPO with $\\gamma>0$, and SFT + SimPO with $\\gamma>0$, where the last one consists of a one-step objective made of the addition between the SFT and the SimPO loss. The settings and hyperparameters are the same as in Section 5 of our paper.  We use the gemma-7b model and show the win-rate as main metric and the length controlled win-rate in parenthesis.  We report the results below. \n| Algorithm   | $\\gamma$  | dpo-mix-7k       | capybara-7k     |\n|-------------|----|------------------|------------------|\n| SimPO       | 0  | 30.50 (26.06)    | 27.52 (25.12)    |\n| SimPO       | 1  | 30.62 (25.35)    | 27.52 (25.54)    |\n| SimPO       | 2  | 32.50 (25.80)    | 27.95 (25.71)    |\n| SimPO       | 5  | 33.42 (28.22)    | 27.02 (25.07)    |\n| SimPO       | 10 | 33.60 (27.12)    | 27.39 (25.00)    |\n| SFT+SimPO   | 1  | 30.31 (26.55)    | 35.07 (28.25)    |\n\nThese new results show that having a large $\\gamma$, which encourages large updates even when $\\pi(\\tau_w)>\\pi(\\tau_l)$, or adding an SFT term to the PO loss are both helpful in improving performance. The performance we report for TA-MPO in our paper is still the highest, meaning that temporal awareness also contributes to a better performance. We can therefore confirm that each of the insights we gained in the MuJoCo benchmark transfers to the LLM alignment setting.\n\n\n\n**[Performance of different TA-MPO algorithms]** The difference in performance of the different TA-MPO algorithms is due to the fact that they induce different learning rate schedules for the SFT and PO terms. As shown by [1], these types of hyperparameters can have a strong influence on the final performance of the algorithm, which is why we explored a few variations for the temporal awareness schedule $\\alpha$.\n\n[1] Meng, Yu, Mengzhou Xia, and Danqi Chen. \"Simpo: Simple preference optimization with a reference-free reward.\" Neural Information Processing Systems, 2024."}]

Author rebuttal:

Meta-review: All reviewers agreed on the primary strength of this paper: it introduces a generalisation of existing preference optimisation algorithms that includes DPO, ORPO, and others; and, on a set of controlled MuJoCo experiments demonstrates its efficiency. The meta-learning aspect appears to be able to discover novel approaches that are useful particularly in high-noise settings, and provide some insight into performance of existing hand-designed algorithms. The primary concerns from reviewers were that the transferability to LLMs (or indeed, general applicability to LLMs where these algorithms are usually applied).

However, after the rebuttal phase all reviewers agreed that the paper deserved acceptance (though the scores are still borderline), on the assumption that the final version wil incorporate the additional results and discussion promised by the authors.
Decision: Accept (poster)