Learning from negative feedback, or positive feedback or both
Authors: Abbas Abdolmaleki
Bilal Piot
Bobak Shahriari
Jost Tobias Springenberg
Tim Hertweck
Michael Bloesch
Rishabh Joshi
Thomas Lampe
Junhyuk Oh
Nicolas Heess
Jonas Buchli
Martin Riedmiller
Venue: ICLR 2025 Spotlight (2025)
Keywords: Preference Optimization
Policy Optimization
Negative Feedback
Positive feedback
Reinforcement Learning
Probabilistic Inference
Primary area: reinforcement learning
TL;DR: A new policy optimization algorithm that learns from different type and number of feedback (positive, negative, or both) to optimize policies.

Abstract: Existing preference optimization methods often assume scenarios where paired preference feedback (preferred/positive vs. dis-preferred/negative examples) is available. This requirement limits their applicability in scenarios where only unpaired feedback—for example, either positive or negative— is available. To address this, we introduce a novel approach that decouples learning from positive and negative feedback. This decoupling enables control over the influence of each feedback type and, importantly, allows learning even when only one feedback type is present. A key contribution is demonstrating stable learning from negative feedback alone, a capability not well-addressed by current methods. Our approach builds upon the probabilistic framework introduced in (Dayan and Hinton, 1997), which uses expectation-maximization (EM) to directly optimize the probability of positive outcomes (as opposed to classic expected reward maximization). We address a key limitation in current EM-based methods: they solely maximize the likelihood of positive examples, while neglecting negative ones. We show how to extend EM algorithms to explicitly incorporate negative examples, leading to a theoretically grounded algorithm that offers an intuitive and versatile way to learn from both positive and negative feedback. We evaluate our approach for training language models based on human feedback as well as training policies for sequential decision-making problems, where learned value functions are available.

Published as a conference paper at ICLR 2025
LEARNING FROM NEGATIVE FEEDBACK, OR POSITIVE FEEDBACK OR BOTH
Abbas Abdolmaleki, Bilal Piot, Bobak Shahriari, Jost Tobias Springenberg Tim Hertweck, Rishabh Joshi, Junhyuk Oh, Michael Bloesch, Thomas Lampe Nicolas Heess, Jonas Buchli, Martin Riedmiller Google DeepMind
ABSTRACT
Existing preference optimization methods often assume scenarios where paired preference feedback (preferred/positiiee vs. dis-preferrednnegatiee examples) is available. This requirement limits their applicability in scenarios where only unpaired feedback- example, either positive or negative- is available. To address this, we introduce a novel approach that decouples learning from positive and negative feedback. This decoupling enables control over the influence of each feedback type and, importantly, allows learning even when only one feedback type is present. key contribution is demonstrating stable learning from negative feedback alone, capability not well-addressed by current methods. Our approach builds upon the probabilistic framework introduced in (Dayan & Hinton 1997). which uses inrectationmmxximizaee
(EM) to directly optimize the probability of positive outcomes (as opposed to classic expected reward maximization). We address key limitation in current EM-based methods: they solely maximize the likelihood of positive examples, while neglecting negative ones. We show how extend EM algorithms to explicitly incorporate negative examples, leading to a theoretically grounded algorithm that offers an intuitive and versatile way to learn from both positive and negative feedback. We evaluate our approach for training language models based on human feedback as well as training policies for sequential decision decision-making problems, where learned value functions are available.
INTRODUCTION
The use of preference annotated data for training machine learning models has long history going back to early algorithms for recommender systems and market research (Guo & Sanner 2010 Boutilier 2002 Bonilla al., 2010 These days preference optimization algorithms are receiving renewed attention since they are a natural candidate for shaping the outputs of deep learning systems, such as large language models (Ouyang et al., 2022] Team (222a) or control policies, via human feedback (Christiano et al., 2017] Rafailov et al. 2023] Azar et al. 2023). Arguably, preference optimization algorithms can also natural choice even when direct human feedback not available but one instead aims to optimize machine learning model based on feedback from hand-coded or learned critic function (judging desirability of solutions). Here preference optimization methods are useful since they let optimize the model to achieve desired outcomes based on relative rankings between outcomes alone (rather than requiring absolute labels or carefully crafted reward functions).
Among preference optimization approaches, those based on directly using preference data as opposed to casting preference optimization as reinforcement learning from (human) feedback - such as DPO (Rafailov access to an offline dataset of paired preference data, and are fairly robust to application domain and hyperparameter settings. However, algorithms within this class make specific assumptions tailored to their application domain. They were designed to optimize LLMs from human feedback in the form of comparisons of generated sentences and thus, by design, require paired preference data (since they directly model a specific choice of preference distribution). We are interested in finding algorithms that are more flexible, and applicable in settings where the assumptions underlying DPO do not apply.
al.] 2023), have emerged particularly successful since they only require
Corresponding Author: Abbas Abdolmaleki <aabdolmaleki@ google.com>

---

Published as a conference paper at ICLR 2025
In this work we take fresh look preference optimization from a probabilistic inference perspective that has been used with great success in the literature on KL regularized reinforcement learning (Dayan) & Hinton 1997; Peters et al. 2010: Abdolmaleki et al. 2018). We find that from this perspective simplified approach to preference optimization can be derived that is intuitive to understand and is capable of leveraging arbitrary number of unpaired preferred or dis-preferred outcomes, even solely one type (positive or negative) of preference feedback. In particular, our method is able to learn even if exclusively positive or negative examples are available. Formally, our method involves an objective consisting of three log likelihood terms that are derived from first principles: maximizing the likelihood of preferred outcomes, minimizing the likelihood of dis tis-preferred outcomes, while staying close to a reference distribution (see equation 10). We show the effectiveness of our method across a wide range of benchmarks including synthetic benchmarks, training policies for continuous control, and training large language models (LLMs) from human feedback.
2
RELATED WORK
2.1 RL AS INFERENCE
Viewing reinforcement learning through the lens of probabilistic inference offers an alternative framing of RL (Dayan & Hinton [997). This "RL as inference" perspective has gained consider- able attention recently (Levine 2018) inspiring various expectatoo-maximizatio (EM) based RL algorithms (Peters al.] 20100 Abdolmaleki et al. 2018). Essentially, these policy improvement algorithms can be viewed as performing EM to optimize the likelihood of a successful outcome. However, a limitation of these algorithms their reliance on successes (preferred outcome) data. In this paper, we extend this framework to incorporate dis dis-preference information; effectively allowing the policy to make unwanted outcomes less likely. We show that this alone can have an positive effect on data efficiency and performance on certain tasks, notwithstanding the added flexibility.
2.2 PREFERENCE OPTIMIZATION
Preference optimization methods like Direct Preference Optimization (DPO; Rafailov et al. 2023 and Identity Preference Optimization (IPO; Azar et al.) (202)) have enjoyed much attention lately. especially in the LLM training literature. This success is mostly due to a so-called direct optimization of human preferences, in contrast to reward model training required in RL from human feedback (RLHF) training pipelines. Nevertheless, these preference optimization methods were designed specifically to learn from a particular type of data: pairs preferred and dis-preferred data, usually coming from humans indicating their preference over pair of LLM responses to their query. This can be restrictive in scenarios where multiple outcomes need to be considered, and DPO has since been extended to multiple generations and compared to novel method Efficient Exact Optimization (EXO:Ji et al. 2024). both shown to outperform the RLHF baseline in cases where a reward model is available. In this paper, we leverage the RI. inference framework generalize preference optimization even further, allowing for more general algorithms derived from first principles. Our approach can not only handle scenarios with multiple generations but it also naturally handles cases where only one type of feedback is accessible (i.e. all generations are failures), which can be particularly useful for challenging task with binary success/failure outcomes (e.g. code, math, safety assurance).
3 USING POSITIVE AND NEGATIVE FEEDBACK FOR POLICY OPTIMIZATION
In this section we present an approach to optimising policies based on preference data. We build upon large body of existing work in probabilistic inference for policy optimization. We will show that, when applied to preference optimization, the Anpectation-Maxiiizztioo (EM) approach results in a natural formulation of maximizing (weighted) likelihood of positive outcomes. Since such a formulation is appealing due to its simplicity but cannot effectively information about negative outcomes, we finally derive a simple extension that enables the use of dis preferree///eegttiee data- points.
The resulting algorithm has multiple intriguing properties: it can make use of preference data containing positive and negative outcomes but it does not require paired outcomes (i.e. it can make
2

---

Published as a conference paper at ICLR 2025
use of data for which we only know whether it is either good or bad, without knowing about relative preference with respect to other data- points) and can thus also naturally utilize unbalanced datasets (where e.g. we have multiple preferred options for each dis-preferred example, or vice-versa). Due to
the close relationship of our algorithm to the existing MPO algorithm (Abdolmaleki et al. (018)) we
refer it as preference based MPO (PMPO). The final update rule is presented in equation 10
3.1 BACKGROUND ON MAXIMISING FOR PREFERRED OUTCOMES
We review the preference based RL formulation common in RLHF Ziegler et al. 2019 Rafailov ct al. 2023) and show how methods from the literature on EM based policy optimization (Rawlik et al. 2013 Peters et al. 2010 Abdolmaleki et al. 2018) can be naturally applied to it.
In the following, x denotes the conditioning variable such as the state/observation in classical RL, or document and query in the LLM finetuning setting. Providing this information to a model (or policy) produces (yy (z) probability distribution over outputs y: these would be actions in classical RL or responses/generations (sequences of tokens) in the LLM finetuning literature. We will also make use of the definition of a KL divergence between conditional distributions which we define as (L(p(-1z) q(-1z)) = KL q; x) E Ey-p((1x) [log p(y|æ)) log (z)[z)).
Objective. Define a binary random variable S, which takes a value of in the event of a pre- ferred/successful outcome and 0 otherwise. To lighten notation, we will use the shorthand p(S) and p(S") to mean p(S = 1) and p(S = 0), respectively, and similarly for the conditioned distributions. In words, our goal is to optimize the parameters 0 of a parametric policy To (t)xx) produce outcomes that have a high likelihood of being preferred as measured by a likelihood function p(S1 y, xr), i.e. optimize the expected likelihood that y is 'preferred or 'successful" response to the condition
max E 0
(()))
Reference model. In addition to the above general formulation we assume access to reference model Tee that can either consist of a previous iteration of the model we would like to improve, or be the outcome of pre-training or supervised fine-tuning phase (as routinely employed in RLHF for LLMs). We refer to this model as the reference policy and in general we use the terms model and policy interchangeably.
Preference information. In order to derive a practical sample based algorithm we assume knowledge of the likelihood function p(S'y a:) to evaluate the samples y ~ Teee (-Ix). We distinguish two cases in this paper. In the first case, this can be achieved through an evaluation function f x) that assigns higher values to preferred responses y. We can then define likelihoods as p(Sly, x) exp( If (y. z)/n) (for preferred) and (S")y,,r) x) x cxp(-f(y- ///1/) (for dis- (isserfferrrdd r) with temperature parameters n and ny. The evaluation function f can be derived from available reward functions state-action value functions ? Alternatively, preference information can be obtained from a dataset of labeled examples:
D ==
(0) fot0, (1.)), s(())) i===1
N.M
where ((-5) are binary (zero or one) preference labels (preferred or dis- (is-preferred) usually obtained from human feedback for samples ylt..) ~ T- In this case we are assuming p(Sly', x²)) g(15) and p(S" y², ,x*) the likelihood of preferred or dis-preferred events for samples drawn from Tref- Note that defining the likelihood function is a design choice and depends on the information available.
g(t.)), Ultimately, our algorithm assumes access to information regarding
Policy optimization. Let us drop the superscripts (i) for now and only consider the objective on a per-condition basis, ultimately we average over the batch. Then for every conditioning x = (0)) the problem is finding a policy π T a)) that achieves the highest expected probability of preferred outcomes for a condition x. This amounts to optimizing
log ( (Sly,))
max logl E. ((S)y,))) yox
= E w~4
+ +KT(((g))) (=()),))))
J(:9.
Defined as x)) = E, rrr(y,, mr) o. yo = v] for a timeseries of observation/action pairs.
3

---

Published as a conference paper at ICLR 2025
where we have used standard formulation from the probabilistic inference literature (Kingma & Welling 2013) to decompose the objective into am evidence lower bound I (m:qq. q, r) and a KL term by introducing an auxiliary variational distribution qf The goal of EM is to iteratively find tight lower bound given the current estimate Tief by optimizing for (E-Step) and improve the lower bound J by optimizing for и (M-Step). More concretely, in the E-step, we fix q which minimizes the KL; this tightens the bound. In the M-step, we fix lower bound (TT; q,4) to update Ty. This process of tightening the bound and improving the policy constitutes one iteration of policy improvement over Tee--
Teee and find the and maximize the
E-step:
Tighten the lower bound by fixing Tref and minimize (L(g(=1a) || Pi (-15, x))-
Following prior work (Dayan & Hinton 1997, Peters et al. 2010 Abdolmaleki et al. 2018). since the KL is minimized when both distributions equal, the solution can be expressed in closed form as (0y(ær) = Pr (y)S,z). Then, according to Bayes rule:
1
Pronl (y)(,x)
Tht (e
Z
f T az (p(pp((Spp) x) dy. Recall that likelihood function p(Sly, x) is still modelling choice discussed in the Preference information section.
where we used the normalization factor 2g =
M-Step: Optimize the lower bound I fixing y = 7 from the previous step. Since this problem does not have analytic solution network, and maximize the following objective via gradient ascent:
parametric function approximator xe, usually large neural
log To (SS)))
I x) = E w-g
2. ( g) ]
E
log (o( + K
(4)
y-u
[(m;:)) = p(Sly. log To To(ulc)
(5)
Zu
where K represents all constant terms that are independent of 0 and are dropped from the final objective. Notice that this objective amounts to a weighted maximum likelihood with preferences determining the weights and samples coming from Tee. Notice also that the final expression subsumes the closed form E-step solution such that we can safely consider only this objective and introduce the short-hand J(Ta; z), dropping the implicit dependence on the E-step solution. In practice, to optimize this objective we need to form a Monte-Carlo approximation of the expectation in Eq. (5). We distinguish the two cases mentioned in the Preference information section.
In the first case, we assume access to a function f that is proportional to the preference log-probatt prgpoobbability, and access to M responses y(ب) for each æ. We can then set p(Sly, x)) w()) x exp(f(y())) xx)n)
Eq. (a softmax of ʃ across the responses وا6 to x)) This is the case commonly studied the literature, c.g., in MPO where one uses ())()),))
It is often unrealistic to assume access to a reliable model of preference labels. For example, preferences often come from human annotations and we thus only have access to samples or we might only have access to learned and unreliable preference model To cover this case, let us partition our dataset of labeled examples = D, DD, where D, = {x 5) e (8(5) = =1)}==an and D, = {g6ع) (s(i) = O)} denote accepted (preferred) samples and rejected (dis-preferred) samples, respectively. In this case we can still use the objective from Eq. (5). using the binary preferences أ0) as weights:
a(T;2))) In(
E. [300 log (o() y(
E DO)-D
log To (1)
which effectively filters rejected generations D, out, thus reverting back to the maximum likelihood objective on preferred data.
We use the identity log p(X) (() log 2(x.2) + KL(g(Z))p(2IX)) obtain the dccomposition. ((2)
A case studied in the offline RL. literature where the authors realised that using binary weights often works hetter as in binary CRR (Wang et al. 2020).
4

---

Published as a conference paper at ICLR 2025
3.2 USING DIS- PREFERRED OUTCOMES VIA REGULARISED MINIMUM LIKELIHOOD
We will now derive a simple way to incorporate negative (dis- (dis-preferred) samples into the optimization to address the shortcomings of naively applying the EM-based perspective from the previous section. We would like to incorporate these examples without changing the overall objective since it has well established policy improvement guarantees (Rawlik al. 2013 Abdolmalekii et al. 2018). To accomplish this we take a second look at the non- -on-paametticc variational distribution g from Eq. that is the solution to our E-step; since it determines the sampling distribution used for the M-step.
We can realise that the restriction to positive/preferred samples stems from the fact that we express q directly in terms of likelihood of preferred event p(Sly, z). A natural question then is: can we re-express in terms of dis-preferences? It turns out the answer to this is positive. Recall that S" denotes the complement of the event 5 i.e. the event that y is not a successful action/response to a conditioning z. Then by definition, p [, x) = 1 p(S ly, a) we can equivalently write
{uyl>) = Ze (y)())=) - (0((11p(S"))) y, x)).
We can plug this form of q into the evidence lower bound expressed in Eq. After rearranging terms and re- ve-writing in terms of two expectations over Ther this gives the alternative form:
(( [y. log To (v)z))
1
[TT;
=
KT. no; x) + K, Z'
(8)
go
Z
where K again denotes terms independent of To and we used the normalization factor Z'2 (rervvl=)(SS"\ ly, dy. This version of the objective now is expressed in terms of the likelihood dis-preference event. Additionally the state-dependent constant zł weights the KL term on per-state basis. This weighting implies that if the reference policy has more negative than positive examples for state x, the KL weight should be lower, permitting greater deviation from the reference policy. Conversely, if there are fewer negative examples, the KL weight should be higher, preserving positive examples. We simplify this by using state- inteeenndeennentt parameter /3 to subsume this weighting. We now write the equation [ based on dis-preferred examples:
I (To; x) RJ Jr x) =
E g(
[
[-
log To (x)) - 3 (T To;x). x))
where 3 should be tuned and set high enough to only remove the dis pilssrrffrred samples from the prior Tier and retain the preferred samples. As before, our use of samples أااو (labelled data) filters out part of the dataset; in this case, it is the accepted responses which are filtered out, hence the expectation over D,. We refer to the Appendix C for a full derivation. This is a fairly intuitive objective to optimize. It tells us to minimize the likelihood of dis- dis-preferred examples while staying close to the reference model. Interestingly, compared to the preferred data case, it has an additional KL term that appears result of the reparameterization of the variational distribution. We will see in the experiments that this term is required when learning from negative data. Intuitively, we can think of the objective as modifying the reference distribution such that the negative examples are removed. Interestingly such an additional KL for the M-step has previously been considered in the literature even for the case where we only learn from positive feedback (Abdolmaleki al. 2018). However, previous work used the additional KL term to prevent rapid entropy loss. In contrast, our motivation for incorporating the KL term is to learn from negative samples, as suggested by the derivations.
3.3 LEARNING FROM PREFERRED AND DIS-PREFERRED OUTCOMES
Finally, we can form a combined objective from our two M-step estimates - which both optimize the same quantity but can utilize different samples. That is, we combine Eq. 6 and Eq. (9):
Jar( = a
E [og [y())² yr
a) D.D
[og no (ylx)) 3 KL ( x))
Learning From Accoptcd/Postive Samples
Learning From Rejoesed/Negative Samples
(10)
"Note that, based on the derivations, 39 should approach zero as a approaches one. However, for cleaner
comparisons in the experiments, we keep the a and @ parameters independent. We will empirically show that, as
suggested by the derivations, KI. term is needed (,3 = 0) when learning only from positive examples (a 1)
while the KI. term indeed necessary for learning from negative examples.
5

---

Published as a conference paper at ICLR 2025
Rosembrock
Schwefel
Sphere
Repect)
(Accopt)
(Pa)cc))
-00
BRO
200
900
402
200
400
500
Shrpp
Figure 1: Performance of PMPO and DPO on Benchmark Functions This figure illustrates the optimization progress of PMPO variants (PMPO-AR, PMPO-A, PMPO-R) selection of standard benchmark functions, showcasing their ability to leverage different types of preference feedback.
where is trade-off parameter between the two estimates. Recall that in practice, this objective will be aggregated over an entire dataset of conditions and corresponding datasets Da and D,. There a few interesting things to note about this objective. First, we emphasize that our objective assumes categorization of samples into good/bad or weeferred/dis-preferee datasets. As result, it can be used even when only positive or only negative samples are available (this is in contrast to e.g. DPO [Rafailov et al. (023) or IPO (Azar et al. 2023) which require relative scores of paired positive and negative examples for each query æ). Furthermore, the objective has also no restriction on the number of positive / negative samples per query and thus it automatically extends to the multi-sample case for fine-tuning language models. Finally, the objective intuitive and simple to implement; it amounts simply maximizing likelihood of good data while minimizing likelihood of bad data and staying close to the reference model. PMPO introduces a and 3 hyperparameter<(se Appendix A for tuning guidance). Furthermore the KL term is implemented in closed form whenever possible. For example, for the autoregressive models used in LLMs, we use the sum of per-token closed-form KL divergences of categorical distributions. This enable to learn only from negative feedback without access to positive feedback as suggested by our derivations. See Appendix B for KL computation details and a discussion on the importance of closed-form KL computation.
EXTRACTING PREFERENCES FROM EVALUATION FUNCTIONS
Our algorithm requires access to preference information, which can come directly from human feedback or be extracted from an evaluation function. This section descrihes the latter. We consider improving policies within a traditional reinforcement learning (RL) setting; bandit optimization and optimization of language models via RLHF, In each setting our preference-basee be used in the policy improvement step. For this we need to extract preference information from (possibly learned) evaluation function. This can be achieved in the following way:
update rule can
Generate Samples: For given input or state x, sample one or multiple generations y from the current reference policy
Evaluate Actions: Calculate the evaluation function f(x y) (e.g. reward model in RLHF) for each input-generations pair (z. y)-
Classify Actions: If f(r y) 2 b(x), classify the generation y as preferred in state x. Otherwise (f(x.)) < b(x)), classify it as dis-preferred. b(x) is state dependent baseline that is typically used to calculate advantage values. Typically in RL average reward used as baseline.
EXPERIMENTS
We evaluate our algorithm a variety of different settings, showcasing its utility as a general preference optimization algorithm that can deal with many different forms of preference feedback. In this section, we aim at confirming our derivations in practice to learn from only negative feedback, only positive feedback, or both. We first test it in a Bandit setting (optimizing synthetic benchmark functions) then a setting where we transform RL on control and robotics tasks into preference optimization. And finally we showcase strong performance for RLHF of large language models. To verify our derivations, we evaluate three different variants of the PMPO algorithm: learning only from accepted samples (a 1), learning only from rejected samples (a 0), and learning from both accepted and rejected samples (a = 0.5). We also use MPO (Abdolmaleki et al., 2018) and DPO (Rafailov et al. 2023) baselines. For all the experiments, will beta value of 0.5 for learning

---

Published as a conference paper at ICLR 2025
PMPShicsapt Pajed)
CATEGORICAL
1000
Macipulatoo- Dring,
800
(00
092
BCD
Yoouel
Eawund
400
490
410
200
202
200
0.6
0.5
1.0
15
2.0
0.0
0.2
0.4
0.6
0.8
1.3
0.2
0.5
1.0
1.5
0.0
0.5
1.0
1.5
2.0
1300
Stand
1050
Humancid Vilk
1820
Manipulator Bringuball
GAUSSIAN
800
550
830
420
400
602
403
0.0
0.5
1.0
1.5
2.0
0.2
0.2
5-4
0.5
0.3
1.0
0.0
0.5
1.0
1.5
3.0
0.0
0.5
1.0
3.5
3.0
Statt
Stegs
Stage
Figure 2: Comparison of PMPO/DPO/MPO for high-dimensional control tasks from the DeepMind Control Suite. We plot average reward over time of training (using 100 episodes for each evaluation).
from accept&reject, 0.0 for learning from accept only, and 2.0 for learning from reject only, unless stated otherwise. Furthermore, in all experiments except experiment 5.3, the reference policy for all baselines is updated every N steps to allow for multiple policy improvement steps and demonstrate that our algorithm can effectively optimize the underlying reward function until convergence. For experiment 5.3, we only have access to samples from the reference policy; therefore, we can make only one improvement step, which means the reference policy is effectively fixed. Please note that experiment 5.3 is designed to have access to only positive or negative feedback for each state.
5.1 BANDIT RL: STANDARD FUNCTIONS
Our algorithm is first evaluated on synthetic benchmarks: Rosenbrock, Sphere, and Schwefel functions (Hansen et al., (200)) with optimum value of zero, framed as multi-armed bandits (Auer et al., 2002) (no state conditioning, z). Per iteration, the policy samples 4 examples, receiving function values. The top 2 samples are labeled as preferred, the others dis tis-ereferred. Figure shows PMPO's performance with different feedback (PMPO-AR: all 4 labeled samples, PMPO-A: accepted only, PMPO-R: rejected only). All variants optimize the functions, demonstrating effective use of diverse preference information. Remarkably, PMPO-R (negative samples only) optimizes with the KL constraint. DPO (best/worst samples) performs similarly to PMPO-AR.
5.2 FULL ONLINE RL: CONTROL SUITE
We evaluate algorithm on range of control tasks from the DeepMind Control Suite (Tunyasuvu nakool et al. 2020). See Appendix E for details. We cast the setting of optimizing a policy for the control suite preference optimization problem by leveraging learned action-value function (a eefunctionn-represeetteddd by separate network trained alongside the policy-to infer preferences for each observed state and action. This is analogous to the actor-critic setting in classical reinforcement learning. Similar to the bandit case, at each iteration, the reference policy proposes four actions for each state in the batch. The top two actions with the highest Q-values are considered preferred samples, while the two actions with the lowest Q-values are treated as dis- dis-preferred samples. We consider two different cases, one where the output of the neural network are mean and standard deviation of Gaussian control policy and one where the actions are discretized into bins (and the network outputs categorical logits over these bins).
Figure 2 demonstrates that, as in the bandit case, our algorithm can effectively learn from different types available signals (accept/reject, accept-only, reject-only) to solve high-dimensional tasks, such as controlling humanoid agents to run, stand, and walk, as well as manipulating objects. In all of them PMPO matches or outperforms the strong MPO baseline. Notably, even with only reject signals (PMPO-R), the algorithm is capable of achieving good performance. As predicted by the theory, not using a KL can quickly lead to collapse when using only dis pieferree samples. We also compare to implementation of DPO (Rafailov al. 2023) which uses the best and worst action sample
7

---

Published as a conference paper at ICLR 2025
Beta (B)
1.5
2.0
Acrobat Swingup
Cartpole Swingup
Chestah Run
Reacher Hard
1000
1000
1000
800
800
800
800
REJECT
Rewerd
600
600
600
Rewerd
600
200
200
200
200
0.6
0.0
0.4 Steps
1.0
0.2
0.8
0.5
Steps
Steps
Steps
1e6
Acrobot Swingup
Cartpole Swingup
Cheetah - Run
Reacher - Hard
1000
1000
1000
1000
ACCEPT
Reward
600
Reward
600
600
Rewerd
600
400
200
200
200
200
0.0
0.2
0.0
10
0.5
167
Steps
Steps
Staps
166
ACCEPT & REJECT
1000
Acrobot Swingup
1000
Cartpole Swingup
1000
Cheetah - Run
1000
Reacher - Hard
800
Reware
000
600
600
000
400
400
200
0.0
0.0
0.4
166 0.0
1.0
0.0
0.5
167
167
106
Figure 3: Impact of the KL weight 'beta' on the performance of PMPO. When learning solely from dispreferences across various Control Suite tasks (Reject, a 0), sufficiently high beta value required for effective learning. However, when learning from preferences only (Accept) PMPO is robustness to the KL weight 'beta' across different Control Suite tasks, confirming theoretical insights. When both both accept and reject signals are used (Accept Reject), PMPO shows a partial sensitivity to KI. Weight 'beta'. While learning is possible with wider range of beta values, beta higher than 0.5 is generally needed for optimal performance.
among the 4 samples. This still results in strong algorithm that works well when using a discretized action representation. However, in the continuous Gaussian case, DPO requires a very high implicit regularization parameter (3 20) which results in slow learning and suboptimal policies. For the sake of fair comparison with DPO that uses the worst and best generation, we also show results for PMPO when only the best labeled as preferred and the worst labeled as dispreferred, which is still competitive with DPO. Also see Appendix F for further comparisons.
We further ablate the impact of the KL term on learning solely from dispreferences (a 0), solely from preferences (a 1), and from both (a = 0.5). For each of these settings, we sweep over the B parameter the range (0.0, 0.5, 1.0, 1.5, 2.0). As depicted in Figure when learning exclusively from dispreferences (PMPO-R), the performance is highly sensitive to B. To achieve effective learning, we need to set 8 sufficiently high (> 1.0). which aligns with our theoretical derivations. In contrast, Figure 3 shows that the algorithm is insensitive to the setting of 3 when learning only from preferred samples (PMPO-A), again confirming our theoretical insights. When learning from both types of signals (PMPO-AR), shown in Figure we observe partial sensitivity the KL weight 33. While the algorithm can learn with a wider range of beta values, a B larger than 0.5 is still necessary to ensure optimal performance across all tasks.
5.3 OFFLINE RL USING ADVANTAGE FUNCTION
In a final set of experiment on control domains we want to show that our algorithm can also be applied to a setting where we have only access to one sample with either a reject or an accept label per state conditioning x. We consider the RGB Stacking benchmark (Lee et al. 2021). pick-and- place manipulation task with image observations (see Appendix E for details). We investigate the effect of positive and negative feedback in the context of offline RL exclude cascading effects from exploration. To this end we take a dataset of 140k episodes from a multi-tat -ask RL experiment trained to convergence (Lampe al. 2024). We then train value function all data and use it
8

---

Published as a conference paper at ICLR 2025
to label the transitions in the first 40k episodes as accept (positive advantage) or reject (negative advantage). Different combinations of acceptance, rejection, and BC losses are then compared in order to understand their respective effects. In summary we use: i) the full 140k episodes to train value function and label the first 40k episodes as accept or reject: ; iij the first 40k episodes to compute the positive weighted part of the loss if labeled accept and to compute the negatively weighted part of the loss if labeled as reject. The KL part of the loss is calculated on all 140k episodes. Note that the value function only used to transform the reward annotations into accept and reject labels.
the achieved reward for different loss combinations. First we run BC on the full 140k episodes and we can observe that the performance mediocre due to the data containing a significant amount of bad episodes. Using only the accepted transitions for BC training does not result in better performance; this due to the limited number of positive examples contained in the first 40k episodes. When combining both BC and using the positive (accept) part of the loss, performance does not significantly improve as the large number of negative episodes not compensated for. On the other hand, combining BC with the negative (reject) part of the loss does significantly improve performance. This is due to the rejection loss successfully pushing the policy away from the negative examples (while keeping close on all other data due to the KL constraint). Finally, best performance is achieved when combining all three losses; and thus effectively utilizing all data. While in this example we have constructed the dataset in a way that the effect is strong, and this might be less the case in more natural settings, it nevertheless shows that using a negative signal can have a significant effect on performance by masking the effect of bad data.
BC
Accept+BC
Accept 27
Reject+BC 77
Accept+Reject-BC
Reward
24
26
93
Table 1: Comparing different mixtures of acceptance, rejection and BC losses. We measure average reward (over 100 evaluation episodes) across stacking of all 5 triplets. Training with BC is corrupted by bad examples. Training on only accepted examples lacks data. Only when integrating the rejection loss bad data can be masked and performance goes up. Best performance is achieved when combining acceptance, rejection and BC loss signals.
5.4 LANGUAGE ALIGNMENT EXPERIMENTS
We apply different versions of the PMPO algorithm to the task of aligning large language models. Specifically, fine-tune Gemma 2B pre-trained model using trained reward model (Team] 2024b) using prompts from the LMSYS--hat- trained on human preference data with Bradley-Terry modelisation as explained in (Christiano et al. 2017). In these experiments, we perform one epoch of training, processing dataset of 500k prompts in approximatively 4000 learner steps, meaning that each batch is composed of 128 prompts and generations per prompt. Similar to the typical RLHF setting, at each iteration, for each prompt in batch, we sample four generations from the model and rank them based on their reward values. The top two generations are labeled as preferred, and the bottom two as dis- dis-preferred. For the sake of fair comparison with DPO that uses the top one (best) and bottom one (worst) generation, we also show results for PMPO when only the top is labeled preferred and the bottom one is labeled as dispreferred. Note that this particular choice could be refined further and tailored to the task. First, Fig. 4 showcases the best PMPO setting, leveraging both accept and reject signals (PMPO-AR) (and we compare to use either feedback signal in isolation). Notably, utilizing both types of feedback leads to faster learning compared to using either signal in isolation (PMPO-A or PMPO-R) and overall our approach is competitive to DPO, which is applicable in this setting by using only the best and worst sample respectively per prompt but would be more restrictive general (i.e. cannot naturally make use of unbalanced preference data). As shown on the right, when performing a side by side comparison using GPT-4 (OpenAl 2024) to judge whether our model is preferred over the base Gemma model (using a set of held-out test prompts) the PMPO fine-tuned model wins over the base model. Note in Fig. right, we see some drop indicating some exploitation of the imperfect reward model; known as reward hacking (Skalse et al., 2022). We can see that PMPO-AR is the quickest to "hack the reward" (see Fig. Ieft), it reaches a good performance but then in the middle of training its start hacking the reward and learns pathological behaviour that makes it performs worse the independent benchmark. This phenomenon has been observed consistently in RLHF Overall, our language alignment experiments provide strong evidence for the effectiveness and versatility of
dataset (Zheng et al. 2023). The reward model has been

---

Published as a conference paper at ICLR 2025
PMPO. Finally, illustrate in Figure that, again, algorithm demonstrates the ability to learn effectively from various preference signals, including scenarios with only accept (PMPO-A), only reject (PMPO-R), or both accept/reject (PMPO-AR) feedback. These results highlight the versatillity of our approach to different preference acquisition settings. The results also underline the critical role of the KI. term in enabling learning exclusively from dis-preferred generations (PMPO-R). As predicted by our derivation, a sufficiently high value 3 > (1 - a) is necessary to stabilize learning in this scenario. In contrast, when learning solely from preferred samples (PMPO-A), the algorithm insensitive to the value of /8 in terms of stability.
CRO Acreptent& Rececitbnn)
PV-o locoad (0bat-d)
sate bo oyroo beacc kute far Virrt pocest
7.0
1090 1390
2013
2000
2000 2500 420
Figure 4: Left: Impact of Combining Accept and Reject Signals- - The plot demonstrates the learning progress of PMPO-AR (using both accept and reject signals) compared to PMPO-A and PMPO-R, showcasing faster learning when leveraging both types of feedback in language alignment task and is competitive with DPO. Right: Win-rate when doing A/B comparisons on held-out prompts for PMPO against the base Gemma checkpoint judged by GPT-4
Beta
0.5
Accept
Reject
Accept & Reject
Mean Reward
Mean Reward
Steps
Steps
1e3
Steps
Figure 5: Rewards obtained by the policy each training step, averaged over the batch and smoothed.
Each curve corresponds to a configuration of 3 specified in the legend. This figure illustrates the ability of PMPO to learn effectively from various preference signals (accept-only, reject-only, or both) in language alignment tasks. highlighting its adaptability to different preference acquisition settings.
6 CONCLUSION
We propose novel algorithm for policy optimization from preference feedback derived from the perspective of RL as probabilistic inference. Our policy improvement algorithm has a clear and intuitive objective: it maximizes the likelihood of preferred data while minimizing the likelihood of dis dis-preferred data. We show that doing the latter in a stable way requires a regularization term foreing the policy to stay close to a reference model. This regularization term follows naturally from the derivation. The main advantage of our algorithm over existing preference optimization algorithms such as DPO is that it does not rely on defining/fitting an explicit model of the preferences and can thus use data containing partial preference information; i.e. we can use data where instead of comparisons between samples we only have accept (or only reject) labels and make no further assumptions their distribution. One limitation of our approach is that to effectively learn from negative feedback, we need a good estimate of the KL term, ideally in closed form; otherwise, enough samples from the reference model are needed.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
We are grateful to the Gemma team for providing the models and infrastructure that enabled our language alignment experiments. We also thank Thomas Hubert and Markus Wulfmeier for their valuable feedback, and John Agapiou for identifying a subtle inconsistency in our derivations and providing suggestions for its resolution.
REFERENCES
Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation. In International Conference on Learning Representations, 2018.
Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47:235-256, 2002.
Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. A general theoretical paradigm understand learning from human preferences. ArXiv, abs/2310. 12036, 2023.
Edwin V. Bonilla, Bonilla,Shengbo Guo, and Scott Sanner. Gaussian process preference elicitation. In Neural Information Processing Systemx, 2010.
Craig Boutilier. A pomdp formulation of preference elicitation problems. Proceedings of the National Conference on Artificial Intelligence, 05 2002.
Paul Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. arXiv preprint 17Xiv:1706.03741, 2017.
Peter Dayan and Geoffrey E Hinton. Using expeccation-maxiiizatioo for reinforcement learning. Neural Computation, (12:2271-278 1997.
Shengbo Guo and Scott Sanner. Real-time multiattribute bayesian preference elicitation with pairwise comparison queries. In Yee Whye Teh and Mike Titterington (eds.), Proceedings the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 289-296, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. mlr wrwpspessssse..ee. v guollb. /ttp////////////// html
PMLR. URL http::::/ppcceeeeeeeee
/
Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (ema-es). Evolutionary computation, 11(1):1-15 2003.
Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient and exact optimization of language model alignment. arXiv preprint 1.Xi::2402.008666 2024.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rX2v:1312.611 14, 2013.
Thomas Lampe, Abbas Abdolmaleki, Sarah Bechtle, Sandy H Huang, Jost Tobias Springenberg.
Michael Bloesch, Oliver Groth, Roland Hafner, Tim Hertweck, Michael Neunert, et al. Mastering stacking of diverse shapes with large-scale iterative reinforcement learning on real robots. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 7772-7779. IEEE, 2024.
Alex X. Lee, Coline Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost Tobias Springenberg. Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, Claudio Fantacci, José Enrique Chen, Akhil Raju, Rae Jeong, Michael Neunert, Antoine Laurens, Stefano Saliceti, Federico Casarini, Martin A. Riedmiller, Raia Hadsell, and Francesco Nori. Beyond pick-and-place: Tackling robotic stacking of diverse shapes. CoRR, abs/21 10.06192, 2021. URL sattps://axxlv.orgg
Sergey Levine. Reinforcementt learning and control as probabilistic inference: Tutorial and review, 2018. URRlhhttss://axii

---

Published as a conference paper at ICLR 2025
OpenAl. Gpt-4 technical report, 2024. /tt.///pps///aaxxt..o///////////
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL
https://arxix> wtg.:bogg/aoooooooooo.oooooo.r.o htrgg/bb/2203..o 02155
Jan Peters, Katharina Mulling, and Yasemin Altun. Relative entropy policy search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 24, pp. 1607-1612, 2010.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https?
wttpiiivrrg//b//// httg///bs2335.... 18290
Konrad Rawlik, Mare Toussaint, and Sethu Vijayakumar. On stochastic optimal control and rein- forcement learning by approximate inference. In R:SS 2021, 2013.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming, Advances in Neural Information Processing Systems, 35:9460-94777 2022.
Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Offline cctor- reinforcement learning scales to large models, 2024. URL https:tt/////..... http::s///axxiv.org.. abs /ta.b///4445 2202 05546
Gemini Team. Gemini: A family of highly capable multimodal models, 2024a. URL https
7/arxiv.rrg/abs//2 2180.
Gemma Team. Gemma 2: Improving open language models at a practical size, 2024b. URL
https://arxix: org/ abs /200 00118
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033. IEFE, 2012. ISBN 978- 1-6673-77-7 URL http:
//dblp.uni-trie de con wtt.://////o///////20oo.oo./ #TooorovETT
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https://doi.
ng/100.1066/jsimpaa220000 2020 http://wwww.......aaao/////..eea wtt.pp/eww..cceeeeeeee.tt.. com/ crom/science/
3rticlee/it/526655552200000
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, and Nando de Freitas. Critic regularized regression. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7768-7778. Curran Asso- ciates, Inc., 2020. URL wttp://eeee/pppeeeesse...eeeeee. 3 32D01111605500000ee1177776660003
files/paper/
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric. P Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-Ine A large-scale real-worke Ilm conversation dataset, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ar Xiv: 1909.089990993333 2019.
12

---

Published as a conference paper at ICLR 2025
A HYPER PARAMETER TUNING
PMPO introduces two hyperparameters, a and 3. However, we found them relatively easy to tune in practice based on the following guidelines:
a: This parameter reflects the relative importance of positive and negative feedback. When both are available, = 0.5 is a reasonable starting point if both types of feedback are equally reliable. Otherwise, can be adjusted to reflect the confidence in each feedback type. In cases with only one type of feedback, is naturally determined by the data.
B: This parameter controls the influence of the prior or reference policy. Our experiments suggest that higher is generally beneficial when learning from negative feedback such that the more is contribution of negative feedback to the policy update (lower a)) the more 3 should be. This can also be motivate from the derivations that KI. term appear as the result of learning from negative feedback. We also find that as long as 3 is large enough, the algorithm fairly insensitive to the exact choice the parameter. Please also see section 3.2 in the main paper for more insights regarding derivation of parameter B.
B COMPUTING THE KULLBACK- LEIBLEER DIVERGENCE
learning from negative feedback is only feasible with a correct KL term and access to reference policy Tref- When logits are available using an exact KI. computation is ideal. In the absence of logits (Experiment 5.3), relied abundance of unlabelled data estimate it. This enables us to learn only from negative feedback without access to positive feedback as suggested by our derivations.
Subsequently, we present the derivations for computing KL divergence for LLMs with autoregressive policies in our experiments.
Let's s consider a single two-token generation (y1, U2) sampled autoregressively from a prompt x. We can factorize the joint distribution associated with such a generation as follows:
x (y/, y2|2) = x (y1l=)) y/)
(11)
Using this factorization for both To and Nea in our KL regularizer, we compute the following:
KL (T = Erg log
Tell
(12)
E E,g log
n. ref ree
(13)
2 Tg g
= E E
log ref
+ EE EE 22
log
Re
(14)
log Te πT
E, log
EE
EE
(15)
where the first term can drop the expectation with respect to Egg since the integrand log Tye does not "5
depend on y/2 The first term can be computed analytically because T and TÿJ are simply categorical distributions over the entire vocabulary of tokens. The second term, however, is problematic due to the outer expectation Exi, which requires us to integrate a KL over all possible values of y/-
While we can easily compute this KL for any one value of y² the integration would be unwieldy and certainly becomes intractable as we extend this to longer sequences ya, y4. etc. Luckily, during training we obtain samples D1, 772 Tref and so ÿ1 T which allows us to use single-sample Monte Carlo unbiased estimate of the second term such that:
2 T Jr, v1)
2d E
Je log
+ E
log
(16)
T
xr, 91
13

---

Published as a conference paper at ICLR 2025
Extending this to sequences of length L, and dropping the superscripts on the policies they autoregressive, we obtain the following approximation:
Teer( Iz, D1:i)
(T(TTlll) R Er log
E E log
(17)
(-(2))
TTo(112; (1:4)
xm1
where VILL Tref-
C FULL DERIVATIONS OF PMPO UPDATE RULES
Consider doccssonnmmkinng scenario characterized by the following elements:
Stationary State Distribution: A stationary distribution (ax) describes the probability of encounter ing different states or contexts z. We typically have access to samples from this distribution.
Policy: A policy π [y [)) dictates the probability of selecting an action/outcome given state æ. We also refer the current estimate of the policy as the reference policy, denoted by Fref (ref(y/z).
Preference Information: We have access to knowledge for driving the likelihood of preference or dis- preferefenece toward actions taken by policy x (z)(2)) This information takes one of these forms:
Preference Probabilities: q / = 1 ly, x). where / is binary indicator with repre- senting preference. When only reward Q-function is available, this probability can be derived as q( 11y, exp(Q(v.=))
2.
Dis-preference Probabilities: q(I Oly, æ). where / denotes dis- pits-prrfeeence When only reward or Q-function is available, this probability can be derived q(l 0ly, x) exp( ((vx))
2
where Z, is normalization constant Z, = (xwpp(ypppppcccpp + eexpp= x)) to ensure q(1 0 y, x) q(I y, 2) 1. Note that q(I 11y,x) and g 0ly, x)) are likelihood functions and can be defined depending on the problem at hand.
Objective: Derive optimal policy to give the highest probability to the preferred outcomes:
max lo = 1) = log ((xx) (o(v))))(I / = 1ly, x)) dy dx
C.I MAXIMIZING FOR PREFERRED OUTCOMES
We start with the stated objective:
max Po = 1) = log [(x)) / (olulz))(( ( 1/y, x) dy dæ
This objective can be solved by the EM algorithm to iteratively create a tight lower bound on the current estimate of the objective log Pres (T= 1), and then optimize the objective. By repeating these two steps, is guaranteed that the algorithm will converge. We will show one iteration EM improve the current estimate Tror. In order do so, we use the following identity:
(()) log p(X,2) dz KT(g(Z))p(21X))
log logp(X) x)
g(Z)
which gives us the following equivalent right-hand side:
log (1 =1)= 1) p(x)) f ((()))) log Po = 1, v)
dydx+ / p(x) KLQGul<))bblll = =1000=1,,0000 11
((v(2))
14

---

Published as a conference paper at ICLR 2025
C.1.1 E-STEP
In the E-step, our goal is choose the variational distribution ((y(2) such that the lower bound
logPret (T
tight possible, which is the when the KL term in the equation above is zero at the current estimate of the policy Tzes. This simply leads to g x)) Pres (v) = 1, 2.), or according to Bayes'
rule, we get:
((yl>) = Tree(y/a)p(1 = 1/,,, 1)
Pref 1
where Prer (1 - 1/x)
Trei (w/=)p(I 1|x, v)dy is normalizer for given state
C.1.2 M-STEP
In the E-step, we found non-parametric variational distributions (v(æ)) for p(x) that give higher probability to preferred actions sampled from Teer- The E-step can be seen as sample-based policy improvement of Tza5 with respect preferences. In the M-step, optimize the lower bound to obtain a new distribution, i.e.,
max / (((x) / ((vll)) log <(ylx)
pa(l = 1, y/ dydr
where
<1)xx) = T (yla))p 1| y)
Tzes (y(=)p(1 11 x)
according to the derivations in the E-step. After rearranging the terms and removing the terms that do not depend on get
max / p(z) /
((ylz) log no (yly))))ddl dyddx
which effectively is weighted maximum likelihood objective fit improved non-parametric policies.
C.2 INCORPORATING DIS-PREFERRED OUTCOMES FOR POLICY OPTIMIZATION
In the previous section, showed how optimizing for preferred outcomes lead to useful policy update rules. However, we only incorporated preferred outcomes q( = 1) to update the policy. Now ask this question: how incorporate dis- dis-preferred outcomes directly optimize the policy? In order to do that, we first write down the policy optimization objective we derived in terms of preferences:
max / ((x) / ((v12)) log no (y(l))) dyddx
where
g(ylz) = Teet (y ())pp((I= I = 1|x, y)
J or ( (2/z)p(1 1/y. x)
We also know that p(/ 1|,, 1 p(1 0|2, This is correct as p(]]2: y) is probability function over a binary random variable I. Therefore, the variational distribution can be rewritten in terms of dis-preferences, i.e.,
15

---

Published as a conference paper at ICLR 2025
Trøt ( (v)x:) - p(T 0/2,8))
((vaa)
Tee (y]zz) (y==((]z))( p(( = 0/sr,
Substituting this into the maximum likelihood term above, get:
max / (u(a) / 1 (x) Tree (1/ ( log (e(ax) dy do
f p(xx) /
T Trer(ylz)p(1 = 0/z, 1) log no (y)z)) () dy dx
1 ('(x)
where ('(x) = T (uee(yl>)p(( = 01 dy-
After rearranging the terms, get the equivalent form:
/ Teec(ylz)) log no (y(x)) dy dx-
max 0
((x)) 1 ('xx)
f ((xx) (xx
(z)
(xx)) Trar (v(=)p( 0/ær, y) log no (u(xr) dy dx
which will simplify
max / ((xz) /
Z'(x) T (vlz) log no dy dx plxx) f
Z'(x) Tha Tro (y|x))(1 = O|z, v) log (y)a)) dy dx
first term can be written as KL term, i.e.
1 [u(z)) Z'(x) KL(Trer (
m(x) /
1 Z'(z) Ter (v)(()))) = 0/z, 3) log Te (e(2)>) dy dr
max- -
Note that (a) is state-dependent constant that weights the KL term state-by-state basis. This constant suggests that for state x, when the reference policy contains more negative examples compared to positive examples, then the KL term weight should be lower so more samples from the reference policy can be removed. Otherwise, when there are not as many negative examples to remove, then the KI. term should have a high weight positive examples remains. For simplicity, we subsume this weight into a parameter (3 that is inteeeeeddeennnt. Now the final update rule reads::
max- / u(xx) /
/
(yllx) log To(y dy dx 3 ((xx) (T)o (m))) dr
where f(ylz) Fee (ubx))p (1= and is tuning parameter. This update rule minimizes the probability of the dis-preferred distribution while staying close to the reference policy. Note that the KI. term and its direction emerge directly from the derivations.
C.3 FINAL UPDATE RULE: LEVERAGING BOTH PREFERENCES AND IRSPERRFESSS
After putting things together, we get the following update rule that maximizes the preferred outcomes and minimizes the dis-preferred outcomes, i.e.,
max a / m((o)/
I a(vlz) log To(u)o) dy dr-
a1-a)l/ a)l M(O) f (ygl)) log #o(y)x) dy dr - 3 /
dx
16

---

Published as a conference paper at ICLR 2025
where is distribution modified with respect to preference probabilities resulting from Rref weighted by preference probabilities. i..,
Tree Tree (ulz))(( = 1/2,1/ v)
(lylx)
J Teet (1)))((1 1 dy
and é is distribution modified with respect to dis-preference probabilities resulting from Tref weighted by dis dis-preference probabilities, i.e.,
t(yla) Tret Tree (1/x)p(I = 0/2, v)
J [ (y)z)p(1 0|z, y) dy
Now we can choose likelihood functions q(1 1,y, x) and q(L Oly, x). Note that can choose different likelihood functions depending the problem and available information; for example, the likelihood function can depend on advantage values.
Intuitively, this objective learns from the preferred distribution ((v(2) and gets away from the dis- preferred distribution (yylæ) while staying close the reference policy (which enables learning from dis dis-preferred distributions).
D FUNDAMENTAL RESULTS
This section explains precisely why the EM method is sound approach optimization. We will present the case discrete function. This section relies on classical results discrete optimization and shows how For simplicity and clarity, we will optimize function f E IR' mapping elements of S to real numbers, where S is finite set. More precisely, our goal is to find distribution ō As (parameterized policy in RI. that maximises the expectation of under (expected value function in RL):
build simple strictly improving algorithm that maximises discrete function.
E Eg [s()] = =E Ess)s((s)
.ES
We recall that a discrete probability distribution E As can be identified as positive real function 5 E R verifying:
Σ 8(s) 1.
SES
To find a good distribution maximise Eg~ [f()), the algorithm relies on the following results:
Starting from distribution 7 E As, there exists unique closed-form argmaximum 8 of the regularised expectation E.-o [ (8) - KKL || n)) where T E i&; is a strictly positive n! jxx((- real number. We have 8* = Eves M(8" exp))
Unless 8 = 77, we have strict improvement between 8* and mp
Eg-5- [f(s)] [f))]
We prove these results in the main paper. Those results implies that the following algorithm that starts at do and computes:
cxp())(1
wk E N. 6x Eves du(s') exp(r-1f(s\))
strictly monotonically improving until there is K E N such that 8x+1 8x: Ey-ho [s)) < E. [f() <... < Ex-sx [f)) = E [f()]
In practice, we have a set of learnable weights 0 to parameterize a distribution qo in order to fit 8 and another set of fixed weights y parameterize distribution qa òg. Then, fit 8 the idea
17

---

Published as a conference paper at ICLR 2025
is to minimize the following KL divergence (this is the maximisation step):
qu(+) (xp
[()) KL
(0)));
E EE (u(s') exp(T=1f((s)) f(s'))
qu (-)))exx exp(r ((-)
Eyes Ua(a') exp(( [(,']]
E
(x()))xp(>-1 ((-1)
log
Ee' 9pte"
90
gu(:) exp(r- f(.))
=-
9x( 150)
[log (ya) + E
() oxp(r (xppp--11 176)
log
EveS (8( exp(r ((())
Es'es capty 1500
Eves anto anttm) caple 1z(0())
The term exp(s=1 F61
log Evtes exx)xx
qu(+) exp(1 s(-)) exp(t 115)))
) does not depend on so is
(o( irrelevant in the minimization. Using the re-weighting formula Bana [[()] Es- L [s)], the minimization problem is equivalent to:
((0) -E
9p (c) (xx())))x))))))) ((-))
[log (q+)),
Eores (xpe (1p() 110'))
exp(T- f(-))
Es-an
log (go)
Evesqu('' (xx')e(((x ((p(())1((()))
As Eves (u(s') exp((=1f((')) is a constant, this is equivalent to minimizing:
((0) = - -gn [cxp(r ((s) log (s)(
D.1 EXISTENCE AND UNIQUENESS OF THE REGULARIZED ARGMAXIMUM
For completeness, briefly recall the proof of existence and uniqueness of the argmaximum of the following regularized criterion that can also be found in the work of Ralailov al. (2023):
(+) = Egno [f(s)] rKL(O II n),
Σ ((s)f(s) 7KL(6 n))
Now, if we define the softmax probability 8* E As as:
(Ts) exp(1 f(s))
(f(s)
Vs E (s)
then, under the previous definitions, we have the following results:
8* arg max C. (8)
SEAs
18

---

Published as a conference paper at ICLR 2025
Proof.
(6)
8(8) f(8)
KL m),
SES
Σ 8(5) f(s) Σ 8(5) log
8(s)
7)(8)
f(0)
log 0(s)
Σ (())(
((s)) )
7
#ES
8())( log ()pp(T-)((()) - log 8(s)
Σ
(7)))
#ES
(88) exp(x ((()))
Σ
3(s) log
8(s)
EES
is nlo' exp(1 (((011)
Σ (lo
EES expl( 1(01))
8(8)
#ES
e(s))xp((-
Σ 8(8)
S'cc (expe)xp(xx""""" exp(T= ((0)))
+ Σ
8(8)
log Σ
n(s') exp(r-
5(s)
SES
sES
dES
Σ 6(s)( log 80(() + log
Σ 7is's exp(r-
5(s)
#ES
E'ES
= KL 8*) + log
(ss) exp(p f())))
SES
By definition of the KL, we now that 8 arg max
KL || 8*) and as:
KL (1)) (6) log Σ ((s) exp(1 exp((-(((s)) f(s)))
(18)
SES
where log (EeEs r(s) exp(p- f(s)) is a constant (does not depend on d) and r a positive mul- tiplicative term, then K6 || 8*) and (5) share the same argmaximum. This concludes the proof.
The fact that have:
n(.) exp((=-((()) arg max L (5), dEAs
8" Eves (7'') exp(r-1s(s))
implies by simply replacing ö by 8* equation (18 that:
KK || 118*) (,))))
log Σ 7(s) exp(r- f(s)))
As KL(6" (") = 0, we have:
C+(8)) = max C,) = 7 log
[
(19)
BEAs
#ES
This result is often expressed in term of an incquality that says that the logsumexp is majorant of the regularized expectation:
YS E As, T log Σ (ss) cxp(T-1
2 Σ 8(s)f(s)
(L
(20)
19

---

Published as a conference paper at ICLR 2025
D.2 PROOF OF IMPROVEMENT.
We use the same notations as in the previous section, and our goal is to show that using the distribution 8" instead of strictly increases the expected value of our function f:
Eg [f()) (s)n)
unless 8 77. This means that we can confidently replace n by d for the next iteration of the algorithm.
Proof.
From Eq. 19 we have:
T log Σ n(s) exp(( = (6*) Es-5- [f(s]] -KL(0" || m)
SES
Using Jensen inequality we have:
T log
Σ exp(r- 2 Σ
(1s)
log (exp(T-) f())))
sES
#ES
Σ n(s)T
SES
Σ
(sssf(s)) = Eg [f())
SES
This implies that:
E [f-)][((s)] - KK I1 2 Exngl (s)]
As rKL(8") || 7) is strictly positive unless 8" 17, we conclude that we have strict improvement of algorithm unless 8 which means in this case that the method has converged.
D.3 LINK BETWEEN IPO AND PMPO
In this section, draw parallel between the IPO and the PMPO losses. For dataset of triplets (x', yh ) the IPO loss is:
N
TO To (axt) Tres (ic
To (ut 2xt)
Liro(() = N
Σ
log (To(vál=*)) + log (TO(V(l=')) + (log
log
Tree lxot)
The IPO loss is composed of two terms the policy optimisation term:
Pieo (0) 1=1
log (mo(vál=')) log (To(vil='))
and the policy regularisation term:
ne (yv, (olllla'))
Te T (z)) Trer (x+
27
RIpo (0) = Σ
N log
log
Trer (vá (vál
When a = 2. the PMPO loss can be written as:
1 N
1 log (To(vál=") + log (To(uil=*)) B.I (l (o
(0)
2
The PMPO loss is also composed of two terms the policy optimisation term:
N Σ
1
Pewso (0)
log (To(Vi(=')) + log (To(v)(z"))
N 1=1 2
20

---

Published as a conference paper at ICLR 2025
and the policy regularisation term:
1 N
REMPO (0)
Therefore, the policy optimisation terms of the IPO (tpo(0) and PMPO Pxre (0) losses are identical at a constant factor. Now we are going to create connection between the IPO and PMPO regularisation terms when ya and Vr are sampled from Zzer Fze=(((z*) The first thing to remark is that (log [ TTuu (Bt (att)
log mo(udla))
is an unbiased estimate of
(ylle (log mo(Miar)) ) log
mo (Y'((a)))
Then, we remind the reader that the
(x')')
variance random variable X under distribution μ verifies:
VARX~M
2 Ex x 1 x)),,
where X
and
X
independent variables with distribution μ. This
[log ==(
means
that
Tent (MC (xr)
log
[one (21)
is an unbiased estimate of
W log
Therefore the expectation (over :=ot) of the IPO regularization term is:
2 N N
log (2(())" Troe (Yro=(Y(rr)) (Y(z))
E Ex (0)
Σ
VARy PaRyyranti (
As VARX~1 VARX~ul- --X, we also have:
Ex (0)
2
Σ
N
log
Tzec (Yrec(((xx) To (px)))
N
This is in contrast with the regularization term of PMPO:
1 N
N [[ =
N
Teer (Y (xx) To (xx))
REMDO (0)
Ey mom Byntaeel
log
N
So the difference between PMPO and IPO is that PMPO will minimize the expectation of the log ratio between the reference policy and the online policy whereas IPO will minimize the variance of the same quantity.
E BENCHMARKS
E. CONTROL SUITE
The DeepMind Control Suite (Tunyasuvunakkoll et al.) 2020) is collection of benchmark tasks implemented in the MuloCo simulator (Todorov et al. 2012). The suite includes variety of embodiments different complexity and action dimensionality. For each of these embodiments, there are multiple tasks implemented, each of which defines single reward function. Example images for some of the domains are shown in Figure
E.2 RGB STACKING
The RGB Stacking benchmark (Lee et al. 1021 is robotics task involving a Rethink Sawyer robot arm outfitted with a Robotiq 2F-85 gripper, as well as a basket containing a number parameterized geometric shapes in red, green and blue color. See Figure (7 for an illustration. The goal of this task is to have the robot arrange the shapes into varying arrangements, such as stacking one on top of another building tower. The policy only provides proprioception information and the
21

---

Published as a conference paper at ICLR 2025
Figure 6: Example domains in the Control Suite. From left to right: Cartpole, Acrobot, Reacher, Manipulator, Cheetah, Humanoid
images from three cameras surrounding the basket; there no explicit tracking of the objects' relative positions, their identities. Specifically, the agent is provided the observations listed in Table 2 Thus the task's 's challenge lies in forcing the agent to learn a control policy directly from vision, and recognizing which objects different manipulation strategies.
the workspace, since their different geometric properties demand
Figure 7: Illustration of the simulated RGB Stacking domain. Top right corner: goal image (left) and current observation (right) for the front left camera, both as provided to the agent. Bottom: reward trace for the dense stack-n objective.
The task is implemented in the MuJoCo physics simulator (Todorov et al. 2012). Using the ground truth positions of the objects' positions (which is not provided to the agent). several reward terms are calculated, including stacking two objects, stacking three objects, and building a pyramid, for each object permutation. Details of these rewards can be found in Springenberg et al. (2024).
ADDITIONAL EXPERIMENTS
In this section we evaluate the performance of PMPO and DPO in figh-dimensional control tasks from the DeepMind Control Suite (Figure [. For each state, sample four responses from reference policy. We compare PMPO and DPO under two conditions:
1. Two Samples: Both algorithms utilize only two samples for learning; the best and worst responses (Accept[Best] Reject[Worst]).
2. Four Samples: Both algorithms utilize all four samples (Accept&Reject), For PMPO, the top responses used preferred generations and the bottom dispreferred
22

---

Published as a conference paper at ICLR 2025
Modality
Dimensions
Arm joint angles
7 7
Arm joint velocities
7 7 3
Arm joint torque
7
Gripper motor angle
x 3
Gripper motor velocity
1 x
Gripper grasp sensor
1 x 3
Cartesian tool center point pose
x
Cartesian wrist endpoint velocity
6 3
Cartesian wrist endpoint force
3
Cartesian wrist endpoint torque
3 3 3
Back left basket camera
80
Front left basket camera Front right basket camera
80 80
80
Table 2: Observations given the agent in the RGB Stacking benchmark. Note that for all ob- servations except the camera images, history of time steps is provided, resulting in the x3 dimensionalities.
SMFOjacenteeejj RejertWentt)
PPOAacepsA@eee ectl
pPo'annenteertie bestll RejentWirmti
CATEGORICAL
1000
Humanoid
1000
Humanold Stand
1000
Humanold Waik
3000
Manipulator Bring ball
992
000
090
600
420
402
420
0.0
1.0
15
0.4
0.
0.2
0.3
1.2
1.0
Stops
1n7
Steps
Steps
1x7
Stops
Humanoid Run
Humanoid Stand
Humanoid Valk
Manipulater Bring ball
1000
1090
1900
1030
GAUSSIAN
baxed
600
Rower
600
600
6:0
200
220
Sl6p4
Steps
307
2heps
307
20pp 107
Figure 8: Comparison of PMPO and DPO for high- digh-dimensional control tasks from the DeepMind Control Suite. We plot average reward over time of training (using 100 episodes for each evaluation). For each state sample 4 responses from reference policy. We compare PMPO and DPO when both use only 2 samples for learning; the best and worst responses only (Accep((esst]& Reject(Worst]]]]]] We also compare PMPO and DPO when both use all 4 samples. For PMPO these would be top two as preferred generations and bottom two as dispreferred generations (Acee &Reectt(eeeWooo For DPO create two sets of pairs, i.e., (best and worst) and (second best and second worst). Results show in general using two samples for learning and all 4 samples in learning yielded similar performance.
generations. For DPO, create two sets of pairs for each prompt: (best and worst) and (second best and second worst).
Figure g indicates that using two samples for learning yields similar performance to using all four samples. Moreover, PMPO remains competitive with DPO when using pairs, while generally performing better when the policy is represented as Gaussian.
23


Reviews:
[{"review_id": "U2jAUVuZF5", "reviewer": "Reviewer_DbEb", "summary": "The paper proposes a preference learning objective (PMPO) that can utilize not just preference pairs but any combination of positive only or negative only samples. The objective is derived by defining an EM formulation for the expected success maximization objective of Eq 1 and defining the M step for both preferred and dispreferred samples. Experiments show that PMPO can operate with preference pairs as well as only preferred or only dispreferred samples.", "questions": "1. Can the authors clarify if PMPO is indeed implemented as the expression in #5 above? If not, could the authors clarify the differences? If so, can the authors clarify its significance?\n2. Can the authors clarify in what setting one would choose to use PMPO over other common baselines and why?\n3. Why doe the authors focus on DPO as the baseline? Why not consider other common methods in RL benchmarks? \n\nOther smaller comments:\n1. In the first paragraph of experiments, shouldn't $\\beta$ be $alpha$?\n2. What is the value of $\\beta$ used in Figure 2?\n3. Why is the MPO baseline in Figure 2 denoted with a dashed horizontal line?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed PMPO and its derivation is, to my knowledge, novel. The objective is also easy to understand and implement, and the derivation has a clear probabilistic grounding in expectation maximization.\n2. The paper is clearly written and tackles a relevant topic to the ICLR community.", "weaknesses": "While the method derivation is clear and well-motivated, the primary weakness in the work lies in the experiments.\n1. The proposed method does not outperform DPO, the main baseline being compared to.\n2. The experiments on bandit RL tasks focus on DPO as a baseline, without considering other methods used in these benchmarks.\n3. The DPO baseline does not seem to use all the data given to PMPO; for instance, the end of Section 5.1 states that DPO uses \"the best and worst action samples among the 4 sample archives\", rather than all samples.\n4. While PMPO can be adapted to a wider array of settings than DPO, for any given setting, it is not clear when and why one would choose to use PMPO over another method for that setting, e.g. PMPO on preferred only vs. SFT on preferred.\n5. Assuming a sequence-level forward KL term, the proposed objective seems to simply amount to a weighted average of positive log prob terms for large enough $\\beta$: namely, if $\\mathcal{J} = \\frac{1}{n} [\\alpha \\sum_{y \\in D_a} \\log \\pi_\\theta(y|x) - (1 - \\alpha) \\sum_{y \\in D_r} \\log \\pi_\\theta(y|x) - \\beta \\sum_{y \\in D_a \\cup D_r} \\log \\pi_{ref}(y|x) + \\beta \\sum_{y \\in D_a \\cup D_r} \\log \\pi_\\theta(y|x)$,\n\nthen for $\\beta > 1 - \\alpha$, the objective is a sum of positive log probs only. Indeed, Figure 3 suggests that a large $\\beta$ value is needed, which seems to suggest that PMPO works when it is close to supervised finetuning (with the only difference being a different non-negative weight for the preferred and dispreferred samples).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTXEylEB5P", "reviewer": "Reviewer_7waa", "summary": "The paper proposes a preference optimization method that can utilize not only paired but also signal preferred or dis-preferred outcomes. The authors extend and improve on EM to tackle this problem and show empirical evidence favoring the proposed approach.", "questions": "- How to optimize $\\alpha, \\beta$ in practice?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Tackling the relevant and complex problem of incomplete data in preference optimization, for example, only having access to a negative examples\n- Thorough and extensive related work making the contribution clear\n- Objective is intuitive and makes sense probabilistically, especially through the use of the prior\n- More flexible than methods like DPO and might apply to novel scenarios \n- Extensive empirical evaluation on a variety of tasks from control, rl, to llm preference optimization", "weaknesses": "- Does introduce new hyperparameters that are potentially non-trivial to tune ($\\alpha, \\beta$)\n- Title could be more specific. For example, something mentioning the capability to learn from dis-preferred examples. This could also help to attract readers interested in this particular problem. Currently, it seems only appealing to researchers interested in probabilistic inference.\n- Does not improve over DPO, but might also be due to missing datasets well-suited for the setup", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KO7dHYRWBs", "reviewer": "Reviewer_uKto", "summary": "This paper presents a preference optimization method that utilizes unpaired examples and can learn from either positive or negative feedback, addressing limitations in existing approaches by that need paired data.", "questions": "See above", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "The proposed method allows for training with unpaired examples and accommodates scenarios where only one type of feedback—positive or negative—is available, making it widely applicable across different contexts.", "weaknesses": "* Personally, I find the presentation of this paper lacking. The main formulation in equation (10) is quite intuitive and provides a straightforward extension of the previous pairwise method to a more general setting. However, the derivations in Sections 3.1 and 3.2 are tedious and difficult to follow. I question the necessity of such extensive derivation from the expectation-maximization (EM) framework. It seems possible that the authors formulated the equation first and then sought a probabilistic framework to justify it. If this is the case, I strongly encourage the authors to present equation (10) prominently and follow it with a brief explanation of its connection to EM in a small subsection or appendix.\n\n* I would particularly like to see a more rigorous comparison, using the Gemma-2B model, both methods could be trained on the pairwise UltraFeedBack / LMSYS-chat-1M dataset and then evaluated on the Alpaca Eval benchmark. If PMPO cannot match DPO, it will be important to delineate the limitations of PMPO and understand when it is appropriate to use this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWfeTDZeuv", "reviewer": "Reviewer_zTUL", "summary": "The paper introduces a way to learn from unpaired preference data which is a constraint for other algorithms such as DPO. The method is motivated by expectation-maximization methods and results in an objective that weights positive and negative samples and applies cross entropy to positive samples and negative cross entropy to negative samples with KL regularization. They demonstrate their method on different benchmarks including bandit settings, DeepMind control suite, and language model alignment.", "questions": "1. How does using DPO with all (positive, negative) pairs from the 4 samples compare to PMPO?\n2. Is there an issue with pairing positive and negative responses from unpaired datasets?\n3. Why does using both accepted and rejected responses for language model alignment perform worse?\n4. Can you explain Figure 5, in particular, the different cutoffs?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper addresses the problem of being able to use unpaired data and allowing for general preference distributions. In particular, they present an objective that is derived from maximizing an expectation that is motivated by existing methods. They perform experiments on multiple datasets and demonstrate that the theory is applicable through varying $\\beta$ for only positive and only negative samples.", "weaknesses": "The main concern is whether the issue of unpaired preference data is a major problem and whether the experiments present a fair comparison between DPO and the proposed methods. In particular, if there are unpaired preference labels for a given state, it seems like a simple fix would be to pair each positive outcome with each negative outcome. Additionally, in the experiments, while the other baselines had access to all 4 samples, only DPO had access to 2. While the preferences should be paired, there is no restriction on having one pair of preferences per state. It would be more clear that the paper addresses an important issue if pairing samples from an unpaired dataset does have poor performance.\n\nFurthermore, it would be more clear if it was mentioned that the reference model was updated as at the end of section 5.1, it is mentioned that there is a slowly changing reference. This varies from the original DPO which has a fixed reference. \n\nThere also seems to be a significant drop in performance with PMPO using both accepted and rejected responses in Figure 4. Furthermore, using all responses does not result in a higher peak than only using accepted responses which is concerning as it seems that using more data actually leads to worse performance. Additionally, there are seemingly incomplete or disrupted lines in Figure 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)