Contrastive Preference Learning: Learning from Human Feedback without Reinforcement Learning
Authors: Joey Hejna
Rafael Rafailov
Harshit Sikchi
Chelsea Finn
Scott Niekum
W. Bradley Knox
Dorsa Sadigh
Venue: ICLR 2024 poster (2024)
Keywords: reinforcement learning from human feedback
preference-based RL
human-in-the-loop RL
preference learning
Primary area: reinforcement learning
TL;DR: We propose Constrastive Preference Learning, a new supervised algorithm for learning optimal policies from regret-based preferences in general MDPs.

Abstract: Reinforcement Learning from Human Feedback (RLHF) has emerged as a popular paradigm for aligning models with human intent. Typically RLHF algorithms operate in two phases: first, use human preferences to learn a reward function and second, align the model by optimizing the learned reward via reinforcement learning (RL). This paradigm assumes that human preferences are distributed according to reward, but recent work suggests that they instead follow the \emph{regret} under the user's optimal policy. Thus, learning a reward function from feedback is not only based on a flawed assumption of human preference, but also leads to unwieldy optimization challenges that stem from policy gradients or bootstrapping in the RL phase. Because of these optimization challenges, contemporary RLHF methods restrict themselves to contextual bandit settings (e.g., as in large language models) or limit observation dimensionality (e.g., state-based robotics). We overcome these limitations by introducing a new family of algorithms for optimizing behavior from human feedback using the \textit{regret}-based model of human preferences. Using the principle of maximum entropy, we derive \fullname (\abv), an algorithm for learning optimal policies from preferences without learning reward functions, circumventing the need for RL. \abv is fully off-policy, uses only a simple contrastive objective, and can be applied to arbitrary MDPs. This enables \abv to elegantly scale to high-dimensional and sequential RLHF problems while being simpler than prior methods.

Published as a conference paper at ICLR 2024
CONTRASTIVE PREFERENCE LEARNING: LEARNING FROM HUMAN FEEDBACK WITHOUT RL
Joey Hejna Stanford University jhe jna@cs stanford.edu
Rafael Rafailov Stanford University rafa 1ovecs stanford...
Harshit Sikchi UT Austin h keh @uteees.eee........ edu
Chelsea Finn Stanford University
Scott Niekum
W. Bradley Knox UT Austin
Dorsa Sadigh Stanford University
UMass Amherst
ABSTRACT
Reinforcement Learning from Human Feedback (RLHF) has emerged as a popu- lar paradigm for aligning models with human intent. Typically RLHF algorithms operate in two phases: first, use human preferences to learn reward function and second, align the model by optimizing the learned reward via reinforcement learning (RL). This paradigm assumes that human preferences are distributed according to reward, but recent work suggests that they instead follow the regret under the user's optimal policy. Thus, learning reward function from feedback is not only based on a flawed assumption of human preference, but also leads to unwieldy optimization challenges that stem from policy gradients or bootstrapping in the RL phase. Because these optimization challenges, contemporary RLHF methods restrict themselves contextual bandit settings (e.g., in large language models) limit observation dimensionality (e.g., state-based robotics). We overcome these limitations by intro- ducing family of algorithms for optimizing behavior from human feedback using the regret-based model of human preferences. Using the principle of maximum entropy, we derive Contrastive Preference Learning (CPL), an algorithm for learning optimal policies from preferences without learning reward functions, circum venting the need for RL. CPL is fully off- oof-policy, uses only simple contrastive objective, and can be applied to arbitrary MDPs. This enables CPL to elegantly scale to high- dimensional and sequential RLHF problems while being simpler than prior methods.
INTRODUCTION
As large pretrained models have become increasingly performant, the problem of aligning them with human preferences has risen to the forefront of research. This alignment especially difficult when larger datasets inevitably include suboptimal behaviors. Reinforcement learning from human feedback (RLHF) has emerged as a popular solution to this problem. Using human preferences, RLHIF techniques discriminate between desirable and undesirable behaviors with the goal of refining a learned policy. This paradigm has shown promising results when applied to finetuning large language models (LLMs) (Ouyang et al., 2022), improving image generation models (Lee et al., 2023), and adapting robot policies (Christiano et al., 2017)- - all from suboptimal data. For most RLHF algorithms, this process includes phases. First, reward model is trained from user preference data. And second, that reward model is optimized by an off-the-shelf reinforcement learning (RL) algorithm.
Unfortunately, this two-phase paradigm is founded reward models from preference data require that human preferences are distributed according to the discounted sum of rewards or partial return of each behavior segment. However, recent work (Knox et al., 2022) calls this into question, positing that humans instead provide preferences based on the regret of each behavior under the optimal policy of the expert's reward function. Intuitively, human's judgement is likely based on optimality, instead of which states and actions have higher quantity for
flawed assumption. Algorithms that learn
"Equal Contribution.
Our code is relcased htttps:///g thub. com/ the jna/ / cpp

---

Published as a conference paper at ICLR 2024
reward. As a result, the correct quantity to learn from feedback might not be the reward, but instead the optimal advantage function or, in other words, the negated regret.
In their second phase, two-phase RLHF algorithms optimize the reward function learned from the first phase with RL. In practice, RL algorithms suffer from a suite of optimization challenges stemming from temporal credit assignment, such as the high-variance of policy gradients (Marbach & Tsitsiklis, 2003) or instability of approximate dynamic programming (Van Hasselt et al., 2018). 8). Thus, past works limit their scope circumvent these issues. For instance, RLHF techniques for LLMs assume a contextual bandit formulation (Ouyang et al., 2022), where the policy receives single reward value in response to a given query to the user. While this reduces the need for long- morizoo credit assignment, and consequently the high variance of policy gradients. in reality user interactions with LLMs are multi-step and sequential, violating the single-step bandit assumption. As another example, RLHF has been applied to low-dimensional state- mate-bbsed robotics problems (Christiano et al., 2017; Sikchi et al., 2023a), a setting where approximate dynamic programming excels, but not yet scaled to more realistic high-ddmensiooal continuous control domains with image inputs. Broadly, RLHF methods not only incorrectly assume that the reward function alone drives human preferences, but also require mitigating the optimization challenges of RL by making restrictive assumptions about the sequential nature of problems or dimensionality.
In this work, we introduce new family of RLHF methods that use regret- based model of preferences, instead of the commonly accepted partial return model that only considers the sum of rewards. Unlike the partial return model, the regret-based model directly provides information about the optimal policy. A fortunate outcome of this is that it completely eliminates the need for RL, allowing us to solve RLHF problems in the general MDP framework with high-dimensional state and action spaces. Our key insight is to combine the regret-based preference framework with the principle of Maximum Entropy (MaxEnt), resulting in bijection between advantage functions and policies. By exchanging optimization over advantages for optimization over policies, we are able to derive purely supervised learning objective whose optimum is the optimal policy under the expert 's reward. We refer to our approach as Contrastive Preference Learning due to its resemblance with commonly accepted contrastive learning objectives.
CPL has three key benefits over prior work. First, CPL can scale as well as supervised learning because it uses only supervised objectives to match the optimal advantage without any policy gradients or dynamic programming. Second, CPL is fully off-policy, enabling effectively using any offline sub-optimas data source. Finally, CPL be applied to arbitrary Markov Decision Processes (MDPs), allowing for learning from preference queries over sequential data. To our knowledge, no prior methods for RLHF simultaneously fulfill all three of these tenets. To demonstrate CPL's adherence to the three aforementioned tenets, we show its effectiveness on sequential decision making problems with sub-optimal and high-dimensional off- off-policy data. Notably, we show that CPL can effectively use the same RLHF fine tuning procedure as dialog models to learn temporally extended manipulation policies in the Meta MetaWorld Benchmark. Specifically, we pretrain policies using supervised learning from high-ding -ineesiienii image observations, before fine tuning them with preferences. Without dynamic programming or policy gradients, CPL is able to macch the performance of prior RL based methods. At the same time, it is 1.6x faster and four times parameter efficient. When using denser preference data, CPL, is able to surpass the performance of RL baselines on 5 out of 6 tasks.
PRELIMINARIES
We consider the reinforreen learning from human feedback (RLHF) problem within a reward- free MDP M/r= (S,A.,,7)) with state space S, action space A, transition dynamics p(8 (s), and discount factor 7- We assume all states are reachable by some policy. The goal of RLHF is to learn policy x<a that maximizes expert user's reward function (s.a). However, since the reward function is not given in an MDP /r, it must be inferred from the expert's 's preferences. Typically, a user preference orders two behavior segments. A length-k: k segment is denoted o ($1,01,82 a2..aa...... We use o > a to indicate that segment o was preferred to a by the user without loss of generality and assume we are given dataset pef = {(o, ,0 ) of such preferences where o+ >
Maximum Entropy Reinforcement Learning. The aim of maximum- maximum-entropp reinforcement learning learn a policy æ that maximizes its causal entropy in addition to the cumulative discounted return:
2

---

Published as a conference paper at ICLR 2024
Standard Two-Phase RLHF
Contrastive Preference Learning
ar
x
log (o
RI. Algorithm
(a77l7)
X
(a]s)
Ty (als)
Regret-based Preferences J K20g442
Contrastive Learning
Phase
Phase
Reward Learning
RL
Pale"> [o)"))")
Lea: = - Roce > >
Figure 1: While most RLHF algorithms use two-phase reward learning, then RL approach, CPL
directly learns a policy using contrastive objective. This is enabled by the regret preference model.
max E 17
Σ
(1)
where a is a temperature parameter. Augmenting the reward function with an additional negated logu(als) term for reference distribution y(a yields the KL-constrained objective used in offline RL (Levine & Koltun, 2013; 3; Garg et al., 2023) and prominent RLHF approaches for LLMs (Ziegler al.. 2019; Ouyang et al., 2022). Though we adopt the standard maximum entropy framework. our approach easily extends to the constrained setting. Under policy æ and reward function we denote the value funetion by Vs" and state-action value function by Q (s,a). The advantage function, A7 (s,a) - (=, - (s), measures how much worse taking action a is than acting according to We use n' short-hand for the solution to Eq. (I) with reward function Tg. and write its corresponding corresponding value functions V*(s) and (*(s,a) instead of Va"" and Q We measure the optimality of behavior directly by using the advantage function of x.", A (s,a).
The Regret (or Advantage) Preference Model. Learning x² requires characterizing how preferences are generated according to a preference model P'tort > a J. or the probability the expert prefers o to Typically, the preference model is chosen to be the Boltzmann rational distribution over each segment's 's discounted partial return, Ek / However, such models have been shown to be inconsistent with real human preferences (Knox et al., 2022). For instance, consider sparse reward g (s,a) = Two segments that do not reach the goal would have the same partial returns even if one moved towards the goal while the other moved away from it. This inconsistency is resolved by considering preferences be distributed according to the
where rg & is the expert's 's hidden reward function.
=g).
Boltzmann rational distribution over the negated discounted regres under TE, or - - (V Q" (st,a)).. ). In this framework, user's preference indicates that a segment has lower regret with respect to their intended optimal policy. Leveraging the equivalence of negated regret and the discounted sum of optimal advantages, we equivalently write the regret-based preference model
exp Ast .ot)
PA [a
(2)
exp (*7 a
where the shorthand "+" and "_" indexing the states and actions of segments o and o- In the next section, we use the regret preference model in combination with the principle maximum causal entropy derive CPL.
CONTRASTIVE PREFERENCE LEARNING
Though recent work has shown that human preferences are better modeled by the optimal advantage function or regret, most existing RLHF algorithms assume otherwise. By learning reward function with mistaken model of preference and then applying RL, traditional RLHF approaches incur vast, unnecessary computational expense (Knox et al., 2023). Our aim derive simple and scalable RLHF algorithms that are purpose- built for the more accurate regret model of human preferences.
Modeling human preferences with regret is not new, but past work suffers from number of short- comings. Specifically, existing algorithms using the regret preference model are brittle, as they rely on estimating gradients with respect to a moving reward function, which thus far has only been approx- imated by computing successor features and assuming correct linear or tabular representation of the expert reward function T' (Knox al., 2022; 2023). Consequently, these algorithms appear unsuitable for complex scenarios beyond the simplistic grid world environments in which they have been tested.
3

---

Published as a conference paper at ICLR 2024
The key idea of our approach is simple: recognize that the advantage function, used in regret preference model, can easily be replaced with the log-rrobability of the policy when using the maximum entropy reinforeement learning framework. The benefit of this simple substitution is however immense. Using the log-pro log-probability of the policy circumvents the need to learn the advantage function grapple with optimization challenges associated with RL.-like algorithms. In sum, this enables us to not only embrace more closely aligned regret preference model, but also to exclusively rely on supervised learning when learning from human feedback.
In this section, we first derive the CPL objective and show that it converges to the optimal policy for rg with unbounded data. Then, we draw connections between CPL and other supervised-leariing approaches. Finally, we provide recipes for using CPL in practice. Our algorithms are the first examples of a new class of methods for sequential decision making problems which directly learn policy from regret based preferences without RI, making them far more efficient.
3.1 FROM OPTIMAL ADVANTAGE TO OPTIMAL POLICY
Under the regret preference model, our preference dataset Peer contains information about the optimal advantage function A (s,a), which can intuitively be seen as measure of how much worse given action a is than an action generated by the optimal policy at state s. Therefore, actions that maximize the optimal advantage are by definition optimal actions and learning the optimal advantage function from preferences should intuitively allow us to extract the optimal policy.
might naïvely follow the standard RLHF reward modeling recipe, but with advantages. This would equate to optimizing a parameterized advantage Ag to maximize the log likelihood of D Pref given the preference model in Eq. (2), or max Ao Bi (=)+ta [logPA, [o+ J]] where PAo is the preference model induced by the learned advantage function. Once advantage function that aligns with the preference data is learned, it could be distilled into a parameterized policy. At first glance. it seems like this simple two-step approach could be used to recover the optimal policy from preference data. However, turns that learning a Bellman-consisten consistent advantage function is non-trivial in both standard and MaxEnt RL, making learning valid intermediate advantage function not only unnecessary, but also harder practice.
Naïve approach. When presented with Drret-
Eliminating the need to learn advantage. In maximum entropy RL, Ziebart (2010) has shown that the following relationship between the optimal advantage function and optimal policy holds:
" (a = A2(,a))
This means that in order for learned advantage function be optimal, it must be normalized, that is e (a.a)/ada 11 Enforeing this constraint is intractable, particularly in continuous spaces with large neural networks, making naively learning Ag via maximum likelihood estimation difficult.
However, one might instead notice that the above equation establishes bijection between the advantage function A and the policy π²', namely that the optimal advantage function is proportional to the optimal policy's ligleiiko:ooo::
((s,a))-ologe (als).
This means that instead of learning the optimal advantage function, we can directly learn the optimal policy. Given preferences distributed according the optimal advantage function for the expert reward function TE, we can write the preference model in terms of the optimal policy n by substituting Eq. (3) into Eq. (2) follows,
exp Σ Enttraloge (a]1 s²
PA [o
7 (a, |s] +exp 7 aloog (0
Thus, the maximum entropy framework has led a model of human preferences that is solely terms of the optimal policy л². Using this equivalent form of the advantage-ban based preference model, we can directly optimize learned policy Ta to match the preference model via maximum likelihood with the following convex objective:
(5) Assuming sufficient representation power, convergence He will perfectly model the user's preferences, and thus exactly recover x* under the advantage-baser preference model given an unbounded amount of preference data. Specifically, in Appendix B, prove the following Theorem:
(CrL(TT (pret = Ei 1,a J-DD log expE. (1 [ss] >+xxpE, (17
exp> cloggo(aa (s
4

---

Published as a conference paper at ICLR 2024
Theorem 1. Assume unbounded number of preferences generated from a noisy rational regret- prefferneee model with expert advantage function A". CPL recovers the optimal policy n corresponding to reward TE-
This proof relies on the bijection between optimal advantage functions and policies in maximum entropy RI and the fact that the regret preference model is identifiable (Knox at.,, 2022), meaning the objective can achieve loss of zero.
Benefits of directly learning the policy. Directly learning in this manner has several benefits, both practical and theoretical. Perhaps most obviously, directly learning the policy circumvents the need for learning any other functions, like reward function or value function. This makes CPL extremely simple in comparison to prior work. When scaling to larger models, only learning the policy reduces both com- plexity and computational cost, Second, as pointed out by prior works (Christiano et al., 2017; Hejna Sadigh, 2023), reward learning can be harmed by the invariance of Boltzmann rational preference mod- els (Eq. (2)) to shifts; i.e., adding a constant to each exponent does not change Pla > o -. In CPI. the distributional constraint of the policy (Ta (a|s) 2 0 for all and To (dllssdd----d da 1) remedies this issue automatically, since adding a constant makes To da 1. Finally, per previous arguments, the policy's distributional constraint guarantees that ee A da=1. Thus, can be shown that CPL's learned implicit advantage function is always the optimal advantage function for some reward function. We call this property, defined below, consistency and prove the following Proposition in Appendix B, Definition 1. An advantage function A(s.a) is consistent if there exists some reward finction r(s,a) for which A is the optimal advantage, or A(s,a)= A, (s.a).
Proposition 1. CPL learns a consistent advantage function..
The consequences this are that no matter the amount of preference data used, CPL will always learn the optimal policy for some reward function, and adding additional preference data only improves the implicit estimate of rg.
Connections to Contrastive Learning. When deriving CPL, we intentionally chose denote preferred and unpreferred behavior segments by "+" and "_" to highlight the similarities between CPL and contrastive learning approaches. Though some two-phase RLHF approaches have drawn connections between their reward learning phase and contrastive learning (Kang et al., 2023), CPL directly uses contrastive objective for policy learning. Specifically, Eq. (5) is an instantiation of the Noise Constrastive Estimation objective (Gutmann & Hyvärinen, 2010) where segment's 's score its discounted sum of log-probabilitic under the policy, the positive example being o+ and the negative o In the appendix we show that when applied to ranking data using a Plackett-Luce Model, recovrs the InfoNCE objective from Oord et al. (2018) where the negative examples are all the segments ranked below the positive segment. Effectively. CPL has fully exchanged the reinforcement learning objective for a supervised, representation learning objective while still converging to the optimal policy. As marked success has been achieved applying contrastive learning objectives to large-scale datasets and neural networks (Chen et al., 2020; He et al., 2020; Radford et al., 2021), !), we expect CPL to scale more performantly than RLHF methods that use traditional RL algorithms.
3.2 PRACTICAL CONSIDERATIONS
The Contrastive Preference I Learning framework provides general loss function for learning policies from advantage-baser preferences, from which many algorithms can be derived. In this section, we detail practical considerations for one particular instantiation of the CPL framework which we found to work well in practice. the appendix, we include several instantiations of CPL for different types of data and conservative regularizers.
CPL with Finite Offline Data. Though CPL converges to the optimal policy with unbounded preference data, in practice policies that extrapolate too much beyond the support of the dataset perform poorly as they take actions leading to out of distribution states. Like many other preference-baset objectives, CPL's objective is not strictly convex (Appendix B.3). Thus, many policies, even those with a high weight on actions not in the dataset, can achieve the same optima of Eq. (5), We demonstrate this by formulating CPL as logistic regression problem. Let the policy be represented by one one-dimensional vector x R S..AAl The difference between positive and negative segments, Ex+ aloooa (o, |st -Eo a'aloga (a, | can be re- re-written dot-product between and "comparison" vector x., whose values are either
often interested in learning from finite offline datasets. In this setting,
5

---

Published as a conference paper at ICLR 2024
y¹. -yy, 0 indicating membership to the comparison ot a_. logistic(z) = Ite-=> we we-te the CPL objective in the finite case as
Using the logistic function,
iPautl
y = -
LCPL( Ppret) -- loglogisticaaz:1ogrr s)); where xy [s(sa)]
0 otherwise
where o denotes the 4th timestep of the preferred segment from the ith comparison in pref. We can reason about the set of all policies that yield the same CPL loss by assembling all comparison vectors into a matrix X, where the ith row of X is the vector z; for the ith comparison in the dataset. Any changes to log'T in the null space of X have no effect on the loss. In practice. SS x A >>> n, making the null space of X often nontrivial such that there are multiple minimizers of the CPL loss, some of which potentially place high probability on state-action pairs not in the dataset. In Appendix B.3 we provide constructions of X where this is true. Next, we show how this problem can be resolved by incorporating regularization into the CPL CPLectietivee
effect on the logits of the logistic function, and consequently
Regularization. In finite settings, we want to choose the policy that minimizes the CPL loss function while placing higher likelihood on actions in the dataset. To accomplish this, we modify Eq. (5) with a conservative regularizer that assigns lower loss when the policy has higher likelihood on actions in Preet- keeping ii-distribution. Though there are many possible choices of regularizers, we use an asymmetric "bias" regularizer adapted from An et al. (2023) as it performed best in our experiments. Within our objective, the bias regularizer nown-weights negative segments by X (0,1) as so:
expE, I a'loogo (s,")
CCPL(X) (TT,
ED
-log exp 7 ('alogoaaa (s, +xpxA (aalogro 1s,
(6)
If the policy places more weight on actions in the dataset, logmo(a)s) will increase. In the standard Boltzmann model, increasing the log-pro rrg-obbaaiilities of both the positive and negative segments by the same amount would have no effect on the loss. The bias, however, weighs the increased bog-probabilities of the negative segments less, which ultimately decreases the loss. Thus, while minimizer of the vanilla CPL loss function could place a high probability on unseen actions, Eq. (6) is minimized with a higher weight on in distribbutiio actions. This formally captured by the following proposition, which shows that, for fixed policy, LCPL(X) is lower when the policy places higher likelihood on actions in the dataset versus other comparisons with the same CPL.Loss. Loss.
T from D Denf and an arbitrary comparison o't > a' such that [CPL(T, at > σ)) = LCPL(T, o' > -'-) for a fixed policy т. If Eot 7 logg(a; Entt>>n(+hh "logT |s{)) then (Tr
Proposition 2. Consider a comparison a+ > >o
(CCLLA)) (T, > -'").
Essentially, this shows that the bias regularizer breaks ties in the CPL loss function by penalizing lower likelihoods. We prove this, along with more general version, in Appendix B.4. In Appendix C we also consider CPL variants with other forms of conservative regularization.
Pretraining. Pre-training the policy अe with behavior cloning (BC) is common practice in RLHF (Ouyang et al., 2022). Thus, before fine-tuning with preferences using the CPL loss, we trained the policy using the standard maximum likelihood BC objective, ming = [aogT(()ss) D could be any dataset, we chose D preef- We found that this helped performance in some cases, but hurt others (Appendix D). We posit that pre-training helps when a policy closer to the data distribution is desirable, particularly when out-of out-o--distribution actions are detrimental.
Though
4 EXPERIMENTS
In this section, we address the following questions about CPL: First, is ePI.eeffetive at fine-tuning policies from regret-based preferences? Second, does CPL scale to high-dimensional control problems and larger networks? Third, what ingredients of CPL are important for attaining high performance? Additional experiments with human data and details are included in the appendix.
Preference Data. We evaluate CPL's ability to learn policies for general MDPs from sub-optimal off- policy rollout data and preferences. In particular, we consider the training procedure commonly used for large foundation models: supervised learning followed by fine-tuning with RLHF. To do this, use six tasks from the Meta World robotics benchmark (Yu et al., 2020). First, we train baseline policies until they approximately reach 50% success rate. Then, we rollout 2500 episodes for each suboptimal

---

Published as a conference paper at ICLR 2024
stochastic policy. We then form synthetic preference datasets Deer of different sizes by sampling segments of length 64 uniformly from the rollout data. We estimate regret-based preference labels using the Q-function and policy of an oracle Soft Actor-Critic (SAC) (Haarnoja et al., 2018) model trained to 100% success on a combination of the suboptimal rollout and online data. In practice, we consider two main types of preference datasets: dense, where we label comparisons between every sampled segment (effectively ranking all segments), and sparse, where we label only one comparison per segment.
Baseline Methods. We consider four strong baselines. First, supervised fine-tuming (SFT) trains policy with BC on all segments, then fine-tunes on only the preferred segments, i.e., all a in D pref- The second baseline is Preference IQL (P-IQL), which learns a reward function from D pret assuming the partial return model, then subsequently learns a policy to maximize it with Implicit Q-Learning (Kostrikov et al., 2022), a cate-oo--thear offline RI. algorithm. Though P-IQL. was first used with the partial return model, here it uses an approximation of A as its reward, which as we show in Appendix B's Corollary preserves the optimal policy. In fact, P-IQI. should be more performant with regret-based labels, since A7,, is highly shaped reward function for TE Ng et al. (1999); Knox et al. (2023). We use Hejna & Sadigh (2023)'s s implementation of P-IQL as it outperformed several contemporary baselines. Third, we consider PPO with a KL-constrad reward commonly used for RLHF with LLMs (Ziegler et al., 2019). This is not a fair comparison, as PPO requires 3.84 million additional online state-action pairs estimate the policy gradient, totalling 25x the data for CPL 2.5K Dense and 4 x the data for CPI. 20K Sparse. Unlike the contextual bandit setting used for LLMss, PPO here require full trajectory rollouts. Finally, to demonstrate CPL's ability to extrapolate beyond the best performance in the data, we compare to %BC, where a policy is trained with behavior cloning on the top X% of rollouts according to the ground truth rg.
4.1 How DOES CPL PERFORM?
How does CPL perform with state-based observations? Our main saat- results can be found in rows 1 and of Table 1. When using sparser comparison data (row 3). CPL outperforms prior methods in 5 of 6 environments, often by substantial margin of over P-IQL, particularly in Button Press, Bin Picking, and Sweep Into environments. When applied to datasets with more dense comparisons, CPL outperforms P-IQL even more (row 1), doing so substantially in all environments. Though the dense-comparison datasets have less action coverage, they have substantially more preference comparisons than the sparse comparison datasets. We posit that more comparisons per segment is more beneficial to CPL than to P-IQL because of its contrastive objective more comparison-rich datasets are likely to have more informative positive-negative pairs that help shape the policy. Moreover, PPO is unable to consistently perform better than CPL despite access vast amounts of online data from the environment. We found PPO to be very sensitive to the KL-constraint coefficient on reward, which makes it difficult to tune as observed in prior work (Touvron al., 2023). This problem is likely exacerbated by the instability of the KL-divergence in continuous spaces and the high variance of both value estimation and policy gradients with longer horizons in robotics tasks versus LLM bandits. We find that CPL consistently outperforms %BC, indicating the CPL is indeed exhibiting policy improvement beyond the best behaviors in the dataset.
How does CPL scale to nigh-dimensioonl observations? To test how CPL's supervised objectives scale to high- aigh-dimensional continuous control problems, we render the MetaWorld datasets discussed above to 64 x 64 images. We use the network architecture from DrQv2 (Yarats et al., 2022) and as our state- based experiments. We additionally use random shift augmentations, which drastically improve the performance of RL from images (Laskin et al., 2020).
the same
Our image-based results can be found in rows 2 and 4 of Table 1. Interestingly, we find that performance moderately increases for SFT but substantially for P-IQL. We posit that this is because tata-augmentttiin, which is inapplicable in state, plays a key role in improving value representation for P-IQL. Despite this, when learning from denser preference data (row 2), CPL still outperforms P-IQL in 4 of 6 environments and ties on Sweep Into. When learning from sparser comparisons (row 4), CPL and P-IQL perform comparably most tasks, even though CPL is drastically simpler than P-IQL. Again, the gap in performance between CPL and P-IQL is higher with denser comparison data.
These results only more impressive considering CPL's significant reduction in complexity. P-IQL must learn a reward function, a Q- function, a value function, and a policy. CPL avoids all this, and only learn a policy, drastically reducing training time and parameter count. As we can see in Table 2,
7

---

Published as a conference paper at ICLR 2024
Bin Picking
Button Press
Door Open
Drawer Open
Plate Slide
Sweep Into
PPO
83.7 3.7
22.7 1.9
79.3 + 1.2
66.7 8.2
51.5 3.9
55.3 6.0
State
2.5k Dene
SFT
66.9 2.1
21.6 +.6++1.6
63.3 1.9
62.6+22
41.6 3.5
51.9
P-IQL
70.6 4.1
16.2 5.4
69.0 + 6.2
71.1 2.3
49.6 +
6006 +
CPL
80.0 2.5 2.5
24.5 +:
80.0 6.8
83.6
61.1 3.0
70.4 3.0
Image 2.5k Dense
SFT
74.7 4.8
20.8 +:
62.9 + 2.3
64.5 7.6
44.5 3.2
52.5 2.5
P-IQL
83.7 0.4
22.1 0.8
68.0 + 4.6
76.0 + 4.6
51.2 2.4
67.7 + 4,4
CPL
4.9
27.5 +
73.6 6.9
80.3 +
57.3 5.9
68.3 4.8
PPO
4.3
24.5
82.8 +: 1.6
6.62
60.7
58.2 0.6
State 200 Spurse
SFT
67.6
21.44227
63.6
63.510099
4f.9.3311
30.9 E3.2
P-IQL
3.3
19.5 1.8
79.0 + 6.6
76.2
55.5 4.2
73,4
CPL
83.2 3.5
29.8 +:
77.9 9.3
79.1 5.0
56.4 + 3.9
81.2 +
Image 20k Sparee
SFT
71.5
22.3
65.2 2.2
67.5 +
41.3 2.8
55.8 2.9
P-IQL
80.0 2.3
27.2 4.1
74.8 + 5.8
80.3 1.2
54.8 5.88
72.5 2.0
CPL
78.5 4.1
31.3 +:
70.2 2.1
79.5 1.4
61.0 + 4,2
72.0 +: 1.8
State
% BC
10%
62.6
18.9 +: 1.7
57.5 4: 3.0
61.5
3.7
39.1 +: 2.5
49.3 2.1
5%
64.6
18.2 +: 0.6
59.8 + 1.6
61.3 + 1.8
38.6 + 2.5
49.2 1.9
Table Success rates (inpercent) of all methods across six tasks on the MetaWorld benchmark on different datasets. The leftmost column contains the observation modality (state or image), the number of segments in the dataset, and the means of labeling comparisons (dense or sparse). Dense refers to labeling every possible pairwise comparison and sparse refers to labeling only comparison for every two segments, e.g., 10k comparisons for 20k segments. We run four seeds for state and three seeds for images. We report the maximum average performance across seeds over 8-checkpoint, 200 episode evaluation window (details in Appendix E). Bolded values are within 1% of the top- popfforrrnin method. The bottom section shows oracle performance of %BC with access to ground truth reward. State-spaces results include PPO (delinated with a dashed line) which is not 1-1 comparison as uses 3.84 million extra online transitions.
CPL runs 1.62x faster than P-IQI. on images and has less than quarter of the the parameters. As networks get larger and larger, the performance gain from using CPL would only increase.
4.2 WHAT CONTRIBUTES TO CPL's PERFORMANCE?
As alluded to in previous sections, find that the gap in performance between CPL and baselines is higher for datasets with denser comparisons. This is consistent with prior works in contrastive learning (Robinson et al., 2021). To study this effect, evaluate CPL's performance as we increase the number of comparisons sampled per segment over a fixed dataset of 5000 segments, We show results of this for Drawer Open with baseed observations on the left of Fig. 2 and include the rest in Appendix D.3 in addition to dense data scaling. Overall, we find that CPL benefits from an increasing number of com- parisons per segment all tasks except Plate Slide. P-IQL is less affected, though sometimes performs worse with more comparisons, which suspect is due to reward under-fitting. This highlights another krawback of P-IQL- due to its higher number of components, has more hyperparameters and is consequently more sensitive changes in the dataset. We tuned hyperparameters for all methods with 10K comparisons, then left them the same for scaling experiments.
Finally, ablate both of CPL's hyperparameters the temperature value and bias regularizer for Drawer Open on the right of Fig. While CPL generally performs well with all values, we find that higher performance could have been attained with further aaper-paramete tuning, particularly for X. In the Appendix C we ablate more design decisions, like the choice of conservative regularizes
5 RELATED WORK
Though RLHF has recently surged in popularity, learning policies from human preferences has been long-studied problem, referred preference-based RI. (PbRL..
8

---

Published as a conference paper at ICLR 2024
Data Density, Drawer Open
Hyperparameter Ablations on Drawer Open
Success Rate
Suconss Rate
8.80
0.79
0.70
alphx
lamboda
alpba 0.1
lambda 0.5
0.60
alpha 0.5
lambda 0.75
Comparisons per Segment CPL
503030
200020
300000
200000
200000
300200
400000
Training Steps
Training ttegs
Figure 2: Left: Performance when increasing the number of comparisons per segment on Drawer Open state with 5k segments on two seeds. Right: Ablations on CPL's hyperparameters on Drawer Open from State. The dotted vertical line shows when BC pretraining stops.
PbRL methods typically start by learning a reward function, usually from pairwise comparisons, then use an RL algorithm for policy optimization (Fümkranz et al., 2012). While Akrour et al. (2012; 2011) Wilson et al. (2012) were some of the first examples of PbRL, more recently several works have shown that, provided thousands of queries or sufficient pretraining, PbRL can train deep neural-network policies for control using comparisons (Christiano et al., 2017; Lee et al., 2021; Ibarz et al., 2018; Brown ct al., 2020; Hejna & Sadigh, 2022; Shin & Brown, 2021) or rankings (Brown et al., 2019; Biyık et 2019; Sikchi et al., 2023a). These approaches, however, dimensional state-based control because of the challenges RI. faces when scaling to larger inputs and net- works (Ota et al., 2021). In the past, removing RI has lead to effective algorithms for soal-conditioned RL from images (Hejna et al.; Eysenbach et al., 2022). CPL does the same but for PbRL. Other works address the problem of selecting feedback (Sadigh et al., 2017; Biyik al., 2020; Daniel et al., 2015), which we consider complementary because CPL can benefit from higher quality data elicitation.
generally demonstrated only on low-
To scale RLHF. recent approaches for refining LLMs have ignored the temporal component of RL, and instead treated text sexe-renerttion as a contextual bandits problem (Ziegler al., 2019). While this approach has proven effective at tasks like summarization (Stiennon et al., 2020; Wu Hu, 2018), instruction following (Ouyang et al., 2022; Nakano et al., 2021), and even image generation (Lee et al., 2023; Black et al., 2023), fundamentally ignores the fact that interaction with spanning multiple turns. Unlike these methods, CPL works with general MDPs. CPL's ' unique ability to learn from sequence data with only supervised objectives makes it a prime candidate for scaling more complex problems. In fact, Direct Preference Optimization (DPO) (Rafailov et al., 2023) recently demonstrated that supervised objective similar CPL works better than RL in the contextual bandits setting. We show in Appendix B that DPO can be derived as a special case of CPL in which segments are of length and always start at the same state. This parallels Knox et al. (2023), who show that the common contextual amdittappprachh is a special case of the naïve approach from Section 3.
Method
Params
Runtime
P-IQL CPL
9.6m
16.5 hrs
2.1m
10.2 hrs
Table 2: Computational efficiency of each method when learning from pixels for 200k training steps on single TitanRTX GPU.
is often sequential,
To derive CPL's objective, we leverage knowledge from works building on the principle of maximum entropy in control (Ziebart et al., 2008; Ziebart, 2010; Haarnoja et al., 2017). The resulting contrastive update directly learns the optimal policy with fully off- off-policy data. This is unlike many Raseb RLHF algorithms in both langauge (Ziegler et al., 2019) or control (Christiano et al., 2017) which require on policy rollouts and additional learned components that have been shown to increase variance (Hejna & Sadigh, 2023). Similar contrastive learning objectives have shown to be effective for temporal representation learning (Ma et al., 2023), even with preference data (Kang et al., 2023; Xu et al., 2023).
DISCUSSION
In this work we introduce CPL, a novel framework for RLHF using the regret preference model. Theo- retically, we proved that CPL always learns consistent advantage function and converges to the optimal policy for the expert's 's reward function. Practically, we showed that CPL's supervised objective is able to outperform RL baselines when learning complex manipulation policies from dense preference data while being simpler and 1.6x faster. Due to space constraints, we include limitations in Appendix A.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by NSF Award 2006388, NSF Award 2218760, NSF Award 1933032, NSF Award 1953032, NSF Award 1941722. NSF Award 2125511, NSF Award IIS- IIS-1749204, AFOSR YIP, ONR (N00014-21-1-266)
AFOSR (FA9550-20-1-000
ARO (78372-CS, ONR, Ford, DARPA YFA, and the Center for AI Safety. JH supported by a DoD NDSEG Fellowship. CF is CIFAR Fellow in the Learning in Machines and Brains program. WBK is supported by UT Austin's 's Good Systems grand challenge. We would like to thank Archit Sharma for valuable discussions the conservative regularizer used in CPL. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the sponsors.
CONTRIBUTIONS
JH led the project, contributing to all aspects including ideation, theory, experimentation, and writing. RR proposed linking advantages and likelihoods and contributed to early stage ideation. HS contributed to the theory, experiment design, and ran experiments. CF, SN, WBK, DS oversaw, advised, and provided feedback on the project.
REFERENCES
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-basse policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, 2011.
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference learning beaming-baseed based reinforce- ment learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, FCML. PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part 11 23, pp. 116-131. Springer, 2012.
Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung- MMn Kim, and Hyun Oh Song. Direct preference-based policy optimization without reward modeling, 2023.
Erdem Biysk, Daniel Lazar, Dorsa Sadigh, and Ramtin Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In 2019 IEEE 58th conference on decision and control (CDC), pp. 347-354. IEEE, 2019.
Erdem Biyik. Nicolas Huynh. Mykel J. Kochenderfer, and Dorsa Sadigh. Active preference- based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprimt arXiv:2305. 2305.33333311 13301, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 1952.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. International conference machine learning. pp. 783-792. PMLR, 2019.
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Niekum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pp. 1165-1177. PMLR, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learninn from human preferences. In Advances in Neural Information Processing Systems, 2017.
10

---

Published as a conference paper at ICLR 2024
Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with novel acquisition function. Autonomous Robots, 39(3):389 40099405,22222 2015.
Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Russ R Salakhutdinov. Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 15:35603-356201 2022.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Johannes Fürkkranz, Eyke Hüllermeier, Weiwei Cheng, and Sang-Hyeun Park. Preference-base
reinforcement learning: a formal framework and a policy iteration algorithm. Machine learning, 89: 123-156, 2012.
Divyansh Garg. Joey Hejna, Matthieu Geist, and Stefano Ermon. Extreme q-learning: Maxent RL without entropy. In The Eleventh International Conference on Learning Representations, 2023. URL httttt openreview n forum? wwworrmiddsSJoof de
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 297-304, Chia Laguna Resort, Sardinia, Italy, 13-15 May 2010. PMLR. URL https: /ttpeeeiicceeessseeeeeeeepesssse .mlr. prwss/ v9 https//////tttt/////ga///// gutmanntnnnnn 0 html.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR,
2017.
Tuomas Haarnoja, Aurick Zhou, Picter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pp. 1861-1870. PMLR, 2018
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Joey Hejna and Dorsa Sadigh. Few-shot preference learning for numan-in-the-logg RL. In Conference on Robot Learning. 2022.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-bast bbaen rl without a reward function, arXiv preprint arXiv:2305. 15363, 2023.
Jocy Hejna, Jensen Gao, and Dorsa Sadigh. Distance weighted supervised learning for offline interaction data. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, PMLR. URL BtRL:t/:::// wtt.ttpss///axxi...ot wwwp//oorg/aaa.o abs 4 13774.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances neural information processing systems, 31, 2018.
Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, pp. 5084-5096. PMLR, 2021.
Yachen Kang. Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference- guided policy optimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 15753-15768. PMLR, 23-29 Jul 2023. URL https:: http::::////cceeeiiee/e mwr. wrr..pres...... v202// kang23b. html.
Changyeon Kim, Jongjin Park. Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. In International Conference on Learning Representations, 2023.
11

---

Published as a conference paper at ICLR 2024
W Bradley Knox, Stephane Hatgis-Kessell, Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro Allievi. Models of human preference for learning reward functions. arXiv preprint 20tp:pi20.22..2222200022231 2022.
W. Bradley Knox, Stephane Hatgis-Kessell, Sigurdur Omn Adalgeirsson, Serena Booth, Anca Dragan, Peter Stone, and Scott Niekum. Learning optimal advantage from preferences and mistaking it for reward, 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances in neural information processing systems, 33: 19884-19895, 2020.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Reedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre- traininiin..... In International Conference on Machine Learning, 2021.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning eex-tto-mmage models using human feedback. arXiv preprint Xixv:2302.12192, 2023.
Sergey Levine and Vladlen Koltun. Guided policy search. In International conference on machine learning, pp. 1-9. PMLR, 2013.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu, Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016643, 2020.
Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. VIP: Towards universal visual reward and representation via value- implicip pre-training. In The Eleventh International Conference Learning Representations, 2023. URL https: // //openevieww net wtw.://www.eee/ooooiioooooo----
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang. Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martin-Marttn. What matters in learning from offline human demonstrations for robot manipulation. In arviv preprint (...v:2118.032200 2021.
Peter Marbach and John N Tsitsiklis. Approximate gradient methods in policy-space optimization of markov reward processes. Discrete Event Dynamic Systems, 13:1 13:111-14,,0 2003.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint arXi:: 2021.
Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. Icml, volume 99, pp. 278-287. Citeseer, 1999.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv: / 0074033788 2018
Kei Ota, Devesh K Jha, and Asako Kanezaki. Training larger networks for deep reinforcement learning. arXiv preprint Stti::2202.099200 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint 22X1v:2203.02155, 2022.
Robin I. Plackett. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24(2):193 193-202, 1975.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-87866 8763. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2024
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint (t:Xfvv23305.82900 2023.
Joshua David Robinson, Ching- Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021. URL
https: httpa:://pppereeeee http//peeeeeeeeeeeeeeeeeeeeeeeee net/ / forum? X
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.
Daniel Shin and Daniel S Brown. Offline preference-based apprenticeship learning. arXiv preprint arXiv:2107.09251, 2021.
Harshit Sikchi, Akanksha Saran, Wonjoon Goo, and Scott Nickum. A ranking game for imitation
learning. Transactions on Machine Learning Research, 2023a. ISSN 2835-8856. URL https:
//openzeview...////////// net./ wtte:/ooummmmmm id dd-iiiiiddd
Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. In Sixteenth European Workshop on Reinforcement Learning, 2023b.
Nisan Stiennon,
Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint 20X0::2009.01:325.. 2020.
Hugo Touvron,
Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wwxXiv:3307.09288 2023.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil.
Deep reinforcement learning and the deadly triad. arXiv preprint arXiv: (018882..226444 2018.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems, 2012.
Yuxiang Wu and Baotian Hu, Learning to extract coherent summary via deep reinforcement learning.
In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018
Wanqiao Xu, Shi Dong, Dilip Arumugam, and Benjamin Van Roy. Shattering the agent-environment interface for fine-tuning inclusive language models. arXiv preprint arXiv:2305. 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022. URL https: /ttp:////eeeeeeeeeeeeeeeeeee iew net/ forum? fimmmidd==ss hi==_ddd_=e_ssssseeeeee
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning, 2020.
Brian D Ziebart, Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Camegie Mellon University, 2010.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1438. Chicago, IL, USA, 2008
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei. Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv 2019.
preprint
13

---

Published as a conference paper at ICLR 2024
A DISCUSSION
Limitations. CPL, like other RLHF approaches, assumes knowledge of the human rater's s temporal discounting (i.e., of the discount factor 7), which practice would be difficult communicate. As CPL's loss function is computed over segments, it requires substantial amount of GPU memory for large segment sizes. Finally, no model of human behavior is perfect.
Future Directions. Several exciting research directions remain. First is scaling CPL. to larger datasets and architectures where we believe its benefits will be more pronounced. One potentially exciting application is LLMs, where CPL enables fine-tuning multiple steps of turn-based dialogue. To our knowledge, no multi-step preferences dataset currently exists for LLMs. Second, our work only considers offline data generated by suboptimal policies. An online version of CPL could be developed that works with online human feedback, allowing policies to continually improve.
B THEORY
B. PROOF OF CONSISTENCY
We first prove lemma about the consistency of CPL as is used when proving convergence.
Lemma 1. Any function A(s,a) that satisfies S da Vs S consistent advantage function under some reward function r in the MaxEntRL setting..
Idea. Given advantage A(s,a), we want to show that there exists a reward function for which A is the optimal advantage function.
Proof. Given [A Als.a)/a du = 1, consider the corresponding policy x (als) Aa Let the reward function be the advantage, or r(s.a) = aolog We can determine the optimal policy x¹ for this reward according to Eq. (1):
π argmaxlE,. ((s0))
argmax
E
Bamp! (5) (as)
argmaa
1=1
8
arggmn
B (a (D a
Thus, the objective is point-wise maximized if and only - VsES. Therefore, Λ is the optimal policy for reward function r(s.a) = A(s,a). Under this reward function, π A which implies that A is a consistent advantage function.
Proposition 1. CPL learns a consistent fuunntage
Optimization via CPL fits a valid policy subject to (alss)s Vs S, with corresponding MaxEnt Advantage function = (alo))
Ala.a)/a da / cloog(aal) da da= (als)dda=1
Thus, by the above Lemma CPL fits consistent advantage function.
Note that assume that all states reachable and therefore have support in (s) for any optimal MaxEnt policy.
14

---

Published as a conference paper at ICLR 2024
The reward function function and the reward function defined the optimal advantage functiion forr, AG, have the same optimal MaxEnt policy.
Corollary 1.
This corollary can be seen by examining the proof of Lemma 1. According to the MaxEnt RL objective for reward r the optimal policy is ग. = (Ziebart, 2010). Therefore A = alogı, Repeating the steps of Lemma by setting r' = A - alogm, we get the following objective for the optimal policy Tp. with respect to
T.. argmaxE (slosa))aloo(((l))
argmax
B erupt (s) ((ag))) (aloolaa))
aargma
Benot (s) -
argmin
E
Since the final expression above is minimized only when = then n.. = In other words, the reward function r and reward function r' A4 have the same optimal MaxEnt policy.
Implication for our P-IQL baseline. With regret-based preferences, algorithm that learns reward function while assuming the partial return preference model and then conducts RL on that learned reward function-including includiig P-IQL-is actually using approximation of A the reward function. This corollary therefore implies that that approximation is error- free, then P-IQL is using a reward function that preserves the optimal policy of the expert user's reward function r'v This application of the corollary extends the similar insight of Knox et al. (2023) to the MaxEnt RL setting. Furthermore, alog= (a|s) = implying that (*(s) as the
as shown in Lemma when π = n². Ep- [r(,ao) reward and entropy regularization over the occupancy measure from any state is exactly the value function. Thus, as originally pointed out by (Ng al., 1999), using A as the reward function results in high amount of shaping, so much so that horizon of one transition is sufficient to determine the optimal action in each state (Knox et al., 2023).
B.2 PROOF OF CONVERGENCE
Theorem 1. Assume unbounded number of preferences generated from noisy rational regret- preference model with expert advantage function
functiogefuunctin A". CPL recovers the optimal policy x²"
Proof. Without loss of generality Jk denote a segment of length i where the state-actions in the segment are denoted by σk = Let y be the label indicating whether the expert regret preference model prefers of of i.., y PA [ot > of). Let Aologá be the implicit estimate of A learned by CPL. For brevity, we will use the shorthand A(ox) = x ((() denote the discounted sum of advan- (a")
let
1. For the purposes of this proof only, let
(80,00,81,4) (0,,0n,s,:1,,.-000. 0k..0..----1,
A(o!)
tages of a segment a. Let (() = Bern( (4~Ga))444400)) and Q(ol. Berní cAle tedion) The cross-entropy CPL loss function can be re-written as follows:
LePL (A, =Eg (Pot
The KL divergence is optimized only when the two distributions exactly equal. Because the preference model rational and we assume sufficient representation power and unbounded data, it is possible for the loss to converge zero by pointwise matching KL--divergence for each two comparisons (See Knox et al. (2022) for more information specific to the identifiability of regret based preferences). Thus, under the assumption of unbounded data, for all possible segments af o we must have that
A
Vol.of.
( + ( (aa)) A(o
15

---

Published as a conference paper at ICLR 2024
Rearranging, get:
(
Consider as - where P [0,1]. We will show that the above equality also holds for all sequences of length k Consider the last action for the segment of denoted as a then:
Σ 11A
Σ 1
eA A(o
Now, we will use the consistency of CPL. Per Ziebart (2010), for the optimal value function A", EeeAeA (s,a) 1, Vs. Because CPL is consistent (Proposition 1), we also have that Lae E A( 1, Vs. We use this, in combination with the fact that all possible dynamically feasible segments of length k k-1 are subset of dynamically feasible segments of length & to arrive at:
Vol 1 Aloh (eA - 1) Alo2)
Inductively have that:
Vol. (a 2(o))
Applying the same argument again, this time for ohe we have
VsP.at e ("(aa))))) = Acf.,09)
which is equivalent to (s,aa)A((s.)) Vs,a.
B.3 CONVEXITY OF CPL WITH FINITE DATA
CPL is convex, but not strictly convex. Here we show that the CPL. loss function is convex in loga. Consider the logistic regression interpretation of CPL for finite data
LCPL(T (Te, Dpeet) =
loglogistic(ax;]ian (ogTT
where x; is the "comaprison" vector for the ith comparison in D poet- We can re-write this using matrix notation as:
CcrL ( (p.D
loglogisii((a2Xlogrrraaa
1=1
The hessian of this objective (logistic regression) with respect to log X DX, where D the diagonal matrix such that D;. = guaranteed to be positive semi-definite making the objective function convex. The distributional constraint of CPL, that E.J. overall objective is convex.
As X DX is symmetric, it
e.w.......aaaaaaa.... (ogn((ls)))))
also convex as cloga convex in loga. Thus, the
However, this does not imply strict convexity, or that there is a unique solution. X DX is only positive definite if it is full rank, which is unlikely to happen in practice as usually |S <A>>>>>>>>> This means that the objective is likely not strictly convex in practice, as there can exist more than one minimizer of the objective, formally denoted म argmin, LeFl (TT, Ppee)) To prove that CPL is not always strictly convex. we construct another policy a' such that LCPL (ft (pet) LCPL (i'' ,Dret). First, demonstrate this on a simple single-state MDP and then provide a general construction for arbitrary MDPs with discrete actions.
simple example. Consider single state MDP with three actions a a2 3 and expert reward function a=").a)) r¹ where indexes the actions. It can be shown that, due the single state nature of this simple MDP, the optimal maximum entropy advantage function A (s. a') = r². Consider preference dataset Drrr consisting only of comparisons between segments (s.a²) and
16

---

Published as a conference paper at ICLR 2024
(s,a²). According to the regret preference model, the expert labels these preferences according to Bern [ern(expr') / /(expr+ expr*)) and thus we expect some labels in the preference matrix X to conflict. The finite CPI. loss becomes
CCPL (T. - loglogistic aoloo--llaaaaaaa (a²)s))
c2loglogistic(alogz (alog( [s][
where 01 and 0 are the number of comparisons where a! and a² were preferred respectively. By taking the gradient of this objective, it can be shown that the loss is optimized only when logistic (alogT (a Intuitively, this makes sense, as the logits are optimized produce the same ratio of preferences as found in the dataset. However, when we consider the unseen action a² to which we can assign arbitrary probability, the existence of multiple optimizers स becomes clear. For example, take ej = 02. By the conditions above its straightforward to see that स [0.5,0.5,0.. is an optimum of the CPL loss function. However, क = (0.1,0.1,0.88 achieves the same loss as its difference in log probabilities loga(a! [s) lo is the same. If c2 =0,
- a trz or reducing, a (logr(("ls))
log _
have no conflicting preferences, = [1,0,0] and the implied A = logf is undefined, implying some of the reward values are infinite. This means we do not have enough data to accurately fit rr. Next, we provide a construction for more general MDPs in the presence of OOD actions.
A more general construction. Let the expert reward function to be bounded. We will interpret the finite preference dataset as matrix as described in Section 3.2 and use (X) to denote the null space of X. For a vector u E (SAAA we use u(s.a) index the vector state and action a. Assume the following about X:
1. There exists a vector u E (XX) such that for state action s,a contained in X, u(s,a) #0. In other words, the null space non-trival on the support of 'D pref-
2. For every state in the dataset where there is an action such that u(s.a) O0, there exists at least one ofi-t--iisttibution (OOD) action dOoD not in the dataset. The indicator vector for 8,000D is thus basis vector for N (X).
Let 7 be the minima of the CPL loss function. We will construct R' as follows. Select a vector E (XX) that is non-zero for at least one s,a pair X. As u (X)), we have that LCPL (ft preet) = CCM1, Ccrn.(ennett (elog+*" (DDpee) However, e clog++a violates the policy constraint as may not integrate one. We can fix this problem by adding or removing probability mass from the OOD actions we have assumed exist states where non-zero. We do this by constructing another vector e E NXX) by choosing one GOoD each state without loss of generality. By examining the total sum of probabilities of the modified policy,
Vs X, Σ tte
Σ [(als)eu(sa) + (upolls)
we can normalize the sum using the indicator vectors for 8,000D. which are necessarily in the nullspace N (X)). Consider vector u such that at each state s, u(s,a)= = except for at dooD, where v(S,00OD)
log(1 Earacon & (als)e log (MooD/s). Then,
Vse
te +x.)))))
Σ ((lls)e + se (5.0000) =1
As is formed from linear combination of basis vectors of N(X), E N(X). Consequently, LCPL(fi, (frD = LCPL ( toug Prret) and by the above construction = " is valid policy. This completes the construction.
We have shown that an infinite number of policies can attain the same optima, just by shifting the amount of probability assigned to OOD actions. For some of these solutions, the entire mode of the policy is potentially oft-oof-dsttrbuun In the offline setting, the pessimism principle dictates that we should discourage modes that out-of-distribution. We fix this by introducing regularization.
17

---

Published as a conference paper at ICLR 2024
B.4 CONSERVATIVE BIAS REGULARIZATION
CPL loss translates a relative weighting between preferences to a policy, but does not employ any mechanism ensure the learned policy is close the dataset. In the offline setting, this can be detri- mental if the learned policy incorrectly extrapolates to out of distribution actions. A similar approach, under the name of pessimism or conservatism, is commonly seen in offline RL literature (Levine et al., 2020; Jin et al., 2021; Sikchi et al., 2023b), As expalined in Section 3.2, we want to learn policies that have a highh-robabbility on the dataset. However, there are many datasets that potentially have the the same loss, as LCPI. depends only on the difference in probability for each preference comparison, or Eo+ 1ogoo [5x) - 7 'logT( (apl (t), and thus constants added to the log probabilities of each segment cancel. However, we would prefer that higher loss is given when the policy is assigns lower probability to actions in the dataset.
To remedy this, introduced bias regularizer X (0,1) in Section 3, which leads to the modified preference loss:
exp Palogg (a 1s{t
(T, Exppr log
exp / >'alogg
(a s Eo 7'aogr/a |s
Next, we prove that this loss discourages the policy from learning modes that are out-of-distribution, starting with the proposition from the main text.
Proposition 2. Consider comparison a x from D prey and an arbitrary comparison o' o such that (CPI(TT,a* > a a)= (cPI(m, o > ) for a fixed policy n. E. 7 {8t)>
Eort 7 logg (a st) ), then (CPl(A((T,O) +o-+-oo - < CCrL(x)(T (T, >ad
Succinetly, this proposition states that if preference comparisons each achieve the same loss. the less likely comparisons under the policy (in this case o'+ > o'-), will have higher regularized CPI. loss. Essentially, this shows that the regularized objective encourages the policy to have higher likelihood on the provided comparisons than any other potential comparison that exists. Proof. By the stated assumptions, it must be that En- y >'logT (as s.) + En- logz(a, (st) for some > 0. As the two comparisons also have the same CPL Loss, their logits must be the same, or
[rloggr(a(ss)) >1'og
Σ PPlog(((((8)))
Σ> (llogT(a))5))
=
Consequently, the same must hold for the negative segments, or E.. 7" log x (a (st) + E. (arls)) We can then examine the regularized CPL loss under each comparison. First, we evaluate the finite regularized loss for o+ > algebraically simplified for clarity:
cxp E.p>.. 7 y'alog (arls:)
(T ) =log
exp E.t
of alooo (a, 8t) +expxEx E
loglogistic
(logT(aa(*t))
= =log
1+exp
(ogs Σ> Er'logT(aelss))
We can then compare this to the regularized loss for o't o
(T )
1+exp
A
(ar(:)))s:)))) (I'oogg(aassss)
= llog
1+exp
A
( (loog 5) Σ (n'loggT(als))) -61)
at
= llog
(+cxp(6(1-A)1p
1 > (lggT((a - Exlogg((alss)
18

---

Published as a conference paper at ICLR 2024
The key step in the above is substituting the relationship between the log probabilities of the comparisons.
As > 0 and 0 <<><1 1, it can easily be seen that the loss is lower for o+ > o .lleting us conclude that
(CPL(N)(T,.' > aa (CPLLA)(TTT"")) >o"))
We can extend this proposition to the regularized CPL loss over entire datasets as follows:
For a fixed policy x, consider two preference datasets D,, (lo) oo ) and D' =(o;; ) such that Vm = 1, 2, .... n, Leri(Dnn T) Lcr (D T). Then, if Ed;t of log π (a) I
E (oogg((a st) all i and stricily for at least one
(CPL(N) <CCLL)) (D
The proof of this amounts to first noticing that, because the preference losses are the same for every ordered subset, the losses for the ith datapoints D' and D,, must be the same. Then, we can repeatedly apply Proposition 2. Since the inequality is strict at at-least one datapoint, the regularized loss will be strictly lower.
We can construct datasets for which this is applicable. For example, consider dataset 'D containing total ordering over n segments, a 2 _o²> ... o". The unregularized loss for this policy and dataset (CPL(T. (T.)). We can construct another dataset D' over different set of totally ordered segments from anywhere in the state space o²1 2 d² 2..> 00 such that:
(ogogT(ar (s:) Σ (agss())ss))
for all i= 1.2 and some 8 0.
B.5 CPL FOR RANKINGS
We can derive a version of CPL for ranking data using a Plackett-Luce model (Plackett, 1975). We denote the chosen ranking as a permutation T: [K]-[K" [K]] where K is the number of segments presented, "...... The Plackett-Lure model under regret based preferences is:
K II
exp Eo . T(k)
A"
P(T "....oK)=
7()) T(5)
This model generalizes to Bradley-Tere (Bradley & Terry, 1952) when K To learn the opti- mal policy, we maximize the log likelihood of the above and make the same substitution CPL, aloor* (als) = (s,aa) This gives us the CPL loss function for rankings, which can be seen as verison of the InfoNCE objective. Without loss generality, we order the permutations such that 20² _o.
exp Exp alogoo (st)
LCPL (To, Drank) E ..... DenD
log Exp E.
Eo Go (alogoo
A=
Except for sum over k. this is the exact objective from Oord et al. (2018) where scores are discounted sum of log probabilities over the segments.
B.6 DIRECT PREFERENCE OPTIMIZATION SPECIAL CASE OF CPL
Reduction via Maximum Entropy Advantage. Note that by the Bellman equation,
A (s.a) ((d
(V'''))
DPO (Rafailov et al., 2023) assumes the cintexxfual-bnndii setting, thus the MDP terminates after a single step and there is no next state s'. As we can see from the above, in this setting, A Assaaa) = (s V* (s). DPO also assumes that all preferences start from the same state s, and thus only actions a and differ. This is consistent with RLHF on LLMs as humans score "responses" to fixed prompts.
19

---

Published as a conference paper at ICLR 2024
The regret preference model becomes:
(spra(s,at)-V11)
PA- 1
expræ(s,a")
expra(ssat))exxprs(e
which is the same preference model used in DPO. From here the same conservative deriva- tion as DPO can be applied by noting that, for KL-constrained contextual bandits, (a(ss) = (s,a)-V*(s) ((a]s)e"(<saa)V"" = (((a)))""(ssaa=VV" for reference distribution μ. Solving for Tg. we can perform a substitution just like in CPL to arrive at the DPO objective.
CPL under Constrained Regret Preferences. We can also consider setting where users provide preferences constrained to reference distribution μ. This might arise in scenarios where users are only shown a fixed set of behaviors, and do not extrapolate far beyond them. Though we do not believe this premise has previously been considered, leads an interesting result.
Assume preferences to be distributed according to the KT constrained advantage function. In this setting, π = (s(s,)) and by substitution the CPL loss becomes
exp E. 7oalog so(a, 1s!)) M(a (47)
CCPL (ne, Bo :o DaDgat
llog
exp *aoog ma(a 1o(1) + exp aalog ula, 1s,
so(a; le;
p(a, [=!
which is essentially multi-step generalization of DPO which has not previously been considered. In the next section, we expand on this as a variant CPL.
C VARIANTS OF CPL
In the main body of the paper, we presented the version of CPL. which we found to consistenly attain good performance. some of our experiments, we also considered two other variants CPL. We detail these below.
BC-Regularized CPL. Instead using our biased conservative regularization from An et (2023), consider using simple BC regularizer. This can be derived by considering the objective:
minCCeL (To D Ppert) s.t. Es-pp (DKL(((1s))]
Relaxing the problem via Lagrangian duality with langrangian 8, we arrive a BC regularized version of CPL.
(TO,LCCLL(TT,,.)) (a [ags]
We refer to this method as (BC).
KL Constrained CPL. We also consider the setting where preferences are assumed to be distributed according to the constrained advantage function. Though in practice we sample preferences according to the maximum entropy advantage function, First, we learn the reference distribution μ using behavior cloning. Then, we use constrained CPL with bias regularization, making the final loss function:
found this approach to still work in many settings.
exp Exp alaoo Talaf (st) u(e, [x!)
(TO, =B BioTo-J-DD - llog
(o (0 (x)) pla (57)
-
To (o 1s5) p(a, (57)
We refer to this method as CPL (KL).
CPL with Dense Preferences. When learning from "dense" preference data, it is possible to augment the batch to include more comparisons using the transitive property. Specifically, given batch of b
20

---

Published as a conference paper at ICLR 2024
segments, compute all possible pairwise comparisons within a batch:
(CPL(A)-D= = 6- 16-1
(nils)) exp Malloaalllllllooo s/) +exp " alogno ( (a/1s²)
(or + oj Hog
1=
This provides as much contrastive signal per-batch as possible. We applied this technique to our CPL experiments with images, and found that lead a slight increase performance for some tasks.
D EXTENDED RESULTS
In this section we provide our full experimental results:
1. Learning curves from state for CPL, baselines, and variants described in Appendix C.
2. Learning curves from images for CPI. and baselines.
3. Scaling results for CPL and P-IQL with different sized dense datasets and fixed sparse datasets with a varying number of comparisons.
4. Results when varying the number of comparisons for fixed dataset.
5. More Flyper-parameter ablations, include temperature and pretraining.
6. D4RL real human results.
D.1 STATE LEARNING CURVES
MeraWorld State 2.5K Dense
Picking
Butzon Press
Door Open
0.9
0.33
Sanooss Bate
0.2
0.8
0.25
0.13
0.4
5.4
100030
205030 305630 100003 300303
100000
200000
300350 430000 tarmio
100000 200m3
a00305 400t0n
Sc0mm
Drawer Open
Plate Süde
Sweep Into
0.0
Suicass Raze
9.2
3.5
100030
203030
305030 100003
500303
100000 200000
300000 430000 Tormo
100000 200m3
a00005 400000 scomi
Training Stops
Training Steps
Training Stops
SPT
P-O1
PPO
CPL(BC)
CP).
Figure 3: State-based results in MetaWorld with 2.5K segments and dense comparisons. This plot also shows CPL BC. The dotted vertical line indicates when BC pretraining stops for CPL, SFT, and PPO. We show KL-coefficient of 2 for PPO, which uses 3.84 million extra state-action pairs.
21

---

Published as a conference paper at ICLR 2024
MetaWorld State 20K Sparse
Bin Picking
Burzon Press
Door Open
Suconde Rota
Drawer
Plate Süde
Sweep
0.
Slicass Rata
6.7
5.6
0.5
0.
5.4
Tratning Steps
Training Steps
Training Stops
Figure 4: State-based results in MetaWorld with 20K segments and sparse comparisons. This plot also shows CPL variants. The dotted vertical line is when BC pretraining stops for CPL, SFT, and PPO. We show KL-coefficient of for PPO, which uses 3.8 3.84 million extra state-action pairs.
D.2 IMAGE LEARNING CURVES
MetaWorld Image 2.5K
Bin Picking
Button Press
Door Open
0.35
0.8
0.30
0.6
Sunness Rate
0.25
0.20
6.6
0.15
0.5
0.10
0.4
20
20000
40000
60000
50000
100000 120000
20000
40000
50006
BOO0D 100R00 120000
20000
40006
60000
BOR00
100000 120000
Drawer Open
Plate Slide
Sweep Into
0.8
0.7
Success Hale
0.4
0.5
0.5
0.4
20000
40000
50000 50000 100000 120000
20000 40000
G0000
80000 100000 120000
20000
40000
60000
80000 100000 120000
Training Steps
Training Steps
Training Steps
Figure 5: Image-based based results in MetaWorld with 2.5K segments and dense comparisons. This plot additionally shows the CPL BC variant. The dotted vertical line indicates when BC pretraining stops for CPL and SFT.
22

---

Published as a conference paper at ICLR 2024
MetaWorld Image 20K Sparse
Bin Picking
Button Press
Open
0.35
Stocess Rate
0.30
6.7
0.2
0.25
0.20
6.6
0.5
Drawer Open
Plate Slide
Swoep Into
0.8
Success Rate
0.5
0.7
0.5
0.4
0.5
0.3
0.4
100000
LG0000
200000
50000
200000
150000
200000
Training Steps
Training Steps
Training Steps
SFT
PUQL
CPL
CPL (BC)
Figure 6: Image-based results in Meta World with 20K segments and sparse comparisons. This plot shows CPL variants in addition to baselines. The dotted vertical line indicates when BC pretraining stops for CPL and SFT.
D.3 DATA SCALING
MetaWorld State Dense
Picking
Press
Door Open
0.35
0.90
0.85
Suo0655 Rate
0.30
0.85
3.80
0.80
0.78
0.20
0.75
9.70
0.15
0.70
0.65
50.00
7500
7500
Open
Slide
Sweep
0.85
0.60
0.75
0.80
0.55
0.70
0.65
8.20
0.45
2500
50.00
7500
2500
5000
7506
2500
5000
7506
Dataset Size
Dataset Size
Dutaset Size
-
PIQL
CPL
Figure 7: Scaling on state-based MorlW environments for different sized dense comparison datasets.
23

---

Published as a conference paper at ICLR 2024
MetaWorld State Sparse, 5000 Segments
Picking
Press
Open
0.35
Stocess Rats
0.80
0.30
0.80
0.25
0.80
0.75
0.20
0.75
0.15
0.65
0.10
Drawer Open
Plate Slide
Sweep Into
0.80
0.90
0.85
0.80
0.75
0.55
0.70
0.75
0.70
0.50
Comparisons Segment
Comparisons Segment:
Comparisons Segment:
-0-
P4OLL
-
CPL
Figure 8: Scaling on state-based MetaWorld environments for 5000 segments varying the number of comparisons per segment. We find that for these results, P-IQL's performance sometimes goes down. Inspecting our training logs reveals that this is likely due to the reward function underfitting. For example, Door Open, the reward modeling loss is near zero with only 2 comparisons per segment for). With 16 comparisons (10K comparisons total, which is the amount we tuned our additional limitation of P-IQL: it requires per segment, the loss ends near 0.16. This highlights tuning an addition reward model, which can be very sensitive to its training parameters. CPL removes this additional complexity, and thus easier scale.
D.4 ADDITIONAL ABLATIONS
Segment Abiation
16
20
207
2000
C2L,02)
Figure 9: Results varying the size of segments on Drawer Open from State. The dataset was fixed to 20K segments with 10K comparisons, but the size of each segment varied.
24

---

Published as a conference paper at ICLR 2024
Walker-Med-Exp
Walker-Med-Replay
Hopper-Med-Exp
Hopper-Med-Replan
P-IQL
99.9 1: 6.2
71.6 5.5
88.6 3.6
60.2 20.6
PT
110.2 1 0.8
76.6 3.2
86.7 6.8
78.9 10.3
CPL
109.2 0.2
48.3 3.7
109.1 0.7
72.4 3.1
Table Results on the D4RL benchmark from real human feedback from Kim al. (2023). "PT" denotes the original Preference Transformer baseline which uses the algorithm after learning transformer- based reward model.
MetaWorld State 2.5K Dense
Button
Open
0.8
Sugcess
0.0
0.6
109200 200000 S10020 400000
500000
100000 200000
200000
400009
500000
100020 200000
500000
400002 500209
Drawer Open
Swoap
6.5
0.6
0.5
100000
200000
SC0020
400000 590000
103000
202000
300000 400309
500000
100020
200000
300000 400002 500209
Training Stape
Training Staps
Training Steps
GPL
CPL (No Pretrain)
Figure 10: Results with and without pretraining on MetaWorld state with 2.5K Dense segments. Interestingly, performance can be higher without pretraining.
D.5 D4RL REAL-HUMM PREFERENCE, RESULTS
To demonstrate both the applicability of the regret preference model and performance of CPL, we perform additional experiments with real-human preferences. Specifically, we adopt the benchmarks from Kim et al. (2023) which use either 100 (expert) or 500 (replay) real human preferences on datasets from the D4RL benchmark (Fu et al., 2020). To faciliate learning from such a limited number of queries, we modified CPL by first learning a logistic regression model to predict the users preference Plo
following An al. (2023). Critically, this not reward or advantage function takes entire segments as inputs, not single state-action pairs. We then use this model to relabel the offline D4RL data with dense preferences for CPL. We borrow logistic regression model borrows the architecture from Kim et al. (2023) and do not use BC pretraining. Our results can be found in Table 3. CPL has the best performance by a large margin in Moopper-Medium--a Exp,, but performs worse in Walker-Medium-RRe closely follow the regret-basee based model as per discussions with the authors of Kim et al. (2023), they collected by single user with pre-planned rules-based approach. Nonetheless, CsL still able to perform well on these tasks with only 100 or 500 human preferences, and no value or Q- Q-function learning.
We posit that this is because the preferences for this dataset, may not
25

---

Published as a conference paper at ICLR 2024
Human Feedback D4RL Locomotion Benchmark
WalkerDD Medium Expert
Walker2D Medium Replay
1100
0.8
0.0
0.6
0.6
0.6
Happer Medmum Export
Hopper Medium Replay
1.0
1.0
08
0.6
0.5
0.1
2.0
02
Training Steps
166
Training Steps
PIOL
ProTTransformer
CPL
Figure 11: Results for the D4RL human preference datasets.
E
EXPERIMENT DETAILS.
Our datasets and code are publicly released at htttss://githbbb..... http:/thhab..coommeeeeeeaaaaa/ee
E.
ENVIRONMENT DETAILS
We use modified version of the MetaWorld environments (Yu et al., 2020) in our experiments, which we found necessary to obtain good regret-based preference labels. Meta World was designed for Meta-RL, and thus by default hides the goal from the state spaces. Prior works like Lee et al. (2021), have randomized the goal but left hidden, making the reward function stochastic, We randomize the goal, but make it observable remove reward stochasticity. We additionally randomize the initial position of the arm, which not done by default. This increases data coverage but also leads to more robust policies. Finally, in Meta World v2 the state by default includes object and proprioceptive history. We remove proprioceptive history to make the environment more Markovian.
E.2 DATASETS AND PREFERENCE LABELING
In Section 4 we provided details on how we generated our datasets. Though we tried to select suboptimal SAC checkpoints that achieves approximately 50% success rate, there was some variance. In Table we show the overall success rate of trajectories in the rollout dataset for each environment. We also apply gaussian noise of standard deviation 0.3 when collecting rollouts. Next, we provide further details on how generated accurate regret-based labels.
Env
Bin Picking
Button Press
Door Open
Drawer Open
Plate Slide
Sweep Into
Success Rate
55.6%
15.56%
53.96%
60. 60.12%
34.4%
42.25%
Table Success rate of suboptimal checkpoints used for generating the rollout datasets.
First, we train an Oracle SAC policy to obtain Q* and -. ensure low TD-error on the offline rollout dataset, we add all rollouts to the replay buffer of the SAC model before start training. We then run SAC as usually, collecting online data, but with a sufficiently large replay buffer such that no data rollout data is overridden.
After training the policy, we estimate regret labels for entire segments at time by writing the negated regret in terms of the value function and reward. We find that this lowers variance. Under deterministic
26

---

Published as a conference paper at ICLR 2024
dynamics, it can be shown that:
regret(o)
Σ 7 (()
R-1 Σ 7r((s)a))) 10
(sk) ((s))
We then estimate (s) = Ea (s,a)))))) by evaluating the SAC Q function on 64 MCMC samples from the policy. Accurately estimating the optimal advantage function over the entire sub- optimal rollout dataset was difficult. We found that this procedure lead to the best results in comparison other means estimating the advantage, like directly evaluating Q" (s.a) - V (s), using alogT (a s) per the bijection in maximum entropy RL, or by using MCMC rollouts of the policy from states in segment. In the figure below we show how these options, though theoretically equivalent, do not agree in practice.
Bin Picking
Drawer Open
1.0
1.0
0.9
0.9
Extimation Mozhod
Essimatiin Mezhod
0.71
0.62
0.71
0.8
Agreecnenn
0.73
0.62
0:70
0.8
MCMC
0.73
0.62
0.72
%
MCMC
0.74
0.62
0.63
LogProb
0.71
0.72
LogProb
0.69
Sum
MCMC Estimation Method
Q-V
LogProb
Sum
MCMC Estimation Method
Q-V
LogProb
Plate Slide
Swaeg Into
1.0
1.0
0.61
0.61
0.5
0.9
Estimatton Method
0.67
0.47
0.71
0.8
% Agreerzent
0.70
0.48
0.73
0.8
Fatimattin
MCMC
0.61
0.47
0.50
0.9
MCMG
0.59
0.48
0.44
0.7
0.6
0.6
Logtrob
0.71
Logtrob
Sum
Q-V
MCMC
LegProh
Sum
0-V
MCMC
LogProh
Estimation
Figure 12: Comparing the agreement between different methods of estimating the advantage for generating regret based preferences. We used the sum variant because we found that it usually had the highest agreement across tasks (average of its row).
27

---

Published as a conference paper at ICLR 2024
E.3 EVALUATION
Evaluating CPI. in comparison to other RL baselines can be difficult, CPL uses only supervised learning, while P-IQL is RL based. Superivsed learning methods, like CPL can casily overfit the training data. On the other hand, off-polics RL methods like P-IQL converge to fixed point and thus often take much longer to train before overfitting. While notable works in imitation learning (Mandlekar et al., 2022) have reported the average of the maximum performance of each seed, we find this to be bit overly optimistic to randomness in evaluation episodes and assumes one can evaluate every checkpoint. on the other hand, offline RL works like (Kostrikov et al., 2022), report evaluation after a fixed amount of training, which can lead to overfitting for supervised approaches like CPL. We take a middle-of- ofaele-of-theerooee of-thheroad approach when reporting numbers in Table 1.
Every 5000 steps for state-based experiments and every 2500 steps for image-based experiments we run 25 evaluation episodes. We then average evaluation performance across eight neighboring checkpoints with running average, totaling 200 evaluation episodes. We then average this value across seeds. Finally, we take the maximum point of the average. This exactly corresponds the peak of the learning curves provided for all of our experiments. This evaluation procedure first averages performance over number of checkpoints and episodes, and then averages over seeds. This maximum-of-the--e avearrgee approach mitigates the over-optimism of average-or of--off-hhhe maximmmm evaluation procedures like those used in Mandlckar et al. (2021). At the same time, it assumes that we can reasonably stop training before over- over-fitting begins.
For the D4RL Preference datasets results we follow the same methodology except perform evaluation episodes every 5000 steps following Kim et al. (2023).
E.4 HYPERPARAMETERS
Below we detail hyper- aapeem-rraeterss for all methods. Note that we assumed all policies to be Gaussian. For Metaworld we used fixed variance and thus compute the log probability logr(a|s) for CPL -[[=(s)-al) al13. For the D4RL datasets we use diagonal gaussian distribution with a learned variance for each action dimension. For preference transformer we use the authors original codebase to re-run their results. For CPL in D4RL, we implement the CPL loss function in the codebase from An et al. (2023) which is based on Kim et al. (2023). We run results for P-IQL using the implementation from Hejna & Sadigh (2023), but remove the data augmentation to be consistent with all other methods. We left all other hyper-parameters, except for the temperatures of CPL., the same as the author's implementations.
Hyperparameter
State
Image Sparse
Image Dense
Total Training Steps
500k
200k
120k
Pre-training Steps (-QL))
200k
80k
40k
Batch Size
96
48
32
Segment Size
64
64
64
Actor Dropout Architecture
0.25
0.5
0.5
[512, 512] MLP
DrQv2
DrQv2
Table 5: Common Meta World Hyper-parameter
Batch size refers to the number of comparisons sampled from the dataset. So, a single batch contains 2x batch size x segment size total states. We use the same batch size for all methods, even if they do not need to operate on full segments. Running so many mage-basee experiments is computationally expensive. Thus, for the image based experiments we lowered the batch size. Because our dense datasets had only 2.5K segments versus the 20K in our sparse datasets, we trained for fewer steps. We also lowered the batch size because the comparisons were more dense. Table reports training speed for image results using the sparse datasets. We use the same architecture for all methods. Pre-training steps counted towards the total training step budget, as shown in the learning curves. We found that dropout helped all methods.
used for CPL and variants and SFT. Pre-training
28

---

Published as a conference paper at ICLR 2024
Hyperparameter
CPL
CPL (BC)
CPL (KL)
CPL for D4RL
Learning Rate
0.0001
0.0001
0.0001
0.0001
Temp a
0.1
0.1
0.1
0.2
Bias
0.5
0.75
0.5
BC Weight 3
0.0
1.0
0.0
0.0
LR Schedule
10% after pretraining
7
Table 6:
for CPL. and variants.
Hyperparameter
P-IQL
Expectile 7
0.7
Hyperparameter Learning Rate
SFT
Temperature
0.3333
0.0001
Bias À
0.5
0.99
Reward Net Steps Learning Rate
50k
0.0003 state, 0.0001 image
Table 7: Hyperparameters for P-IQL and SFT for MetaWorld. We closely follow details from Kostrikov et al. (2022) and Hejna & Sadigh (2023) and tune parameters, particularly the number of reward net steps, is crucial for performance. Parameters not listed are left their defaults
Hyperparameter
PPO
Batch Size
128
Collection steps
4096
Epochs per collection
10
learning rate
0.0003
KL reward weight
or
GAE A
0.95
Discount 7
0.99
Policy std-dev Limits Activation
(0.5, 1.5)
Tanh
Table 8: PPO Hyperparameters. Note that we spent lot of time tuning PPO to try and get better results. We found that limited the standard deviation of the policy to be at least 0.5 crucial, as well as bounding the reward values the pretrained reward model to stay between and 1. We left the network architecture the same other baselines, but used Tanh activation since it is suggested for PPO.
29


Reviews:
[{"review_id": "x4BhIdZRej", "reviewer": "Reviewer_NpZJ", "summary": "This paper presents Contrastive Preference Learning (CPL), an algorithm to learn optimal policies from preferences without learning exlicitly a reward function which is commonly done in RLHF scenario. This circumvents the issue of having an underoptimized/overoptimized reward model. The authors then show the performance of CPL for MetaWorld benchmark.", "questions": "How does CPL compare with RLHF for language models scenario?\nHow does CPL compare with other baselines, which may or may not have reward models", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. learning a reward model from human preferences has flaws. And on top of that, using RL to optimize for this reward model can sometimes lead to poor performance. This paper solves this issue by not having a reward model.\n2. CPL has supervised objectives so it is scalable \n3. The proposed algorithm is generic", "weaknesses": "1. In my experience, learning a \"good\" reward model and then doing RL always outperforms offline RL algorithms. The authors only compare it with IQL and not with methods with reward models to highlight more\n2. The authors claim that the method is generic but then it is only applied to MetaWorld benchmark. The RLHF scenario is much more interesting in aligning language models with human feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IAbOywqKNV", "reviewer": "Reviewer_Vhgt", "summary": "This paper presents a Contrastive Preference Learning (CPL) framework for learning optimal policies from human preference data without learning the reward function. Specifically, the paper models human preferences using the advantage function and proposes a general loss function for learning policies. The loss function ensembles the contrastive learning objective and can be optimized directly without learning a reward function. As a result, the method can scale to high-dimensional environments and sequential RLHF problems (i.e., beyond contextual bandits). Theoretically, by optimizing the loss function, CPL provably converges to the optimal policy of the underlying max-entropy RL problem. The paper tests one instantiation of the CPL framework and shows its promising performance in practice.", "questions": "Since the CPL loss function has a super elegant form, is it possible to derive finite sample analysis for learning a near-optimal policy like [1]?\n\n[1] Zhu et al., Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons, ICML 2023", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The proposed algorithmic framework is novel and elegant. The motivation for the problem is clear.\n\n- The method is scalable without the use of RL.\n\n- The experimental results are adequate.\n\n- The paper is very well-written and easy to follow.", "weaknesses": "I did not identify any noticeable weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "47eN47dQCf", "reviewer": "Reviewer_moLC", "summary": "The authors address critical aspects of the PBRL framework, with a specific emphasis on the optimization challenges in the RL phase. To solve this problem, the authors introduce a novel approach called Contrative Preference Learning (CPL). This method leverages a regret-based model of human preferences, from which a contrastive objective is derived with the principle of maximum entropy. This approach bypasses the need for reward learning and RL, instead directly learn the policy through a supervised learning paradigm. To evaluate the effectiveness of CPL, the authors conducted experiments within the offline PbRL setting, comparing it against strong baselines in terms of the success rate across distinct tasks in the Metaworld domain. The experimental results show that CPL outperforms baselines with less runtime and smaller model size. The primary contribution of this work lies in the conversion of the traditional two-phase PbRL framework into a novel paradigm capable of directly learning the policy with a new contrastive objective.", "questions": "1. I still have questions regarding regret-based preference model. I agree with the authors that the regret-based preference model makes more sense when we consider the hand-engineered example in section 2. However, when we talk about data collection with a real human, the human labeler would have a preference over two trajectory segments. This implies the existence of underlying dense rewards that explain the human's preferences. In such cases, I feel that the key issue lies in the hand-engineered reward is incorrect (i.e., reward design issue) in your example, rather than in the issues of the reward-based preference model.\n\nTherefore, when we consider experiments with real humans and apply the reward-based preference model, could it also perform effectively? Is it possible that the learned reward captures the regret information to a large extent? Please correct me if I have misunderstood.\n\n2. Despite considering the model complexity of CPL, the results are promising. In terms of feedback efficiency, does CPL require more human preference data compared to the conventional two-phase PbRL framework in order to perform well? This is especially relevant considering the Metaworld tasks in the experiments, where obtaining dense data could be challenging if collected from real humans.\n\n3. In the experiments, the authors pretrain the model with behavior cloning. To what extent does this pretraining phase impact the model's final performance? Does P-IQL also have this pretraining phase?\n\n4. Similar to DPO, CPL employs a supervised learning framework without reward learning and RL. Does it potentially lose the generalization power of RL?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This work focuses on addressing critical challenges in PbRL. It is well-motivated and accompanied by a clear and thorough discussion of existing issues within both the reward learning and RL phases.\n\nThe proposed CPL bypasses the need for reward learning and RL by optimizing a supervised objective, enabling it to learn policy from offline high-dimensional suboptimal data. Moreover, it can be applied to arbitrary MDPs. I feel this approach can be seen as a counterpart to DPO, as discussed by the authors in the paper—one for NLP tasks with LLMs and the other for continuous control tasks. This work has the potential to make a significant impact in the community, and I am eager to see how CPL performs in broader applications.\n\nGenerally, the organization and presentation of the content are well-structured, facilitating ease of reading and comprehension. The authors provide comprehensive theoretical proofs that make the work sound. The experimental results are impressive in terms of runtime, model size, and performance. In the limitation section, I appreciate the authors acknowledge the imperfections of the human model and raise considerations regarding the application of this approach to online human feedback.", "weaknesses": "Please see Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Io7tbPciJT", "reviewer": "Reviewer_WpDS", "summary": "This work introduces Contrastive Preference Learning (CPL), a novel algorithm designed for learning optimal policies from preferences, eliminating the need to learn reward functions. CPL integrates the regret-based preference framework with the principle of Maximum Entropy, establishing a one-to-one correspondence between advantage functions and policies.\nThe experimental results highlight CPL's superior performance compared to SFT and offline Reinforcement Learning (P-IQL).", "questions": "What if you were to employ an online RL algorithm for the reinforcement learning experiment instead of an offline one?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The motivation is evidently well-defined.\n2. It adeptly combines theoretical analysis with empirical findings.\n3. The proposed method is written in a clear and easily understandable manner.", "weaknesses": "This article exclusively compares CQL with offline RL, but to my knowledge, the majority of RLHF (Reinforcement Learning from Human Feedback) algorithms employ **online** RL algorithms [1]. There appears to be a fundamental distinction between these two training paradigms. Offline algorithms exclusively train the model on static datasets, whereas online algorithms train the model on the trajectories gathered by the training policies. \n\nI strongly encourage the authors to include a baseline that trains the reward model using the dataset and subsequently employs an **online** training methodology, such as PPO. This addition is crucial to substantiate the authors' claims.\n\n\n[1] Training language models to follow instructions with human feedback.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)