Title: Direct Preference-based Policy Optimization without Reward Modeling
Venue: NeurIPS 2023
Accepted: yes
Keywords: Preference-based reinforcement learning

Contrastive learning

Offline reinforcement learning

RLHF
Abstract: Preference-based reinforcement learning (PbRL) is an approach that enables RL agents to learn from preference, which is particularly useful when formulating a reward function is challenging. Existing PbRL methods generally involve a two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference is from human teachers, can be difficult. Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design a novel policy scoring metric that assigns a high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms or is on par with the existing PbRL methods. Notably, on high-dimensional control tasks, our algorithm surpasses offline RL methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.

Direct Preference-based Policy Optimization without Reward Modeling

Abstract

Preference-based reinforcement learning (PbRL) is an approach that enables RL agents to learn from preference, which is particularly useful when formulating a reward function is challenging. Existing PbRL methods generally involve a two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference is from human teachers, can be difficult. Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design a novel policy scoring metric that assigns a high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms or is on par with the existing PbRL methods. Notably, on high-dimensional control tasks, our algorithm surpasses offline RL methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.

1 Introduction

Deep reinforcement learning has been successful in solving various decision-making tasks where a well-defined reward function is available [34, 35, 50, 5, 54]. However, in many real-world tasks, it can be challenging to design a quantitative reward function that accurately reflects the desired behavior, particularly when the task involves human evaluation. Preference-based RL (PbRL) seeks to provide an alternative solution by enabling agents to learn from preference information between pairs of trajectory segments [1, 10]. PbRL has gained considerable interest in recent years as making a relative judgment is much easier than providing a real-valued score, which makes human feedback much more viable [39, 38, 52, 45, 56].

Junhyeok Lee ∗

Norio Kosaka NAVER

Hyun Oh Song †

Preference

dataset

Unlabeled

dataset

Preference-reward

modeling

Preference

model

Unlabeled

dataset

DPPO (Ours)

Agent

Preference

dataset

Reward

model

Preference

model

Reward-based

RL

Agent

1.5

Predicted Rewards

1.0

0.5

0.0

-0.5

2

4

6

True Rewards

Recent PbRL methods take a two-step approach: they first learn a reward model from the given preference dataset and then run off-the-shelf reinforcement learning algorithms on top of the learned reward model [10, 31, 40]. However, acquiring an accurate reward model only from preference labels, typically provided by human teachers, poses a significant challenge as it is unclear how to extract the underlying reward structure from preference. Current methods rely on modeling the reward with certain specific assumptions, though there are some concerns regarding whether those assumptions hold in practice [13, 27].

Alternatively, predicting the preference itself is comparatively more straightforward since we have direct access to training labels, allowing us to leverage powerful techniques from supervised learning. Building upon this observation, we introduce a PbRL algorithm that bypasses the need for reward function modeling by directly learning from preference labels. Our approach begins by devising a policy scoring metric that assigns high scores to policies aligning with the provided preference dataset. Concretely, the PbRL objective is formulated as a contrastive learning problem, guiding the learned policy to be closer to more preferred trajectory segments while distancing itself from the less preferred ones [9, 20]. Furthermore, we enhance the performance of the preference predictors from previous works by introducing a novel prediction smoothness regularizer. Experiment results on offline RL settings with actual human preference labels show that the proposed algorithm outperforms or is on par with the baselines on all of the tasks considered [16]. Notably, in high-dimensional control tasks, our algorithm outperforms offline RL methods that utilize ground-truth reward information. Moreover, our preliminary experiments show that our algorithm can be successfully applied for fine-tuning large language models. Our official code is available at https://github.com/snu-mllab/DPPO .

2 Preliminaries

2.1 Preference-based reinforcement learning

Reinforcement learning considers an environment formulated as a Markov Decision Process (MDP) defined by a tuple ( S , A , T, R , p 0 , H ) , where S is a state space, A is an action space, T ( s ′ | s , a ) is the state transition dynamics, R ( s , a ) is the reward function, p 0 ( s ) is the initial state distribution, and H is the time horizon. The goal of reinforcement learning is to learn a policy π that optimizes the expected return:

Conventional RL assumes the reward information ( r t ) is given and uses this to optimize their policy. However, finding a suitable reward metric can be costly in many real-world scenarios.

For example, if the goal task is to train a robot to scramble an egg, it would be unclear how to design a reward function that captures all the desired properties. Instead, PbRL assumes the supervision is given in the form of preference. Concretely, for a pair of trajectory segments ( σ 0 , σ 1 ) , where σ i = ( s i 0 , a i 0 , s i 1 , a i 1 , . . . , s i k , a i k ) , a (human) teacher indicates which segment it prefers. The preference label y is given as 0 if σ 0 is preferred, i.e. , σ 0 ≻ σ 1 . In a similar manner, y is set to 1 if σ 1 is preferred and 0 . 5 if the two are equally preferred.

To learn a reward model ̂ r , prior works assume the preference depends on the value of the underlying rewards summed over each timestep [10, 6, 31, 40]:

where ψ denotes the learnable parameters of the reward model. Given a dataset D pref of preference triples ( σ 0 , σ 1 , y ) , the reward model is trained by minimizing the cross-entropy loss between the preference predictions and the ground-truth labels:

After training, any standard RL algorithm can be used to maximize the expected return under the learned reward model. Especially in the offline PbRL setting, we assume there exists a small dataset D pref with preference labels along with a much larger unlabeled dataset D without any reward or preference labels [27, 49]. A typical approach involves utilizing D pref to learn the reward model and applying the model to label D .

2.2 Contrastive learning

The goal of contrastive learning is to learn representations where similar sample pairs are close to each other while dissimilar pairs are far apart. For an anchor sample x ( e.g. , an image), suppose we have a positive sample x + ( e.g. , the same image with data augmentation applied) and a set of negative samples { x -i } i m =1 ( e.g. , samples from other images). To learn an encoder f , contrastive learning minimizes the following loss:

where a dot product of two representations ( e.g. , f ( x ) ⊺ f ( x + ) ) is considered as the similarity score between the two samples.

3 Learning directly from preference

While prior PbRL approaches adopt a two-step procedure involving the construction of a reward model using the preference data, the fidelity of the learned reward model in reflecting the original reward function remains uncertain. The main challenge lies in extracting the underlying rewards from the given preference. Previous works assume that the preference for a trajectory segment can be represented as an average of the underlying rewards [10, 21]. However, considering PbRL typically assumes human teachers provide the preference labels, it is unclear whether human preferences can also be modeled in this manner. For instance, humans may focus on a specific subset of the segment while ignoring other parts [24, 27]. Empirically, we find that the reward models learned from preferences often fail to accurately capture the underlying reward structure, as illustrated in Figure 2. This issue is problematic since within the current PBRL framework, the quality of the learned policy relies heavily on the quality of the learned rewards.

Meanwhile, predicting the preference itself is a more straightforward task as we have explicit labels to train with and can therefore apply powerful tools from supervised learning [18, 11, 36]. Based on this observation, our goal is to introduce a PbRL algorithm that directly learns with preference information without the need for reward modeling, as illustrated in Figure 1. To achieve this, we design a policy scoring metric that yields a high score when a policy aligns with the given preference dataset. In other words, we assume that a desirable policy should be closer to σ 0 than σ 1 if σ 0 ≻ σ 1 . To formulate this property into an optimizable objective, we first define the closeness between a policy and a trajectory segment.

Policy-Transition Tuple Distances

Policy-Segment Distance

(1)

(2)

(3)

(4)

.0

E

a~πTe(-1s)

[1la - a l12]

E

a~πe(-1s)

[la - a²l1]

Ea~πe(1sg)

[1la - a% 1]

d =d(π,00)

exp(-d°)

s = log

exp(-d°)

exp(-d)+exp(-入d)

一

90

人

91

3.1 Policy-segment distance

We define the distance between a policy and a trajectory segment as an aggregation of the distance between a policy and each transition tuple in the trajectory segment. Concretely, the policy-segment distance can be expressed as

where d sa denotes the policy-transition tuple distance and AGG denotes an aggregation function. There can be multiple ways for instantiating d sa . For simplicity, we employ the expected ℓ 2 distance between the policy action and the trajectory action: d sa ( π, s , a ) = E ˜ a ∼ π ( ·| s ) [ ∥ ˜ a -a ∥ 2 ] . Similarly, we opt to use the mean operator as the aggregation function. To sum up, our policy-segment distance function becomes

3.2 Preference score metric

Using the policy-segment distance defined above, we can now build a score metric that satisfies our desired property. Given a preference triple ( σ 0 , σ 1 , y ) , assume that σ 0 is preferred over σ 1 , i.e. , y =0 . We want to assign a high score if a policy is closer to σ 0 than σ 1 . This condition can be expressed in terms of the policy-segment distance as σ 0 ≻ σ 1 ⇒ d ( π, σ 0 ) < d ( π, σ 1 ) . To capture this condition across multiple segment pairs into a single metric, we adopt a contrastive learning formulation:

where θ denotes the learnable parameters of the policy π . In terms of contrastive learning, we can interpret exp ( -d ( π θ , σ i )) as representing the similarity between the policy π θ and the segment σ i . Then, π θ , σ 0 , and σ 1 are each considered the anchor, the positive sample, and the negative sample, assuming σ 0 ≻ σ 1 . Note that a high score can only be achieved if the policy is closer to more preferred trajectory segments.

While Equation (2) is sufficient to express our desired property, a minor drawback is that the score function is indifferent to the increase or decrease of the distances in the same magnitude. To understand this in detail, let us first denote d i = d ( π θ , σ i ) for brevity. Then, for a preference triple ( σ 0 , σ 1 , y ) with y = 0 ,

0

25

50

75

100

125

150

175

200

Segment index (

i

)

0.0

0.2

0.4

0.6

0.8

1.0

Preference Prediction

Not smoothed

Smoothed

Therefore, the score value remains the same when the distances increase or decrease in the same magnitude. In other words, the score for the distances ( d 0 , d 1 ) and ( d 0 + α, d 1 + α ) are identical, indicating there is no penalty when a policy deviates from even the preferred trajectory segment. To solve this, we add a regularizing factor λ ∈ (0 , 1) that decreases the score when the overall scale of the policy-segment distances increases:

If we set λ smaller than 1 and plug in this new formulation to Equation (3), it is easy to find out that the score will decrease when the overall distances increase. The resulting score calculation process for Equation (4) is illustrated in Figure 3.

3.3 Policy optimization with preference predictor

We can directly optimize a policy with preference labels by maximizing the score function in Equation (4). However, to leverage the unlabeled dataset D , we train a preference predictor using the labeled dataset D pref. We formulate this process as a simple binary classification problem and use the cross-entropy loss to optimize the predictor ̂ P :

where ϕ denotes the learnable parameters of the preference predictor ̂ P . The first term resembles Equation (1), but the difference is that here we directly model the preference predictor instead of modeling the reward function. The second term is a smoothness regularizer which guides the predictor to have a similar preference against two largely overlapping segments. Concretely, given σ = ( s i , a i , . . . , s i + k , a i + k ) , we sample σ ′ from the same trajectory as ( s i + α , a i + α , . . . , s i + α + k , a i + α + k ) , where α ∼ N (0 , m 2 ) with m ≪ k . We observe that if no smoothness regularization is applied ( ν = 0 ), the preference can vary significantly between two almost identical segments, as illustrated in Figure 4. This behavior is undesirable as a human teacher will unlikely be able to even identify the difference between the two segments.

Algorithm 1 Direct Preference-based Policy Optimization

Input: Unlabeled dataset D , preference dataset D pref, learning rate η ϕ and η θ , number of training steps M and N , and regularization parameters λ , ν .

Initialize network parameters ϕ and θ

for step

= 1

to

M

do

Update the predictor parameter:

end for for

step

= 1

to

N

do

Update the policy parameter:

θ

θ

+

η

θ

∇

θ

S

(

θ

;

D

, ϕ, λ

end for

)

After training the preference predictor with Equation (5), we can train our policy with the unlabeled dataset D by sampling pairs of trajectory segments and labeling their preferences:

✶ Algorithm 1 summarizes the full process of our PbRL algorithm. Figure 5 shows an example of two policies each learned using our algorithm and behavior cloning (BC), which shows that vanilla BC suffers from imitating the behavior of unpreferred trajectory segments while our algorithm successfully distances from it. We name our preference-based policy optimization algorithm as DPPO, an abbreviation for Direct Preference-based Policy Optimization . The sections below evaluate the performance of DPPO in the offline setting.

4 Experiments

4.1 Offline RL experiment details

Following recent PbRL works, we evaluate our algorithm on the offline setting which assumes a large unlabeled dataset D is given along with a much smaller preference-labeled dataset D pref [49, 27]. We evaluate our algorithm on D4RL, a standard benchmark for offline RL, with preference datasets generated by actual human teachers [16]. We did not experiment on Antmaze, a widely used task in D4RL, due to a crucial bug in its environment implementation (please refer to Appendix F for more details). Below is a brief description of the tasks considered in our experiments:

Gym D4RL Gym provides datasets for the Gym locomotion environments where the goal task is to move forward while maintaining body balance. D4RL Gym contains various types of datasets each from a different data collection process. Among those datasets, we focus on the medium-replay and medium-expert datasets, which contain trajectories with the most diverse quality.

Adroit pen Adroit tasks involve controlling a 24-DoF robotic hand to perform tasks such as grabbing a pen or hammering a nail [44]. Adroit tests if offline RL methods can learn from human demonstrations on these high-dimensional tasks. We focus on the pen task since standard offline RL algorithms fail to achieve reasonable performance on other tasks, even with the reward information.

Kitchen Kitchen requires learning a 9-DoF robot manipulation task [19]. Similar to Adroit, the reward function is sparse and the dataset contains human demonstrations. However, Kitchen requires solving multiple sub-tasks sequentially ( e.g. , opening the microwave, then turning off the switch) in a single episode. While Kitchen has three types of datasets ( complete , partial , and mixed ), we consider the latter two as complete only contains successful trajectories with almost identical quality.

For the Gym hopper, Gym walker2d, and Adroit pen tasks, we utilize publicly available human preference datasets released by [27]. For the other tasks, we generate a new preference dataset as there are no preference datasets available. To collect the human preference datasets for each task, we strictly adhere to the protocol outlined by [27]. This involves defining a set of desirable behaviors and instructing the human teacher to label preferences accordingly. For example, a desired behavior in the Hopper environment would be to maintain body balance and prevent falling down. The size of the resulting preference datasets ranges from 100 to 500 samples, depending on the specific task. For more details regarding the dataset generation process, please refer to Appendix A.

We consider PreferenceTransformer (PT) as our baseline method, which is a state-of-the-art approach in offline PbRL [27]. PT employs a transformer network to train the reward model and leverages IQL, an offline RL algorithm, for policy optimization using the learned rewards [28]. This original version is denoted as PT+IQL. Considering that the choice of policy optimization algorithm significantly impacts the performance of reward modeling methods, we also explore using CQL for policy optimization [30]. This modified version of PT is denoted as PT+CQL. Additionally, we present the performance of CQL and IQL when utilizing the ground-truth reward information for reference. Note that these reward-based RL baselines do not provide a fair comparison with our method as they have access to a much denser supervision signal. For more implementation details regarding the baselines and our algorithm, please refer to Appendix B. For more experiments including behavior-cloning baselines, please refer to Appendix C.

4.2 Evaluation results

Table 1 shows the evaluation results for the D4RL Gym tasks. DPPO demonstrates superior or comparable performance to the preference-based learning methods across all considered tasks. In terms of average performance, our method outperforms the baselines by a large margin with a minimum of %11p and reaches a performance level similar to the methods that learn with groundtruth rewards. Also, DPPO exhibits significantly lower variance in performance compared to the baseline methods like PT+IQL, which suffer from pronounced fluctuations in performance.

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

0

20

40

60

80

Average Return

=0.25

=0.50

=0.75

=1.00

(a)

λ

0.0

0.2

0.4

0.6

0.8

1.0

Step

1e6

0

20

40

60

80

Average Return

=0.0

=0.5

=1.0

=2.0

(b)

ν

Table 2 shows the results on the more challenging D4RL Adroit pen and Kitchen tasks. Once again, DPPO outperforms all the baselines by a large margin. Especially, the performance gap is the largest in the Adroit pen tasks, where DPPO even surpasses the methods that learn with ground-truth rewards. A major distinguishing factor between the Adroit pen and the other environments is the dimensionality of the action space, which amounts to 24, significantly larger than the action spaces ranging from 3 to 9 in other environments. We conjecture that the value learning approach of the baseline methods struggles to scale up to the high-dimensional action spaces of Adroit. Furthermore, DPPO outperforms the baselines on the Kitchen tasks as well, underscoring the scalability of our method to tackle more complex tasks.

4.3 Ablation studies

We assess the importance of the two crucial components of DPPO, which are the conservativeness regularizer λ and the smoothness regularizer ν . The ablation results are presented in Figure 6. The results show that both components are crucial for achieving high performance. Moreover, the experiments demonstrate that DPPO exhibits a considerable degree of robustness to variations in the strength of the regularizations. Since our smoothness regularizer can also be easily applied to reward-modeling baselines, we evaluate how applying this regularizer to the baselines affects the overall performance in Appendix D. While the smoothness regularizer does improve the performance of the baselines, the improved performance falls behind DPPO.

4.4 Effect of dataset size

100

200

300

400

500

Dataset Size

10

20

30

40

50

60

70

80

Average Return

DPPO (Ours)

PT+IQL

(a) hopper-medium-replay-v2

10

20

30

40

50

60

70

80

90

100

Dataset Size

30

40

50

60

70

80

90

100

110

Average Return

DPPO (Ours)

PT+IQL

(b) hopper-medium-expert-v2

Previous works primarily focus on evaluating their algorithms using a fixed preference dataset for each task [27, 13]. In this section, we evaluate how the size of the preference dataset impacts the overall performance of PbRL algorithms. Concretely, we vary the size of the preference dataset in the hoppermedium-replay and hopper-medium-expert tasks and measure the performance of PbRL methods on each dataset size. The results are shown in Figure 7. We observe that DPPO consistently outperforms the baseline method on all dataset sizes, displaying higher average performance. Interestingly, on hopper-medium-replay, we find that the baseline method PT+IQL falls into a unique failure mode when the dataset size is 200, which is to stand still and not move forward.

4.5 Experiments with scripted teachers

Following prior works [10, 31, 32], we additionally evaluate our algorithm using preference labels generated by scripted teachers. A scripted teacher is a synthetic teacher who adheres strictly to the task rewards when making decisions: σ 0 ≻ σ 1 ⇔ ∑ k t =0 r 0 t > ∑ k t =0 r 1 t . It is important to note that the scripted teacher setting is not as realistic as the human teacher setting, and we include this section for reference purposes. The results in Figure 8 show that our algorithm continues to outperform the baselines in terms of aver-

Gym

pen

Kitchen

0

20

40

60

80

100

Average Return

PT+CQL

PT+IQL

DPPO (Ours)

age performance, even in this synthetic setting. In detail, the results for DPPO and PT+IQL resemble the results from the human teacher setting. However, the performance of PT+CQL has dropped significantly compared to the human teacher setting. This disparity of performance reassures the observations from [27] that preferences from human teachers and scripted teachers have different characteristics and should not be treated interchangeably.

4.6 Fine-tuning LLMs with DPPO

Recent works show that PbRL can be used to fine-tune large language models with human preference, which is commonly termed RLHF (reinforcement learning from human feedback) [52, 39, 4]. In RLHF, a reward model is trained to predict human preference between two output texts, and then the learned reward model is employed to fine-tune the language model through off-the-shelf RL algorithms such as PPO. As from offline RL, we can fine-tune a language model directly with the preference predictions by replacing PPO with DPPO. This replacement allows removing unnecessary assumptions required to run reward-based policy optimization techniques like PPO, for example assuming the reward is given only at the end of the output sequence. To empirically examine if DPPO can be used to fine-tune large language models, we conducted some preliminary experiments using human preference datasets to train the preference models and human evaluation to evaluate the tuned language model. We refer to Appendix E for more details regarding the implementation details and the experimental setup.

The results in Table 3 show that DPPO can be successfully used to fine-tune large language models with real human preference, achieving performance comparable to the reward-based RLHF method. We believe that this result is promising as it shows that DPPO can be used to fine-tune large language models without the need for reward modeling.

5 Related works

Preference-based reinforcement learning A long line of works has studied learning agents from human preference [1, 41, 12, 3, 47]. Notably, [10] showed that we can scale PbRL by learning a reward model from preference and applying it to off-the-shelf RL algorithms. Several follow-up works have been proposed building upon this work. For example, a work merges the PbRL framework with imitation learning by introducing other types of supervision such as demonstrations [21]. Other works utilize techniques from semi-supervised learning or data augmentation to improve sample efficiency [31, 40]. Another line of work aims to improve the reward model by removing Markovian assumptions [13, 27]. Recent works have shown that PbRL can greatly boost the performance of large-scale language models by finetuning them with human feedback [52, 56, 37, 39, 45, 38]. Similar to our work, [55] leverages the preference information to directly optimize the policy using the concept of trajectory distance. However, their method requires the two trajectories to start from the same initial state and to roll out from the current policy being trained, which makes it challenging to utilize extensive pre-collected data. Concurrent to our work, [25, 43] also explore the idea of directly optimizing the policy with preference information.

Offline reinforcement learning Offline reinforcement learning methods adopt various techniques to bias the learned policy towards the given offline dataset. For example, some works directly regularize the policy to prefer the actions from the dataset [29, 57, 17]. Another line of works implicitly biases the policy by regularizing the value function [30, 2, 58]. Some works instead start from a SARSA-style TD-update algorithm to avoid querying values for out-of-distribution actions [7, 28]. A more recent line of work casts the offline RL problem as a sequence modeling problem, where a model is learned to generate actions conditional to the given target return [8, 22, 33, 46, 14]. This approach soon gained popularity in the literature due to its high stability and scalability compared to the more traditional value-based approaches [33, 46].

Contrastive learning A core design choice for contrastive learning is to define the positive and negative sample pairs. An earlier work utilizes the structure of the data and considers two different patches from a single sample as a positive pair [53]. The most popular approach is to apply a heavy data augmentation to a single sample repeatedly to produce positive pairs [9, 20]. The success of these methods in unsupervised vision representation learning has inspired many follow-up works, such as application to supervised learning or extension to other domains such as NLP or graph [26, 42, 59].

In the RL domain, there also have been works that leverage contrastive learning to learn unsupervised representations tailored for RL [51]. Also, a recent work directly casts the contrastive learning framework as a goal-conditioned RL problem [15].

6 Discussion

Our proposed PbRL algorithm enables agents to learn directly from preference signals, removing the need for reward modeling. We achieve this by formulating a new policy optimization problem under the contrastive learning framework. Thorough empirical evaluations on various offline PbRL tasks with actual human rewards show our method outperforms the baselines in most of the tasks considered. Interestingly, our algorithm shows better scalability to high-dimensional control tasks when compared to other RL baselines, including those that learn with ground-truth reward information. Additionally, our algorithm demonstrates better data efficiency. These results show that directly utilizing preference signals without reward modeling is a promising direction for PbRL.

A limitation of our work is that label noise, which is likely to exist for human evaluations, is not modeled in our algorithm. Investigating the effect of label noise stemming from human teachers would be an interesting direction for future research [23]. Also, our current algorithm does not incorporate the prediction confidence information, which was excluded as the empirical performance is already strong compared to the baselines. How to appropriately incorporate the prediction confidence would also be a meaningful research direction.

References

Riad Akrour, Marc Schoenauer, and Michele Sebag. Preference-based policy learning. In Machine Learning and Knowledge Discovery in Databases , 2011.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In NeurIPS , 2021.

Layla El Asri, Bilal Piot, Matthieu Geist, Romain Laroche, and Olivier Pietquin. Score-based inverse reinforcement learning. In AAMAS , 2016.

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d.O. Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 1952, 1952.

David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. In NeurIPS , 2021.

Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurIPS , 2021.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML , 2020.

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurIPS , 2017.

David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological) , 20(2):215-232, 1958.

Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acquisition function. Autonomous Robots , 2015.

Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-markovian reward modelling from trajectory labels via interpretable multiple instance learning. In NeurIPS , 2022.

Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In ICLR , 2022.

Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. In NeurIPS , 2022.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML , 2019.

I. J. Good. Rational decisions. Journal of the Royal Statistical Society. Series B (Methodological) , 14(1):107-114, 1952.

Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv:1910.11956 , 2019.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR , 2020.

Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In NeurIPS , 2018.

Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In NeurIPS , 2021.

Anca Dragan Joey Hong, Kush Bhatia. On the sensitivity of reward inference to misspecified human models. In ICLR , 2023.

Daniel Kahneman. Evaluation by moments: Past and future. Choices, values and frames , pages 693-708, 2000.

Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline preference-guided policy optimization. In ICML , 2023.

Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS , 2020.

Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. In ICLR , 2023.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In ICLR , 2022.

Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurIPS , 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurIPS , 2020.

Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In ICML , 2021.

Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preferencebased reinforcement learning. In NeurIPS Datasets and Benchmarks Track , 2021.

Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-game decision transformers. In NeurIPS , 2022.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518, 2015.

Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICML , 2016.

Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In NeurIPS , 2019.

Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 , 2021.

OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 , 2023.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155 , 2022.

Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In ICLR , 2022.

Patrick M. Pilarski, Michael R. Dawson, Thomas Degris, Farbod Fahimi, Jason P. Carey, and Richard S. Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In IEEE International Conference on Rehabilitation Robotics , 2011.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurIPS , 2023.

Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In RSS , 2018.

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In ICLR , 2023.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. In TMLR , 2022.

Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems , 2017.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Daniel Shin, Daniel S. Brown, and Anca D. Dragan. Offline preference-based apprenticeship learning. In ICML Workshop on Human-AI Collaboration in Sequential Decision-Making , 2021.

David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature , 529, 2016.

Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In ICML , 2020.

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS , 2020.

Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 , 2018.

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature , 575, 2019.

Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In NeurIPS , 2012.

Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 , 2021.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In ICML , 2021.

Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurIPS , 2020.

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068 , 2022.

A Human preference datasets

For the D4RL Gym hopper, walker2d, and Adroit pen tasks, we use the human preference datasets provided by [27]. Each dataset consists of 100 preference triples, except for the Gym ∗ -mediumreplay datasets which contain 500 preference triples 3 . Each trajectory segment is of length 100 ( i.e. , k=100). For the other tasks such as Gym halfcheetah and Kitchen, we could not find publicly available human preference datasets. Therefore, we generate new preference datasets with human evaluation by strictly following the procedure from [27]. The preference labeling process consists of two steps: sampling a pair of trajectory segments from the dataset and labeling their preference under some predefined guidelines. For the first step, [27] does not specify how the segments were sampled, so we first randomly sample two trajectories from the offline dataset and then randomly choose a segment from each of the two trajectories to form a segment pair. We provide more details of the labeling process for each task below.

A.1 Gym HalfCheetah

Following the experiment settings from [27], we sample 500 and 100 segment pairs from halfcheetahmedium-replay and halfcheetah-medium-expert datasets, respectively. We assign the human teachers (the authors) to make preference decisions under the following guidelines:

The primary goal for the agent is to move forward as far as possible.

If the two agents both satisfy the primary goal similarly, the agent that has a more stable posture is preferred. In other words, prefer the agent that staggers less.

A.2 Franka Kitchen

Franka Kitchen environment provides six available sub-tasks, where the goal is to perform four of them in a designated order: open the microwave oven, move the kettle to the top left burner, turn on the light switch, and open the slide cabinet. The other two tasks, which are turning on the burner switch and opening the hinge cabinet, are dummy tasks without any positive reward. Figure 9 shows an example of a goal-related task and a dummy task. The robot on the left is moving a kettle, which is a goal-related task. On the other hand, the robot on the right is turning on a burner, which is a dummy task.

For each offline dataset (kitchen-partial and kitchen-mixed), we sample 100 pairs of trajectory segments and label them under the following guidelines:

An agent that solves more goal-related tasks (in order) is preferred.

If the two agents solved the same number of goal-related tasks, the agent that solved fewer dummy tasks is preferred.

If the two agents solved the same number of goal-related and dummy tasks, the agent that was working on another goal-related task is preferred.

B Offline RL experiment details

For each method, we evaluate its performance by collecting ten trajectories with online interaction and measuring the average return. Following the standard protocol of the offline RL literature, we rescale the average return by 100 · R -R random R expert -R random , where R denotes the (empirical) average return of the policy, and R random and R expert are the expected returns of a random policy and an online expert policy, respectively. We provide the implementation details for each PbRL method below.

B.1 PreferenceTransformer (PT)

We use the official implementation from the authors to train the reward model 4 . Then, for PT+IQL, we run IQL with the default hyperparameters for each environment using the official code 5 . Similarly, for PT+CQL, we run CQL with the default hyperparameters for each environment using the official code 6 . Table 4 and Table 5 lists the detailed hyperparameter settings for IQL and CQL.

B.2 DPPO (Ours)

For the preference predictor, we start from a GPT-2 based transformer architecture from PT and apply some modifications. First, we removed the preference attention layer proposed by PT. Also, since PT requires per-step reward predictions, their transformer output is restricted to a scalar value for each

timestep. Instead, our preference model outputs a vector embedding per step, which is aggregated and forwarded into a final MLP layer for the preference score prediction. We train the predictor for 10,000 update steps on the preference dataset. Table 6 lists more details of the training of the preference predictor. For policy optimization, we use a 2-layer MLP architecture and update the policy for 1e6 update steps, following the experimental protocol of IQL. Unlike CQL and IQL which adopt a stochastic policy model, we use a deterministic policy model as we find that both models have no difference in empirical performance. More detailed settings for the policy optimization procedure are specified in Table 7.

C Offline RL results with more baselines

Our offline RL experiments in the main paper focus on baselines that apply value-based RL algorithms after modeling the reward, following [27]. However, another interesting line of work in offline RL is the behavior cloning-based RL algorithms, which typically learn a policy conditional to a target return or goal [8, 14]. Here, we provide the full experiment results including two behavior cloning-based baselines: %BC, which performs behavior cloning on trajectories with top-10% return values from the offline dataset, and RvS [14].

The results in Table 8 show that DPPO outperforms the behavior cloning-based baselines on all tasks, with the performance gap being especially large on the challenging pen tasks.

D Additional ablation on the smoothness regularizer

In recognition of the fact that our novel smoothness regularizer can also be integrated into any reward-modeling baselines, we assess how its application to the baselines would influence the overall performance and present results in Table 9.

Although our smoothness regularizer does enhance the baseline methods' performance, this improved performance still lags behind our proposed method. This outcome suggests that the process of directly optimizing the policy through the use of preference information plays a critical role in the overall performance.

E RLHF experiment details

The fine-tuning procedure of RLHF is similar to the PbRL process discussed in our paper. First, a preference predictor is trained to assign high scores to texts that are more preferred by human teachers. Then, the reward for fine-tuning a language model is defined by combining the preference predictor and a regularizer on policy (language model) shift. Concretely, r := r ϕ -ξr KL, where r ϕ is the output of the preference predictor and r KL is the KL divergence between the output distributions of the fine-tuned model and the original (pretrained) model. This reward function aims to guide the fine-tuned model to generate outputs that are more preferred by human teachers while not deviating far from the original model. After defining the reward function, PPO [48] is applied to optimize the language model to maximize the expected reward. Recent works show that this fine-tuning process can effectively align the language models with human preference, for example by generating more helpful and harmless sentences.

We can define a DPPO-style score metric that resembles the reward function from the original RLHF process as

✶ where D is a prompt dataset, σ i is the output sequence sampled from the fine-tuned language model π θ , π θ 0 is the original pretrained language model, and D KL is the KL divergence. Here we remove the regularizer λ as the KL divergence term naturally prevents the policy from deviating far from the preferred sequences. Now we can directly fine-tune the policy by maximizing this score metric.

We evaluate our resulting RLHF algorithm by fine-tuning a pretrained OPT-1.3b language model which has 1.3 billion trainable parameters [60]. To train the preference predictor, we use the HH-RLHF dataset which contains 161K pairs of human preference data about helpfulness and harmlessness [4]. We choose OPT-350m with 350 million trainable parameters as the architecture of the predictor model. We evaluate our RLHF algorithm by comparing it with the conventional RLHF that uses the same preference predictor model and performs PPO. Our algorithm was implemented on DeepSpeed-Chat, an open code base for RLHF training 7 .

We consider several evaluation metrics to measure the performance of our RLHF algorithm. First, we calculate the expected reward, which measures the empirical mean of r ϕ from the outputs of the fine-tuned model. Second, we measure the KL divergence between the outputs of the fine-tuned model and the original model. These two metrics evaluate how well the two competing objectives

(maximizing the alignment with human preference and staying close to the original model) are optimized. Additionally, we perform an actual human evaluation on whether the fine-tuned models are more preferred than the original model. In detail, for a randomly sampled prompt from D , we generate two responses each from the fine-tuned model and the original model. Then, we ask three human workers to choose which response they prefer more in terms of helpfulness, and designate the final preferred response by majority voting 8 . We repeat this trial 300 times for each fine-tuning method and measure the win rate of the fine-tuned models.

The evaluation results in Table 3 show that our RLHF algorithm achieves a higher average reward with lower KL divergence from the original model when compared to the conventional RLHF using PPO. This shows that DPPO exhibits a better trade-off between human preference alignment and closeness to the original model. Also, DPPO achieves a higher win rate versus the original model on actual human evaluation, which shows the fine-tuned model from our algorithm is more aligned with human preference. These results demonstrate that DPPO is a promising approach for addressing preference-based learning problems in NLP.

F Discussion on the Antmaze environment in D4RL

We acknowledge that many offline RL studies highlight experiment results from the D4RL Antmaze task to demonstrate the efficacy of their methods in more challenging environments. Yet, it must be noted that the official implementation of Antmaze has a critical bug in its goal-setting procedure 9 . Concretely, the offline datasets of Antmaze assume a multi-goal setting, where a random goal location is chosen at the start of each episode, while the actual environment (unintentionally) uses a single fixed goal for every episode. Consequently, the optimal policies with regard to the offline dataset and the environment differ: According to the offline dataset, the optimal policy is to quickly sweep through all the goal candidates, while within the environment context, the optimal policy is to move directly to the solitary fixed goal.

To fix this, we can modify the Antmaze environment to choose a random goal location for each episode, thereby aligning with the provided offline dataset. Surprisingly, after this modification, we observe that current offline RL algorithms struggle to achieve meaningful performance:

Due to this stark difference in the evaluation performance, we decided not to include the Antmaze results in our main paper. We hope the offline RL community recognizes this problem in future studies involving Antmaze.

G Resources

All our offline RL experiments were run on a single RTX 3090 GPU with 10 CPU cores (Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz). For the RLHF experiments, we used an A100 GPU with 10 CPU cores (AMD EPYC 7402 24-Core Processor).

H Broad impact

Our proposed method can be used to train RL agents that better align with human preferences. This method has the potential to significantly facilitate the deployment of AI services that cater to the specific requirements of the general public. However, it is essential to recognize that the learned RL agent can be biased toward certain values ( e.g. , races, ethnic groups, religions, ...) if the preference dataset itself exhibits biases toward those values. Therefore, practitioners should exercise caution and thoroughly examine their preference dataset to identify and address any potential biases.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates learning a policy directly from preferences, i.e., pairwise comparison between trajectory segments. By defining a metric between policy and trajectory segments, the authors show that this problem resembles contrastive learning problems, and they propose to learn a policy by minimizing the distance with preferred trajectory segments. The author verified the proposed approach using D4RL benchmarks.\n\nSTRENGTHS:\n1. The storytelling is concise and relevant. Materials are organized in a way that facilitates reading.\n2. The proposed policy scoring metric is intuitive and novel.\n\nWEAKNESSES:\n1. Discussion and comparison with some very important alternative methods are missing, which weakens the significance of this paper.\nFor example, the problem of learning policies from preferences directly was studied by Liu et al. [1] before. Although they carried out their experiments in an online PbRL setting, it is straightforward to use their approach in the setting of this paper. Moreover, the proposed approach utilizes both offline data and preferences in learning. Two studies in the literature [2,3] also work in the same setting. Although the authors mentioned [2] in the related work section, they did not include results for the approach proposed in [2] or [3]. Importantly, the proposed approach outperforms the preference transformer mainly on medium-expert datasets, which are of high quality. Missing comparison with the imitation-based approach [2,3] undermines the significance of experiments.\n\n2. There lack of discussion on the connection between the proposed approach and behavior cloning. In fact, it was not straightforward for me to understand why minimizing a distance metric leads to a policy when reading sec 3.2 for the first time. It became clear after I realized the connection between $$d(\\pi,\\sigma^i)$$ and the objective of behavior cloning.\n\n\n[1] Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning, NeurIPS 2022.\n\n[2] Reward learning from human preferences and demonstrations in Atari, NeurIPS 2018.\n\n[3] Conﬁdence-Aware Imitation Learning from Demonstrations with Varying Optimality, NeurIPS 2021.\n\nQUESTIONS:\n1. I would suggest the authors compare the proposed approach with [1] to show the efficacy of their method for bypassing reward learning.\n\n2. The comparison with [2,3] is also relevant, especially for datasets with high quality.\n\n3. While the connection between PbRL and contrastive learning is interesting, the objective (eq. (4)) is an extension of the Bradley—Terry model. In other words, the output policy is not pulled away from MULTIPLE negative samples. Since the authors made use of unlabeled trajectories in a preference labeler (eq (5)), I think it is insightful to discuss a more close utilization of contrastive learning for PbRL.\n\n4. I would suggest the authors discuss the connection between their approach and behavior cloning.\n\nLIMITATIONS:\nThe authors discussed some technical limitations of their approach, but there lacks a discussion for broader impact or social impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the context of Preference-based RL (PbRL), the authors propose to learn a preference model that ranks trajectory segments without assuming that the model decomposes into a sum of rewards in an MDP sense. The motivation behind this choice is that the human teachers themselves might not follow a model of the sort, and hence modeling these preferences can be made simpler and more accurate without the usual reward decomposition assumption in PbRL. \n\nUsing this preference prediction model, the authors define a non-learned policy scoring function for ranking policies based on average similarity in action with trajectory segments in a dataset. The overall method is evaluated in an offline setting with a mix of labeled and unlabeled data. The preference model is trained on the labeled data, and used to predict preferences over the unlabeled dataset. Using these predictions, a policy optimizing the scoring function on the unlabeled dataset is returned. Comparisons with a recent baseline was provided on several robotics tasks.\n\nSTRENGTHS:\n- The hypothesis of many PbRL papers that humans provide preferences according to a Markovian reward might be too strong, and it's important to have papers exploring the validity of this hypothesis\n- The method shows improvements over a recent baseline on several tasks by using a very simple policy scoring function\n\nWEAKNESSES:\n- Some references are missing, for example in line 80 the authors state that \"While prior PbRL approaches adopt a two-step procedure involving the construction of a reward (...)\", whereas there are in fact PbRL papers that follow a very similar idea of using trajectory distances to compare policies without a reward model, e.g. Wilson et al., A Bayesian Approach for Policy Learning from Trajectory Preference Queries, NeurIPS2012.\n\n- While the main claim of the paper is to introduce a new model for predicting trajectory preferences that is not based on rewards, I was very surprised that a formal definition of this model is never provided in the paper! In the appendix, we learn that the implementation is based on PreferenceTransformer [25] that is in fact decomposing the score as a sum of (history dependent) rewards, so this is a bit confusing. \n\n- The experiments are limited to the offline setting, which is fine, but then it becomes unfair to compare to methods that assume a reward decomposition structure of the preferences because while this assumption might be restrictive, in exchange RL can be used to collect new data by optimizing the current reward model. In contrast, it is unclear how well the Direct Preference-based Policy Optimization method would work in the absence of offline data, since simply maximizing the scoring function on the labeled data will probably amount to behavioral cloning and offer very little in terms of exploration.\n\nQUESTIONS:\n- If hat{P} is based on PreferenceTransformer, is it still learning a (history dependent) reward model? Why hat{P} was not formally defined in the paper?\n- How would you perform policy optimization in the absence of offline data? How will this offline data be generated in a given RL problem?\n- If your approach and the baseline use the same preference model, what contributes the most to the improved performance? The added regularizers or the optimization of the scoring function instead of IQL/CQL\n- In the appendix you state that policy optimization is \"following the experimental protocol of IQL\". What does this mean?\n\nLIMITATIONS:\nA limitation section is included but the paper does not discuss the counter part of not having a reward model and not being able to use RL for policy optimization, which seems important to me.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents DPPO, a method for offline Preference-based Reinforcement Learning (PbRL). Rather than using an Offline RL algorithm, DPPO uses a supervised learning objective inspired from contrastive learning. A distance metric is defined, and the policy is trained to have lower distance with preferred segments than with un-preferred segments. To attain good performance, the authors learn a preference predictor (analogous to a reward model, but with the whole segment as input) with a smoothness regularizer and introduce a scale penalty to policy-segment distances, both of which correlate with performance. Their approach is evaluated on D4RL with preference labels collected from real humans.\n\nSTRENGTHS:\nThe paper was very easy to read (I appreciate that!) Explanations of major topics were largely complete.\n\nThe idea of reformulating the PbRL problem using ideas from contrastive learning is interesting, and appears to perform well in practice for the chosen benchmarks. \n\nThat brings me to experimentation. I believe the authors did a good job providing results on a number of applicable benchmarks, and I appreciate the use of real human feedback. That being said, there are a few points I wish they addressed (see weaknesses), mainly related to increased understanding of why the proposed contrastive objective works well.\n\nThe authors also ablated both hyper-parameters they introduced. \n\nI believe that this has the potential to be a very strong paper. However, there are a few oversights that I would like to see addressed listed in the weaknesses section. If these concerns are addressed and questions answered, I would be happy to revise my score.\n\nWEAKNESSES:\n**Is Contrastive Learning Important?**\nI wish the authors included an ablation showing the importance of the contrastive learning part of DPPO versus the segment smoothing objective. The smoothness regularization introduced in Eq. 5 could be used with *any* preference based RL algorithm, and does not appear to be unique to “reward” modeling vs. “preference” modeling. It would be great if the authors could include an ablation that trains PT-IQL with the preference smoothness regularizer. That would show that the performance of DPPO is due to the unique choice of contrastive objective, instead of the smoothness regularizer which could be applied to any reward-modeling algorithm.\n\n**Limitations of DPPO**\nI found the discussion of limitations and experiments surrounding limitations to be lacking. The authors opt to use a contrastive objective instead of approximate dynamic programming, and did not evaluate on any datasets where stitching is necessary. Supervised RL objectives (AWR, contrastive RL, GCSL etc.) do well when great policies can be learned with a single step of policy improvement, or when trajectory stitching isn’t necessary. On most of the chosen benchmarks, particularly Androit and Franka Kitchen, BC can do surprisingly well! For example, a score of 54 can be achieved on Androit pen cloned with just BC. This trend holds for many datasets (See Table 2 of https://arxiv.org/pdf/2004.07219.pdf). At the same time, the authors do not include results on AntMaze, which is known to require trajectory stitching, and was also included in the PreferenceTransformer paper to which the authors directly compare. \n\nI want to emphasize that it is OK if DPPO does not do as well as IQL-based PbRL algorithms on AntMaze. Including these results and an enlightening discussion on when DPPO will perform better or worse would incline me to consider raising my score, not lower it.\n\n**Modeling Rewards vs. Preferences**\n\nPersonally, I think the framing of the paper to be potentially mis-leading, specifically the part about it being “without reward modeling”. Yes, the authors do not have a “reward” model, but they still use a preference predictor with the same architecture as Preference Transformer (Appendix B). In fact the distinction between modeling a non-Markovian Reward (CITATION) and “modeling preferences” seems small: both have a neural network that takes in a state action segment and produces scores – the difference being that one is at each step (reward) and one is at the end (preference). It would be great if the authors could place more emphasis on why this difference matters. \n\n**Related Work**\nI found the related work section to be lacking. In particular, the authors omit several foundational works in preference-based RL [1,2,3,4,5], and more recent works as well [6,7,8]. I’ve included an incomplete number of examples.\n\nSecond, I will not penalize the authors for this since these works were made public after submission, but two highly related works have recently been released. [9], published at ICML also takes inspiration from contrastive learning and applies it to offline PbRL. [10], a preprint, removes the reward network from preference-based RL algorithms. \n\n[1] A. Wilson, A. Fern, and P. Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper/2012/file/ 16c222aa19898e5058938167c8ab6c57-Paper.pdf\n\n[2]  D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), July 2017. doi: 10.15607/RSS.2017.XIII.053.\n\n[3]  J. R. Lepird, M. P. Owen, and M. J. Kochenderfer. Bayesian preference elicitation for multiobjective engineering design optimization. Journal of Aerospace Information Systems, 12(10): 634–645, 2015.\n\n[4] E. Biyik, N. Huynh, M. J. Kochenderfer, and D. Sadigh. Active preference-based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.\n\n[5] W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning, pages 292–297. IEEE, 2008.\n\n[6] Joey Hejna and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning, 2022\n\n [7] Liu, Yi, et al. \"Efficient preference-based reinforcement learning using learned dynamics models.\" arXiv preprint arXiv:2301.04741 (2023).\n\n[8] Daniel Shin and Daniel S Brown. Offline preference-based apprenticeship learning. arXiv preprint arXiv:2107.09251, 2021 \n\n[9] Kang, Yachen, et al. \"Beyond reward: Offline preference-guided policy optimization.\" arXiv preprint arXiv:2305.16217 (2023).\n\n[10] Hejna, Joey, and Dorsa Sadigh. \"Inverse Preference Learning: Preference-based RL without a Reward Function.\" arXiv preprint arXiv:2305.15363 (2023).\n\nQUESTIONS:\nMajor:\n1. **There appear to be inconsistent results in comparison to Preference Transformer**, the primary paper with which the authors compare. For example, Preference Transformer achieves a score of 84.5 on Hopper Medium Replay in their publication (higher than DPPO), but in this paper has a score of 59.7. The results are also off by 30 points for Walker Medium Replay! Why is this? This feels important to answer and address in the paper, especially since the authors in the appendix state to use the same codebase as PT.\n2. How would DPPO perform on the antmaze benchmark? (See weaknesses)\n3. How would other preference-based RL algorithms perform when using the preference smoothness regularizer (see Weaknesses)?\n\nMinor:\n1. Figure 2: “Reward model from PbRL” Which model is this? Is it the preference transformer or a Markovian model?\n2. Is it true that $d_{sa}$ and AGG must be differentiable? I believe it to be the case given the method. This should be explicitly stated!\n\nLIMITATIONS:\nThe authors have very little discussion on the limitations of their approach. I would have liked to see an explicit section on limitations with some of the experiments in the Weaknesses section included.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an algorithm called DPPO that can learn a policy from paired trajectory comparisons without having to learn a reward function or do online rollouts. PBRL algorithms typically learn a reward function with a Boltzmann comparison model and then optimize it with an online RL algorithm, like PPO. In contrast, DPPO uses a kind of pseudo-Boltzmann model expressed in terms of difference in similarity to the learned policy rather than difference in returns. In particular, if trajectory $a$ is preferred to trajectory $b$, then the learned policy $\\pi$ should be more similar to trajectory $a$, on average, than to trajectory $b$. DPPO is augmented with a BC-like regularizer that biases $\\pi$ towards having low average divergence with the trajectories (which prevents it from having consistently high divergence over all trajectories), as well as an auxiliary loss term for learning from unlabeled trajectories (essentially the same as the main loss but with a 0.5 label for unlabeled trajectories).\n\nExperiments on D4RL Gym, Adroit, and Franka Kitchen show DPPO typically obtains higher mean return than both an existing offline PBRL algorithm and two existing offline RL algorithms, even though the latter two have access to the ground truth reward function. This seems to be the case for both human and synthetic labels.\n\nSTRENGTHS:\n- **[S1]** The need for online rollouts is a major practical obstacle to using reward-learning algorithms (e.g. IRL, PBRL) for control. Stable offline preference-based reinforcement learning algorithms would make it much easier to apply PBRL to systems where rollouts are expensive, since it is more difficult to tune such systems (examples include LLMs and real robots—although neither are used in this paper). Thus this paper is a welcome and timely addition to the existing toolkit of PBRL techniques.\n- **[S2]** I found the exposition clear, and the motivation for various parts of the algorithm was solid. The ablations were also adequate at demonstrating the need for the most complex parts of this algorithm (the regularizers/auxiliary losses that encourage low average policy divergence & help the algorithm learn from unlabeled demonstrations, respectively).\n- **[S3]** Empirical results are strong, especially for Adroit Pen and Kitchen. The experiments section also adequately explains why these specific benchmark environments were chosen over others in the D4RL suite, which is appreciated.\n\nWEAKNESSES:\nMy main concern with this paper is that the baselines might be too weak, especially given that the method looks fairly similar to BC. It would also be nice to have more analysis of the policy deviation-based loss function. In more detail:\n\n- **[W1]** The results in Table 1 and Table 2 (Gym and Kitchen only) seem relatively weak. Table 1 of the [RvS paper](https://arxiv.org/pdf/2112.10751.pdf) shows a method that is essentially BC (RvS) getting comparable or better performance to DPPO on most of these tasks (see RvS-R column and RvS-G column—the former is reward-conditioned and the latter is goal-conditioned). Are these results applicable in this setting, and if so, why are CQL and IQL used instead of these stronger and simpler baselines?\n  \n  The regularizer in Equation (4) seems to have the effect of encouraging the policy to have lower average divergence from the labeled trajectories in general, which is vaguely BC-like, and so I think it’s particularly important to separate this method from BC-style methods (filtered BC is going to be one of the leading competitors to any PBRL method like this).\n- **[W2]** The loss function in Equation (2) is a significant departure from the traditional Boltzmann PBRL loss. It seems to be saying that demonstrators prefer trajectories that deviate less, on average, from the optimal policy, or something of that nature. The paper doesn’t spend much time exploring the implications of this change, or the degree to which the new algorithm is or isn’t equivalent to traditional PBRL. Having more explanation of this would be helpful—when is this the “right thing” to do? Are there any situations in which it’s obviously the wrong thing to do? (due to how the labeller might be judging trajectory fragments) etc. \n\nOn the whole I’d be fairly comfortable accepting this paper if the concern about baselines is addressed. I’m marking as a weak accept pending the response to the points above, but I’m leaning towards acceptance overall.\n\nQUESTIONS:\nIn addition to responding to the weaknesses above, it would be helpful if the authors could answer these minor questions:\n\n- **[Q1]** Is Figure 2 illustrating reward or return? Also, what dataset is it using? I would not expect step-level rewards to be especially correlated given how many equivalent reward functions there are for Hopper (the goal is super simple—just run forward!), but I would expect return to be correlated on any sufficiently diverse dataset.\n- **[Q2]** What do the ranges and error bars in the tables represent? (std dev, std error, CIs, min/max, etc.)\n\nLIMITATIONS:\n**[L1]** Depending on the response to W2 above, it might be appropriate to expand the limitations with more discussion of when equation (2) is the right loss function to pursue. I expect this could tie in with the existing remarks about label noise—being more explicit about the assumptions behind equation (2) will probably reveal what kinds of label noise it can or can’t handle well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}