Direct Preference-based Policy Optimization without Reward Modeling
Authors: Gaon An
Junhyeok Lee
Xingdong Zuo
Norio Kosaka
Kyung-Min Kim
Hyun Oh Song
Venue: NeurIPS 2023 poster (2023)
Keywords: Preference-based reinforcement learning
Contrastive learning
Offline reinforcement learning
RLHF
TL;DR: We propose a contrastive learning-based PbRL algorithm which can directly learn from preference information without the need for reward modeling.

Abstract: Preference-based reinforcement learning (PbRL) is an approach that enables RL agents to learn from preference, which is particularly useful when formulating a reward function is challenging. Existing PbRL methods generally involve a two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference is from human teachers, can be difficult. Instead, we propose a PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design a novel policy scoring metric that assigns a high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms or is on par with the existing PbRL methods. Notably, on high-dimensional control tasks, our algorithm surpasses offline RL methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.

Direct Preference-based Policy Optimization without Reward Modeling
Gaon An' Seoul National University white0234@m1lab. snu. ac
Junhyeok Lee"
Seoul National University
riman3140m1lab snu.ac ac.
Xingdong Zuo NAVER
Norio Kosaka NAVER
Line Corporation kosaka. www..nooooooon.nc.ccoomm
Kyung-Min Kim NAVER kyungmin. kim. ml @naverccrpp.com
Hyun Oh Song
Seoul National University hyunoh@ml 1 lab. kr
Abstract
sefference-bssed reinforcement learning (PbRL) is approach that enables RL agents to learn from preference, which is particularly useful when formulating reward function is challenging. Existing PbRL methods generally involve two-step procedure: they first learn a reward model based on given preference data and then employ off-the-shelf reinforcement learning algorithms using the learned reward model. However, obtaining an accurate reward model solely from preference information, especially when the preference from human teachers, can be difficult. Instead, we propose PbRL algorithm that directly learns from preference without requiring any reward modeling. To achieve this, we adopt a contrastive learning framework to design novel policy scoring metric that assigns high score to policies that align with the given preferences. We apply our algorithm to offline RL tasks with actual human preference labels and show that our algorithm outperforms
on par with the existing PbRL methods. Notably, nigh-dimensional control tasks, our algorithm surpasses offline RI. methods that learn with ground-truth reward information. Finally, we show that our algorithm can be successfully applied to fine-tune large language models.
Introduction
Deep reinforcement learning has been successful in solving various decision makiing tasks where a well-defined reward function is available [34] 35 50 [4]. However, in many real-world tasks, can be challenging design a quantitative reward function that accurately reflects the desired behavior, particularly when the task involves human evaluation. Preference based RL (PbRL) seeks to provide an alternative solution by enabling agents learn from preference information between pairs of trajectory segments [0] PbRL has gained considerable interest in recent years as making a relative judgment is much easier than providing a real-valued score, which makes human feedback
much more viable [39 38 52 45 56
First two authors have equal contributions t Cooresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Preference model
1.5
Roward model
Reward based
Proforance dhtseet
Predicted Rewards
1.0
d
Preformno-eewad moddling
Agen
0.5
Unlabeted dasaset
Preference moddel
0.0
DPPO (Ours)
Preference dataset
-0.5
Agent
Unlabeled
True Rewards
dasaset
Figure 1: An overview of the difference between our approach (below) and the baselines (top). Our approach does not require modeling the reward from the prefer- ence predictor our policy optimization algorithm can learn directly from preference labels.
Figure 2: Predicted reward vs. true reward on the Hopper environment when using reward model from PbRL [27]. The re- ward model fails to accurately capture underlying reward structure.
Recent PbRL methods take two-step approach: they first learn reward model from the given preference dataset and then run off-the-shelf reinforcement learning algorithms on top of the learned reward model [10 31 (4).. However, acquiring accurate reward model only from preference labels, typically provided by human teachers, poses a significant challenge as is unclear how to extract the underlying reward structure from preference. Current methods rely modeling the reward with certain specific assumptions, though there are some concerns regarding whether those assumptions hold in practice 27]
Alternatively, predicting the preference itself comparatively more straightforward since we have direct access to training labels, allowing to leverage powerful techniques from supervised learning. Building upon this observation, we introduce a PbRL algorithm that bypasses the need for reward function modeling by directly learning from preference labels. Our approach begins by devising policy scoring metric that assigns high scores to policies aligning with the provided preference dataset. Concretely, the PbRL objective is formulated policy to be closer to more preferred trajectory segments while distancing itself from the less preferred enhance the performance of the preference predictors from previous
contrastive learning problem, guiding the learned
9 20) Furthermore, works by introducing a novel prediction smoothness regularizer. Experiment results on offline RL settings with actual human preference labels show that the proposed algorithm outperforms or is par with the baselines on all of the tasks considered [16]. Notably. in high high-dimensional control tasks, algorithm outperforms offline RL methods that utilize ground-truth reward information. Moreover, our preliminary experiments show that our algorithm can be successfully applied for fine-tuning large language models. Our official code available at https: petps://gtt //github. com/ /t///nuumllll mm11ab/PPDP
2 Preliminaries
2.1 Preference- reinforcement learning
Reinforcement learning considers an environment formulated Markov Decision Process (MDP) defined by a tuple AS, A, T, R, Po, H), where is a state space, A an action space. T (s' ss, a) is the state transition dynamies, R(s, a) is the reward function, (s(s) is the initial state distribution. and И is the time horizon. The goal of reinforcement learning is to learn a policy æ that optimizes the expected return:
Z Σ
( ap (s..))) ax))
Conventional assumes the reward information (rt) is given and uses this to optimize their policy. However, finding suitable reward metric can be costly in many real-world scenarios.

---

For example, if the goal task is to train a robot to scramble an egg, it would be unclear how to design reward function that captures all the desired properties. Instead, PbRL assumes the supervision is given in the form of preference. Concretely, for pair of trajectory segments (a"; .²), where o² (%): abs si ai, si:= a a (human) teacher indicates which segment prefers. The preference label y is given 0 if a² is preferred, i.e., o² > ot. In similar manner, y is set to I if is preferred and 0.5 if the two are equally preferred.
To learn reward model f. prior works assume the preference depends on the value of the underlying rewards summed over each timestep [0 31 40]
Ei F (s/. af:
exp
Pia" > o'; exp EE i (sf: al: 1)) exp (E (5) al:
where y denotes the learnable parameters of the reward model. Given a dataset Prref of preference triples (a "e",,",,3 g)), the reward model trained by minimizing the cross-entropy loss between the preference predictions and the ground-truth labels:
Ex(1;; Drret) = (o" ('))p' (1
E
y) log P p > =o';ul + yy P Plo' > o (I)
After training, any standard RL algorithm can be used to maximize the expected return under the learned reward model. Especially in the offline PbRL setting, assume there exists small dataset Dpref with preference labels along with a much larger unlabeled dataset without any reward or preference labels [27 49]. A typical approach involves utilizing D D to learn the reward model and applying the model to label DD
2.2 Contrastive learning
The goal of contrastive learning is to learn representations where similar sample pairs are close to each other while dissimilar pairs are far apart. For an anchor sample (e.g., an image), suppose we have a positive sample x* (e.g., the same image with data augmentation applied) and set of negative samples {x, Ji=1 (e.g., samples from other images). To learn an encoder f, contrastive learning minimizes the following loss:
exp (f (x)
ls (x,xt x [x ) = log exp (f (x) (x+)) Σ exp ( (fx)
where dot product of two representations (e.g., fxx) fxxt)) is considered the similarity score between the two samples.
3 Learning directly from preference
While prior PbRL approaches adopt a two-step procedure involving the construction of a reward model using the preference data, the fidelity of the learned reward model in reflecting the original reward function remains uncertain. The main challenge lies in extracting the underlying rewards from the given preference. Previous works assume that the preference for trajectory segment can be represented as an average of the underlying rewards [1]. However, considering PbRL typically assumes human teachers provide the preference labels, unclear whether human preferences can also be modeled in this manner. For instance, humans may focus a specific subset of the segment while ignoring other parts (4 27 Empirically, we find that the reward models learned from preferences often fail to accurately capture the underlying reward structure, illustrated in Figure 2 This issue is problematic since within the current PBRL framework, the quality of the learned policy relies heavily on the quality of the learned rewards.
Meanwhile, predicting the preference itself more straightforward task as we have explicit labels to train with and can therefore apply powerful tools from supervised learning [8 I1] 36). Based on this observation, our goal is to introduce a PbRL algorithm that directly learns with preference information without the need for reward modeling, as illustrated in Figure [I] To achieve this, we design a policy scoring metric that yields a high score when a policy aligns with the given preference dataset. In other words, we assume that desirable policy should be closer to のd than ot if o > o.. To formulate this property into an optimizable objective, we first define the closeness between a policy and trajectory segment.
3

---

Foliey-Transition Tuplo Distances
Policy- Segment Distance
B (14)
||
af
40
d d
Tgg T
log
d
(4) (0: Dret))
Figure 3: An overview the score calculation process. To score a given policy, (1) the first step is to calculate the distance between each transition tuple and the policy. (2) Second, these distances aggregated to policy-segment distance through a predefined aggregation function. (3) Finally, we obtain the score value by contrasting the policy-segment distances according their preference.
3.1 Policy-segment distance
We define the distance between a policy and trajectory segment as an aggregation of the distance between policy and each transition tuple in the trajectory segment. Concretely, the policy-segment distance can be expressed as
d (π,,O') AGG (dan (TT, sb: ab) dsa sk. ak.))
where dsa denotes the policy-transition tuple distance and AGG denotes an aggregation function. There can be multiple ways for instantiating dan. For simplicity, employ the expected & distance between the policy action and the trajectory action: daa [T, 8, a) Bawa((Iss) [|]a alla]. Similarly, we opt to use the mean operator as the aggregation function. To sum up, our policy-segmen distance function becomes
d(T,
I|| at x1l|2]
k: ++1
(-a())s;)
3.2 Preference score metric
Using the policy-segment distance defined above, we can now build score metric that satisfies our desired property. Given preference triple (a", "e"'',, 3). assume that σ" is preferred over a i.e. y= 0. We want assign a high score policy is closer dd than a². This condition can be expressed in terms of the
segment distance as o > a¹ + ((T,O") < d(TT, d!). To capture this condition across multiple segment pairs into a single metric, we adopt a contrastive learning formulation:
S(0; Pruct) (00.a) 100,01.000 pp
E
[[1 (1- (To. 00 ,0.00))))))))))) +
(Te, (Te, ,0)
(2)
exp (-d (T, 0')
s.t. 8 (TT,O',0") log exp ( + exp (-d(T,o'))
where 0 denotes the learnable parameters of the policy In terms of contrastive learning, interpret exp (-d (TT, o²)) as representing the similarity between the policy Ty and the segment o². Then, Ty. " and o are each considered the anchor, the positive sample, and the negative sample, assuming の o². Note that high score can only be achieved if the policy is closer to more preferred trajectory segments.
While Equation (2) is sufficient to express our desired property, minor drawback is that the score function is indifferent to the increase or decrease of the distances in the same magnitude. To understand this in detail, let us first denote d' d(Tu, o²) for brevity. Then, for preference triple (o", at, y) with y =
(TO. 00, ,000'))))))) = -d -loog (exp (-d0) + exp (-d')) = max (0, l - -d'}
(3)

---

1.0
smoothed
Algorithm Direct Preference-basee Policy Optimization
Smoothed
Preference Prediction
0.8
Input: Unlaheled dataset DD preference dataset 2 peef- learning no and rjo, ber of training steps M and N, and regular- ization parameters v. Initialize network parameters & and 0 for step to M do
0.4
0.2
Update the predictor parameter:
Φ + Φ 77 v. (0; prett D) end for
0
25
50
75
100
125
150 175 200
for step = to do
Segment index Figure 4: Predicted preference of overlapping seg- ments from single trajectory. In detail, measure Plo > o'+1], where o² ai....... Sitk, a.+*).
Update the policy parameter:
< + no (o D.
end for
Therefore, the score value remains the same when the distances increase or decrease the same magnitude. In other words, the score for the distances (d ,d' and (d" + d' a) are identical, indicating there no penalty when a policy deviates from even the preferred trajectory segment. To solve this, we add regularizing factor À E (0, 1) that decreases the score when the overall scale of the policy-segment distances increases:
S(0; D prets A) (0",a'.0))
[11 (1- (To 00 ,o -.-
(4)
exp (T.
(T, ,0; log exp (-d 0')) exp (- dd ( od))
we set smaller than and plug in this new formulation to Equation (3). is easy to find out that the score will decrease when the overall distances increase. The resulting score calculation process for Equation is illustrated in Figure
3.3 Policy optimization with preference predictor
We can directly optimize policy with preference labels by maximizing the score function in Equation 4 However, to leverage the unlabeled dataset D, we train a preference predictor using the labeled dataset Ppeet We formulate this process as a simple binary classification problem and use the cross-entropy loss optimize the predictor P.
( Drrets D) =
E (a",a' ())w
(1 4) log P P > of: + y log Plo' > of:
(5)
Preference Correctness
E (a.0")
Preference Smoothness
where & denotes the learnable parameters of the preference predictor P. The first term resembles Equation ( but the difference is that here we directly model the preference predictor instead of modeling the reward function. The second term is smoothness regularizer which guides the predictor have similar preference against two largely overlapping segments. Concretely, given
(s;. a, ........ Si+k. ar++), ), we sample o' from the same trajectory as (Si+a. aita..... Sitatkk a where a (0, m²) with m & k. We observe that if no smoothness regularization applied (u 0), the preference can vary significantly between two almost identical segments, as illustrated in Figure This behavior is undesirable as a human teacher will unlikely be able to even identify the difference between the two segments.
5

---

prroo sp
0.17
0.07
0.18
0.07
0.02
0.02
0.06
dm (TBci ap)
0.13
0.05
0.21
0.09
0.06
0.02
0.12
/
1
TDPPO の²
0.06
o0
d (TBC. o
0.10
TDPPOs o² d TBc. o*)
= 0.47
o
= 0.10
Torro 5/ a/)
0.24
0.67
0.60
0.67
0.40
0.41
0.59
dea 8/
0.14
0.21
0.28
0.19
0.16
0.21
0.12
Figure 5: DPPO BC for two example trajectory segments the walker2d-medium dataset
of D4RL Gym. DPPO successfully avoids the behavior from the unpreferred trajectory segment while BC also clones the unpreferred behavior.
After training the preference predictor with Equation we can train our policy with the unlabeled dataset D by sampling pairs of trajectory segments and labeling their preferences:
S(0; D, o, A) = (00,0')~0 [[L 7) s (TT. oº o': + D. (TT, o o";
s.t. 7 7 6 > 0.5
Algorithm D summarizes the full process of PbRL algorithm. Figure shows an example of two policies each learned using our algorithm and behavior cloning (BC), which shows that vanilla BC suffers from imitating the behavior of unpreferred trajectory segments while our algorithm successfully distances from it. We name our preference-basee policy optimization algorithm as DPPO, an abbreviation for Direct Preference-base based Policy Optimization. The sections below evaluate the performance of DPPO in the offline setting.
4
Experiments
4.1
Offline RI. experiment details
Following recent PbRL. works, we evaluate our algorithm on the offline setting which assumes large unlabeled dataset is given along with much smaller preference-labeled dataset Drret [49 27]. We evaluate our algorithm on D4RL, standard benchmark for offline RL, with preference datasets generated by actual human teachers [16]. We did not experiment on Antmaze, a widely used task in D4RL, due to crucial bug in its environment implementation (please refer to Appendix F for more details), Below is brief description the tasks considered our experiments:
Gym D4RL Gym provides datasets for the Gym locomotion environments where the goal task is to move forward while maintaining body balance. D4RL Gym contains various types datasets each from different data collection process. Among those datasets, we focus the medium-replay and medium-expert datasets, which contain trajectories with the most diverse quality.
Adroit pen Adroit tasks involve controlling 24-DoF robotic hand to perform tasks such as grabbing pen or hammering nail (44 Adroit tests if offline RI methods can learn from human demonstrations on these high-dimensional tasks. We focus on the pen task since standard offline RI. algorithms fail to achieve reasonable performance on other tasks, even with the reward information.
Kitchen Kitchen requires learning 9-DoF robot manipulation task 19 Similar to Adroit, the reward function sparse and the dataset contains human demonstrations. However, Kitchen requires solving multiple sub-tasks sequentially (e.g., opening the microwave, then turning off the switch) in single episode. While Kitchen has three types of datasets (complete, partial, and mixed), we consider the latter two complete only contains successful trajectories with almost identical quality.

---

For the Gym hopper, Gym walker2d, and Adroit pen tasks, we utilize publicly available human preference datasets released by [27]. For the other tasks, we generate a new preference dataset as there are no preference datasets available. To collect the human preference datasets for each task, we strictly adhere to the protocol outlined by [27]. This involves defining set of desirable behaviors and instructing the human teacher to label preferences accordingly. For example, desired behavior in the Hopper environment would be to maintain body balance and prevent falling down. The size of the resulting preference datasets ranges from 100 to 500 samples, depending the specific task. For more details regarding the dataset generation process, please refer to Appendix
We consider Preference Transformer (PT) our baseline method, which is state-of-the the-art approach in offline PbRL (27]. PT employs transformer network train the reward model and leverages IQL, an offline RL algorithm, for policy optimization using the learned rewards [28]. This original version denoted as PT+IQL. Considering that the choice of policy optimization algorithm significantly impacts the performance of reward modeling methods, we also explore using CQL for policy optimization [30 This modified version of PT denoted as PT+CQL. Additionally, we present the performance of CQL and IQL when utilizing the ground-truth truth reward information for reference. Note that these reward-baser RL baselines do not provide fair comparison with our method as they have access to much denser supervision signal. For more implementation details regarding the baselines and our algorithm, please refer to Appendix B For more experiments including behavior-cloning baselines, please refer to Appendix C
4.2 Evaluation results
Table 1: Normalized average return D4RL Gym tasks, averaged over 5 seeds. + denotes the
standard deviation.
Learning with task rewards
Learning with preference only
Task Name
CQL 44.3 4 0.7
IQL
PT+CQL
PT+IQL
DPPO (Ours)
halfcheetah-meddee -edium-reppayy
45.7 +: 0.6
27.1 +: 17.7
42.3 + 0.5
40.8 + 0.4
hopper meeeee-ddddmdiieeeeee replay
84.1 + 14.2
100.5 + 1.4
49.1 + 22.0
59.7 = 25.8
73.2 - 4.7
walker-medium -eplaa replay
80.0 + 3.4
74.8 3.4
52.8 7.2
43.3 = 39.8
50.9 + 5.1
maaffcheeta-mmddium-eee
88.5 + 9.7
7.4
77.1 + 0.9
83.6 3.8
92.6 + 0.7
hopper-medium-oxperr walker24--eedimm-xxprr
103.7 7.5
84.1 24.1
89.2
14.4
67.8
32.3
107.2 + 5.2
108.4 + 0.3
107.5 + 4.4
77.7 + 1.2
109.8 + 0.4
108.6 + 0.1
Average
85.1
82.7
62.2
67.8
78.8
Table shows the evaluation results for the D4RL Gym tasks. DPPO demonstrates superior or comparable performance to the preference-based learning methods across all considered tasks. In terms of average performance, our method outperforms the baselines by a large margin with minimum of % lp and reaches a performance level similar to the methods that learn with ground- truth rewards. Also, DPPO exhibits significantly lower variance in performance compared to the baseline methods like PT+IQL, which suffer from pronounced fluctuations in performance.
Table Normalized average return on D4RL Adroit pen and Kitchen tasks, averaged over seeds. + denotes the standard deviation.
Learning with task rewards
Learning with preference only
Task Name
COL
IQL
PT+CQL.
PT+IQL
DPPO (Ours)
pen-human
44.2 + 7.8
53.8 + 36.9
31.6 3.3
53.0 + 31.7
76.3 + 14.4
pen-eloned
42.4 5.1
51.3 + 37.1
18.3 + 10.6
42.9 24.4
75.1 7.7
Average
43.3
$2.6
25.0
48.0
75.7
kitchen-mixed
10.7 + 10.8
50.6 6.2
12.3 + 7.7
48.0 + 11.9
52.5 + 3.1
kitchen-partial
12.9 13.0
58.8 6.5
14.1 + 13.0
40.2
12.3
49.4 5.7
Average
11.8
$4.7
13.2
44.1
51.0

---

60
Average Return
60
Average
A-0.25
0-0.0
20
A-0.50
20
0.
v=1.0
A=1.00
2=2.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Step
1e6
Step
(a) A
(b)
Figure 6: Ablation study results on the hopper-mediee mediumm-rrplas dataset. (a) and (b) each shows the average performance results for DPPO while varying and v.
Table 2 shows the results on the more challenging D4RL, Adroit pen and Kitchen tasks. Once again, DPPO outperforms all the baselines by large margin. Especially, the performance gap is the largest in the Adroit pen tasks, where DPPO even surpasses the methods that learn with ground-truth rewards. A major distinguishing factor between the Adroit pen and the other environments is the dimensionality of the action space, which amounts to 24, significantly larger than action spaces ranging from to 9 in other environments. We conjecture that the value learning approach of the baseline methods struggles to scale up to the high- digh-dimensional action spaces of Adroit. Furthermore, DPPO outperforms the baselines on the Kitchen tasks well, underscoring the scalability of our method to tackle more complex tasks.
4.3 Ablation studies
We assess the importance of the two crucial components of DPPO, which are the conservativeness regularizer À and the smoothness regularizer v. The ablation results are presented in Figure 6 The results show that both components crucial for achieving high performance. Moreover, the experiments demonstrate that DPPO exhibits considerable degree of robustness to variations in the strength of the regularizations. Since smoothness regularizer soward-nodeling baselines, we evaluate how applying this regularizer to the baselines affects the overall performance in Appendix D While the smoothness regularizer does improve the performance of the baselines, the improved performance falls behind DPPO.
also be easily applied to
4.4 Effect of dataset size
110
70
60
90
Average Retur
50
Average
60
50
DPPO (Ours)
40
DPPO (Ours)
PT+
Dataset Size
Dataset Size
(a) hhopper-meiineee -epllay---e
(b) hodper-medium-epperr
Figure 7: Average return results of each method while varying the size of the preference dataset.

---

Previous works primarily focus on evaluating their algorithms using fixed preference dataset for each task [27 13. In this section, we evaluate how the size of the preference dataset impacts the overall performance of PbRI. algorithms. Concretely, we vary the size of the preference dataset in the hopper- medium-replay and hopper-medium--vyy each dataset size. The results are shown in Figure 7 We observe that DPPO consistently outperforms the baseline method on all dataset sizes, displaying higher average performance. Interestingly, on hopper-medium--rpllay we find that the baseline method PT+IQL falls into unique failure mode when the dataset size 200, which to stand still and not move forward.
tasks and measure the performance PbRL. methods on
4.5 Experiments with scripted teachers
PT+CQL
100
PT+IQL
DPPO (Ours)
Following prior works TO 32 we additionally
Average Return
80
evaluate our algorithm using preference labels gener- ated by scripted teachers. A scripted teacher is syn- thetic teacher who adheres strictly to the task rewards when making decisions: σº o¹ E rf Ei=orl. r/. It is important to note that the scripted teacher setting is not realistic as the human teacher setting, and we include this section for reference pur- poses. The results in Figure 8 show that our algorithm continues to outperform the baselines in terms of aver- age performance, even in this synthetic setting. In detail, the results for DPPO and PT+IQL resemble the results from the human teacher setting. However, the performance of PT+CQL has dropped significantly compared the human teacher setting. This disparity of performance reassures the observations from [27 that preferences from human teachers and scripted teachers have different characteristics and should not be treated interchangeably.
60
Gym
pen
Figure 8: Average performance the scripted teacher setting.
4.6 Fine-tuning LLMS with DPPO
Recent works show that PbRL can be used to fine-tune large language models with human preference, which is commonly termed RLHF (reinforcement learning from human feedback) [52] 39, 4 In RLHF, a reward model is trained to predict human preference between two output texts, and then the learned reward model is employed to fine-tune the language model through off-the-shelf RL algorithms such PPO. As from offline RL, we can fine-tune language model directly with the preference predictions by replacing PPO with DPPO. This replacement allows removing unnecessary assumptions required to run reward-based policy optimization techniques like PPO, for example assuming the reward is given only at the end of the output sequence. To empirically examine if DPPO can be used fine-tune large language models, we conducted some preliminary experiments using human preference datasets to train the preference models and human evaluation to evaluate the tuned language model. We refer to Appendix for more details regarding the implementation details and the experimental setup.
Table 3: RLHF results using DPPO (ours) compared to PPO. The values in the parentheses denote the gain on average reward compared the original model.
Human eval.
Fine-tuning method
Avg. reward (t)
KL divergence (4)
win rate (t)
PPO
4.335 (+1.192)
0.0091
0.667
DPPO (Ours)
4.515 (+1.372)
0.0083
0.697
The results Table 3 show that DPPO can be successfully used to fine-tune large language models with real human preference, achieving performance comparable to the reward-based RLHF method. We believe that this result is promising as shows that DPPO can be used to fine-tune large language models without the need for reward modeling.

---

5
Related works
Preference-based reinforcement learning long line of works has studied learning agents from human preference 4 12 B 471. Notably, [10] showed that
scale PbRL by learning reward model from preference and applying it to off- fff-the-shelf RL algorithms. Several follow-up works have been proposed building upon this work. For example, work merges the PbRL framework with imitation learning by introducing other types of supervision such as demonstrations [21]. Other works utilize techniques from semi-supervised learning or data augmentation to improve sample Another line of work aims to improve the reward model by removing Markovian
efficiency 31
40
13 27 Recent works have shown that PbRL can greatly boost the performance of large-scale language models by finetuning them with human feedback [52] 56, 37 39, 45, (38). Similar to our work, [55] leverages the preference information to directly optimize the policy using the concept of trajectory distance. However, their method requires the two trajectories to start from the same initial state and roll out from the current policy being trained. which makes it challenging also explore the idea of
assumptions
to utilize extensive pre-collected data. Concurrent to our work, [25 43 directly optimizing the policy with preference information.
Offline reinforcement learning Offline reinforcement learning methods adopt various techniques to bias the learned policy towards the given offline dataset. For example. some works directly regularize the policy to prefer the actions from the dataset [29 57 [7]. Another line of works implicitly biases the policy by regularizing the value function 30 2 58 Some works instead start from a SARSA-styl TD-update algorithm to avoid querying values for out-of-distribution actions sequence modeling problem, where a model is learned to generate actions conditional to the given target return [8] 22 33 46 14]. This approach soon gained popularity in the literature due its high stability and scalability compared to the more traditional value-based approaches (33 46).
7 28 A more recent line of work casts the offline RL problem
Contrastive learning A core design choice for contrastive learning is to define the positive and negative sample pairs. An earlier work utilizes the structure of the data and considers two different patches from a single sample as a positive pair (53). The most popular approach is to apply heavy data augmentation to a single sample repeatedly to produce positive pairs 19 20). The success of these methods in unsupervised vision representation learning has inspired many follow-up works, such as application to supervised learning or extension to other domains such as NLP or graph 226 59].
In the RL domain, there also have been works that leverage contrastive learning to learn unsupervised representations tailored for RL [5]]. Also, a recent work directly casts the contrastive learning framework as goal-conditioned RL problem [[5].
Discussion
Our proposed PbRL algorithm enables agents learn directly from preference signals, removing the need for reward modeling. We achieve this by formulating new policy optimization problem under the contrastive learning framework. Thorough empirical evaluations on various offline PbRL tasks with actual human rewards show our method outperforms the baselines in most of the tasks considered. Interestingly, our algorithm shows better scalability to nigh-dimensional control tasks when compared to other RL basclines, including those that learn with ground-trui reward information. Additionally, our algorithm demonstrates better data efficiency. These results show that directly utilizing preference signals without reward modeling is a promising direction for PbRL.
limitation of our work is that label noise, which is likely to exist for human evaluations, is not modeled in our algorithm. Investigating the effect of label noise stemming from human teachers would be an interesting direction for future research (23]. Also, our current algorithm does not incorporate the prediction confidence information, which was excluded as the empirical performance is already strong compared to the baselines. How to appropriately incorporate the prediction confidence would also be a meaningful research direction.
10

---

Acknowledgements
We would like to express our gratitude to Deokjae Lee for his assistance in our natural language domain experiments. This work was supported by SNU-NAVER Hyperscale Al Center, Institute of Information Communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) [No. 2020-0-00882, (SW STAR LAB) Development of deployable learning intelligence via self-sustainable and trustworthy machine learning, 55% and No. 2022-0-00480, Development of Training and Inference Methods for Goal-Oriented Artificial Intelligence Agents, 25%], and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (RS-2023-00274288), 20%. Hyun Oh Song is the corresponding author.
References
[1] Riad Akrour, Marc Schoenauer, and Michele Scbag. Preference-basee policy learning. In Machine Learning and Knowledge Discovery in Databases, 201 1.
[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In NeurlPS, 2021.
[3] Layla El Asri, Bilal Piot, Matthieu Geist, Romain Laroche, and Olivier Pietquin. Score-based inverse reinforcement learning. In AAMAS, 2016.
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodd, Danny Hernandez, Tristan Hume. Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 05862,, 2022.
[5] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal Józefowicz. Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique P. d.O. Pinto, Jonathan Raiman, Tim Salimans, Jeremmy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912:069/266688 2019.
[6] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952, 1952.
[7] David Brandfonbrener, William F. Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off off-policy evaluation. In NeurIPS, 2021.
[8] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning sequence modeling. In NeurIPS, 2021.
[9] Ting Chen, Simon Korblithh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020.
[10] Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In NeurlPS, 2017.
[11] David R Cox. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series (Methodological), 20(2):115-232, 1958.
[12] Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acquisition function. Autonomous Robots, 2015.
[13] Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-markovian reward modelling from trajectory labels via interpretable multiple instance learning. In NewrIPS, 2022.

---

[14] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline RL via supervised learning? In ICLR, 2022.
[15] Benjamin Eysenbach, Tiajjun Zhang, Ruslan Salakhutdinov, and Sergey Levine. Contrastive learning as goal-conditioned reinforcement learning. In NewrIPS, 2022.
[16] Justin Fu, Aviral Kumar, Ofrr Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[17] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
[18] J. Good. Rational decisions. Journal of the Royal Statistical Society. Series B (Methodological), 14(1):107-114, 1952.
[19] Abhishek Gupta, Vikash Kumar, Corey Lynch. Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. arXiv preprint arXiv: arXiv:1910.11956, 2019.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
[21] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In NewrlPS, 2018
[22] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In NeurIPS, 2021.
[23] Anca Dragan Joey Hong, Kush Bhatia. On the sensitivity of reward inference to misspecified human models. ICLR, 2023.
[24] Daniel Kahneman. Evaluation by moments: Past and future. Choices, values and frames, pages 693-708, 2000.
[25] Yachen Kang, Diyuan Shi, Jinxin Liu, Li He, and Donglin Wang. Beyond reward: Offline guidee policy optimization. In ICML, 2023.
[26] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. NeuriPS, 2020.
[27] Changyeon Kim, Jongjin Park, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Preference transformer: Modeling human preferences using transformers for rl. In ICLR, 2023.
[28] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learnin. In ICLR, 2022.
[29] Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurIPS, 2019.
[30] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurlPS, 2020.
[31] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble:
Feedback-effcieen interactive reinforce- ment learning via relabeling experience and unsupervised pre- pre-training. In ICML, 2021.
[32] Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. In NeurlPS Datasets and Benchmarky Track, 2021.
[33] Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch, Multi-game decision transformers. In NewrIPS, 2022.
12

---

[34] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller. Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518, 2015.
[35] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lill- icrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In ICMI,, 2016.
[36] Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In NeurlPS, 2019.
[37] Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schul- man. Webgpt: Browser-assisee question-answeriig with human feedback. arXiv preprint wwXi::2/2..09332 2021.
[38] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[39]
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
[40] Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: Semi-supervised reward learning with data augmentation for feedback-efficient preference-based reinforcement learning. In ICLR, 2022.
[41] Patrick M. Pilarski, Michael R. Dawson, Thomas Degris, Farbod Fahimi, Jason P. Carey, and Richard Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In IEEE International Conference on Rehabilitation Robotics, 2011. 1.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[43] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In NeurlPS, 2023.
[44] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. In RSS, 2018
[45] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In ICLR, 2023.
[46] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackic Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. In TMLR, 2022.
[47] Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference-basee learning of reward functions. In Robotics: Science and Systems, 2017.
[48] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
13

---

[49] Daniel Shin, Daniel S. Brown, and Anca D. Dragan. Offline preference-bace apprenticeship learning. In ICML Workshop on Human-Al Collaboration in Sequential Decision Decision-Making. 2021.
[50] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, llya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game go with deep neural networks and tree search. Nature, 529, 2016.
[51] Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised represen- tations for reinforcement learning. In ICML, 2020.
[52] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. In NeurIPS, 2020.
[53] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint
2018.
[54] Oriol Vinyals, Igor Babuschkin, Wojeiech M. Czarnecki, Michaël Mathicu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg. Alexander Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McK- inney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575, 2019.
[55] Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In NeurlPS, 2012.
[56] Jeff Wu, Long Ouyang, Daniel M. Ziegler. Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint Stt::2109.10662. 2021.
[57] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:/911.11366 2019.
[58] Yue Wu, Shuangfei Zhai. Nitish Srivastava, Joshua M. Susskind. Jian Zhang, Ruslan Salakhut- dinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In ICML, 2021.
[59] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurlPS, 2020.
[60] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. arXiv preprint 200::2205500066 2022.
14

---

A Human preference datasets
For the D4RL Gym hopper, walker2d, and Adroit pen tasks, we use the human preference datasets provided by 27]. Each dataset consists of 100 preference triples, except for the Gym *-medium- replay datasets which contain 500 preference triple Each trajectory segment is of length 100 (i.e., k=100). For the other tasks such Gym halfcheetah and Kitchen, could not find publicly available human preference datasets. Therefore, we generate new preference datasets with human evaluation by strictly following the procedure from [27]. The preference labeling process consists of two steps: sampling a pair of trajectory segments from the dataset and labeling their preference under sampled, so we first randomly sample two trajectories from the offline dataset and then randomly choose segment from each of the two trajectories form segment pair. We provide more details of the labeling process for each task below.
some predefined guidelines. For the first step, [27 does not specify how the segments
A.1 Gym HalfCheetah
Following the experiment settings from 27 we sample 500 and 100 segment pairs from halfcheetah- medium-replay and (the authors) to make preference decisions under the following guidelines:
datasets, respectively. We assign the human teachers
The primary goal for the agent is to move forward as far as possible.
If the agents both satisfy the primary goal similarly, the agent that has more stable posture is preferred. In other words, prefer the agent that staggers less.
A.2 Franka Kitchen
Franka Kitchen environment provides six available sub-tasks, where the goal is to perform four of them in designated order: open the microwave oven, the kettle the top left burner, turn the light switch, and open the slide cabinet. The other two tasks, which are turning on the burner switch and opening the hinge cabinet, are dummy tasks without any positive reward. Figure 9 shows an example of a goal-related task and a dummy task. The robot on the left is moving kettle, which goal-related task. On the other hand, the robot on the right is turning on a burner, which is dummy task.
For each offline dataset kkitchen-partic and kitchen kitchen-mixede mixed), sample 100 pairs of trajectory segments and label them under the following guidelines:
An agent that solves more goal-related tasks (in order) is preferred.
If the two agents solved the number of goal-related tasks, the agent that solved fewer dummy tasks is preferred.
If the two agents solved the same number of goal-related and dummy tasks, the agent that was working on another geal-related task is preferred.
'For walker medium-replay, [27] originally uses preference dataset of size 500, but we subsample 100 samples from this dataset as we find that a smaller dataset suffices for PbRL algorithms to achieve reasonable performance.
15

---

Figure 9: Example of goal-related task (left) and a dummy task (right) in Franka Kitchen. The task on the left is to move a kettle and the task on the right is to turn on burner.
B Offline RL experiment details
For each method, we evaluate its performance by collecting ten trajectories with online interaction and measuring the average return. Following the standard protocol of the offiine RL literature, rescale the average return by 100 R--auam
Ressen Runden where R denotes the (empirical) average return of the policy, and Rrandom and Rexpert are the expected returns of a random policy and an online expert policy, respectively. We provide the implementation details for each PbRL method below.
B.1 Transornncerannsfommrrrr (PT)
Transformer
use the official implementation from the authors to train reward mode Then, for PT+IQL, we run IQL with the default hyperparameters for each environment using the official code Similarly, for PT+CQL. we run CQL with the default hyperparameters for each environment using the official code Table 4 and Table 5 lists the detailed hyperparameter settings for IQL and CQL.
Table 4: Hyperparameter settings for IQL in PT+IQL.
Task name
Learning rate
Layers
Hidden layer size
T
3
Dropout
Gym
3 - 10
2
256
0.7
3.0
Adroit pen, Kitchen
3 10
2
256
0.7
0.5
0.1
Table Hyperparameter settings for CQL in PT+CQL.
Task
Learning rate
# Layers
Hidden layer size
Min Ver.
Gym
1
256
10.0
Adroit pen
3 10-1
256
1.0
5.0
Kitchen
3 3-10-
256
1.0
5.0
B.2
DPPO (Ours)
For the preference predictor, we start from a GPT-2 based transformer architecture from PT and apply some modifications. First, removed the preference attention layer proposed by PT. Also, since PT requires per-step reward predictions, their transformer output is restricted to a scalar value for each
16

---

timestep. Instead, our preference model outputs vector embedding per step, which is aggregated and forwarded into final MLP layer for the preference score prediction. We train the predictor for 10,000 update steps on the preference dataset. Table 6 lists more details of the training of the preference predictor. For policy optimization, we use a 2-layer ML. architecture and update the policy for leó update steps, following the experimental protocol of IQL.. Unlike CQL and IQL. which adopt a stochastic policy model, we use deterministic policy model as we find that both models have no difference in empirical performance. More detailed settings for the policy optimization procedure are specified Table
Table 6: Hyperparameter settings of the preference predictor training process in DPPO (Ours).
Task name
Learning rate
Layers embedding dim
1/
m
All tasks
1 10-4
256
1.0
20
Table 7: Hyperparameter settings of the policy optimization process in DPPO (Ours).
Task name
Learning rate
# Layers
layer size
X
Dropout
Gym
3 10-1
2
256
0.5
0.25
Adroit pen, Kitchen
3 10
2
256
0.1
0.25
C
Offline RL results with more baselines
Our offline RL experiments in the main paper focus on baselines that apply value-based RL algorithms after modeling the reward, following (27). However, another interesting line of work in offline RI is the behavior boaning-based RL algorithms, which typically learn policy conditional to target return or goal 88 [1]. Here, we provide the full experiment results including two behavior cloning- based baselines: %BC, which performs behavior cloning on trajectories with top-10% return values from the offline dataset, and RyS 14
Table 8: Normalized average return on D4RL Adroit pen and Kitchen tasks, averaged over 5 seeds. + denotes the standard deviation.
Task Name
PT+%BC
PT+RvS
PT+CQL
PT+IQL
DPPO (Ours)
pen-human
19.4 1 6.7
-1.8 1 0.5
31.6 3.3
53.0 +: 31.7
76.3 1: 14.4
pen-cloned
37.4 1 7.5
-2.2 1 0.3
18.3 10.6
42.9 1 24.4
75.1 1: 7.7
Average
28.4
-2.0
25.0
48.0
75.7
kitchen-mixed
40.9 1 9.0
27.5 9.4
12.3 7.7
48.0 +: 11.9
52.5 +: 3.1
kitchen-partial
53.4 + 9.0
26.0 + 4.9
14.1
13.0
40.2 + 12.3
49.4 1: 5.7
Average
47.2
26.8
13.2
44.1
51.0
The results in Table show that DPPO outperforms the behavior cloning -aased haselines on all tasks, with the performance gap being especially large the challenging pen tasks.
D Additional ablation on the smoothness regularizer
In recognition of the fact that our novel smoothness regularizer can also be integrated into any reward-modeling baselines, we assess how its application to the baselines would influence the overall
performance and present results in Table 9
17

---

Table Normalized average return on D4RL Adroit pen and Kitchen tasks, averaged over seeds. + denotes the standard deviation.
Task Name
PT+CQL
PT+CQL+v
PT+IQL
PT+IQL+v
DPPO (Ours)
pen-human
31.6 3.3 18.3 - 10.6
18.3 17.2
$3.0 = 31.7
53.7 = 42.3
76.3 14.4
pen-cloned
32.7 = 11.2
42,9 = 24.4
49.8 = 32.2
75.1 = 7.7
Average
25.0
25.5
48.0
51.8
75.7
kitchen-mixed
12.3 + 7.7
12.0 5.0
48.0 = 11.9
49.4 = 5.2
52.5 3.1
kitchen-partial
14.1 + 13.0
11.4 = 11.2
40.2 = 12.3
49.4 + 5.2
49.4 5.7
Average
13.2
11.7
44.1
49.4
51.0
Although our smoothness regularizer does enhance the baseline methods' performance, this improved performance still lags behind our proposed method. This outcome suggests that the process of directly optimizing the policy through the use of preference information plays critical role in the overall performance.
E RLHF experiment details
The fine-tuning procedure of RLHF is similar the PbRL process discussed in our paper. First, preference predictor is trained to assign high scores to texts that are more preferred by human teachers. Then, the reward for fine-tuning language model defined by combining the preference predictor and regularizer on policy (language model) shift. Concretely, the output of the preference predictor and FxL. is the KI. divergence between the output distributions of the fine-tuned model and the original (pretrained) model. This reward function aims to guide the fine-tuned model to generate outputs that are more preferred by human teachers while not deviating far from the original model. After defining the reward function, PPO 48) is applied to optimize the language model to maximize the expected reward. Recent works show that this fine-tuning process can effectively align the language models with human preference, for example by generating more helpful and harmless sentences.
T frKL., where r is
We can define DPPO-style score metric that resembles the reward function from the original RLHF process as
=
E
[11 77) -s (TT, + T- (Ta, o0)]
, 00,0"~a0(-100 x
E x~D
( To, [x))]
s.t. 1 =1
0
1 10 > ,²: Øl 0.5].
where D is a prompt dataset, o² is the output sequence sampled from the fine-tuned language model Ty, Te, is the original pretrained language model, and DKL is the KI. divergence. Here we remove the regularizer À as the KL divergence term naturally prevents the policy from deviating far from the preferred sequences. Now we can directly fine-tune the policy by maximizing this score metric.
We evaluate our resulting RLHF algorithm by fine-tuning a pretrained OPT-1.3b language model which has 1.3 billion trainable parameters [60]. To train the preference predictor, we use the HH-RLHF dataset which contains 161K pairs of human preference data about helpfulness and harmlessness
4 We choose OPT-350m with 350 million trainable parameters as the architecture of the predictor model. We evaluate our RLHF algorithm by comparing it with the conventional RI.HF that uses the same preference predictor model and performs PPO. Our algorithm was implemented on DeepSpped-Chhar an open code base for RI.HF training
We consider several evaluation metrics measure the performance of our RLHF algorithm. First, we calculate the expected reward, which measures the empirical mean of rs from the outputs of the fine-tuned model. Second, we measure the KL divergence between the outputs of the fine-tuned model and the original model. These two metrics evaluate how well the two competing objectives
DeepSpeed- Chat
18

---

(maximizing the alignment with human preference and staying close to the original model) are optimized. Additionally, we perform an actual human evaluation on whether the fine-tuned models are more preferred than the original model. In detail, for a randomly sampled prompt from D, we generate two responses each from the fine-tuned model and the original model. Then, we ask three human workers to choose which response they prrferr more in terms of helpfulness, and designate the final preferred response by majority voting 8] We repeat this trial 300 times for each fine-tuning method and measure the win rate of the fine-tuned models.
The evaluation results in Table B show that our RLHF algorithm achieves a higher average reward with lower KL divergence from the original model when compared to the conventional RLHF using PPO. This shows that DPPO exhibits a better trade-off between human preference alignment and closeness to the original model. Also, DPPO achieves a higher win rate versus the original model on actual human evaluation, which shows the fine-tuned model from our algorithm more aligned with human preference. These results demonstrate that DPPO is a promising approach for addressing preference-based learning problems NLP.
F Discussion on the Antmaze environment in D4RL
We acknowledge that many offline RL studies highlight experiment results from the D4RL Antmaze task to demonstrate the efficacy of their methods in more challenging environments. Yet, must be noted that the official implementation of Antmaze has critical bug in its goal-setting procedure' Concretely, the offline datasets of Antmaze assume a multi-goal setting, where a random goal location chosen the start of each episode, while the actual environment (unintentionally) uses a single fixed goal for every episode. Consequently, the optimal policies with regard to the offline dataset and the environment differ: According the offline dataset, the optimal policy is quickly sweep through all the goal candidates, while within the environment context, the optimal policy is to move directly the solitary fixed goal.
To this, we can modify the Antmaze environment to choose a random goal location for each episode, thereby aligning with the provided offline dataset. Surprisingly, after this modification, observe that current offline RL algorithms struggle to achieve meaningful performance:
Table 10: Performance of offline RL algorithms on the original and fixed Antmaze environment.
Orig. Antmaze
Fixed Antmaze
Task Name
&BC
CQL
IQL
%BC
CQL
IQL
medium-play-v2
48.7 +:: 3.3
44.7 + 27.8
62.0
+ 4.1
10.5 + 2.9
9.8
+ 4.8
1.5 + 1.5
medium-divee diverse-v2
35.5 + 6.0
24.4 32.3
66.2
6.3
10.4 3.2
3.1 3.3
1,8
+ 0.8
large-play-v2
12.2 1 7.5
11.7
59.8 8.1
11.1 = 1.4
3.1
14.5
+ 4.7
nugge-diverse-2
10.8 3.6
5.5
55.2 6.5
8.0
4.8
11.2 + 1.6
Due to this stark difference in the evaluation performance, we decided not to include the Antmaze results in our main paper. We hope the offline RI. community recognizes this problem in future studies involving Antmaze.
G Resources
All our offline RL experiments were run single RTX 3090 GPU with 10 CPU cores (Intel(R) Xeon(R) Gold 5218R CPU @ 2.10GHz). For the RLHF experiments, we used an A100 GPU with 10 CPU cores (AMD EPYC 7402 24-Core Processor).
We used Amazon MTurk htwp://wwwaaaaaaatttt//////aattr to conduct our human evaluation experiments.
19

---

H
Broad impact
Our proposed method can be used train RL agents that better align with human preferences. This method has the potential to significantly facilitate the deployment of AI services that cater to the specific requirements of the general public. However, it essential recognize that the learned RL agent can be biased toward certain values (e.g,, races, ethnic groups, religions, ...) if the preference dataset itself exhibits biases toward those values. Therefore, practitioners should exercise caution and thoroughly examine their preference dataset to identify and address any potential biases.
20


Reviews:
[{"review_id": "NKytOC45oQ", "reviewer": "Reviewer_CbsJ", "summary": "This paper presents an algorithm called DPPO that can learn a policy from paired trajectory comparisons without having to learn a reward function or do online rollouts. PBRL algorithms typically learn a reward function with a Boltzmann comparison model and then optimize it with an online RL algorithm, like PPO. In contrast, DPPO uses a kind of pseudo-Boltzmann model expressed in terms of difference in similarity to the learned policy rather than difference in returns. In particular, if trajectory $a$ is preferred to trajectory $b$, then the learned policy $\\pi$ should be more similar to trajectory $a$, on average, than to trajectory $b$. DPPO is augmented with a BC-like regularizer that biases $\\pi$ towards having low average divergence with the trajectories (which prevents it from having consistently high divergence over all trajectories), as well as an auxiliary loss term for learning from unlabeled trajectories (essentially the same as the main loss but with a 0.5 label for unlabeled trajectories).\n\nExperiments on D4RL Gym, Adroit, and Franka Kitchen show DPPO typically obtains higher mean return than both an existing offline PBRL algorithm and two existing offline RL algorithms, even though the latter two have access to the ground truth reward function. This seems to be the case for both human and synthetic labels.", "questions": "In addition to responding to the weaknesses above, it would be helpful if the authors could answer these minor questions:\n\n- **[Q1]** Is Figure 2 illustrating reward or return? Also, what dataset is it using? I would not expect step-level rewards to be especially correlated given how many equivalent reward functions there are for Hopper (the goal is super simple—just run forward!), but I would expect return to be correlated on any sufficiently diverse dataset.\n- **[Q2]** What do the ranges and error bars in the tables represent? (std dev, std error, CIs, min/max, etc.)", "limitations": "**[L1]** Depending on the response to W2 above, it might be appropriate to expand the limitations with more discussion of when equation (2) is the right loss function to pursue. I expect this could tie in with the existing remarks about label noise—being more explicit about the assumptions behind equation (2) will probably reveal what kinds of label noise it can or can’t handle well.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": " - **[S1]** The need for online rollouts is a major practical obstacle to using reward-learning algorithms (e.g. IRL, PBRL) for control. Stable offline preference-based reinforcement learning algorithms would make it much easier to apply PBRL to systems where rollouts are expensive, since it is more difficult to tune such systems (examples include LLMs and real robots—although neither are used in this paper). Thus this paper is a welcome and timely addition to the existing toolkit of PBRL techniques.\n- **[S2]** I found the exposition clear, and the motivation for various parts of the algorithm was solid. The ablations were also adequate at demonstrating the need for the most complex parts of this algorithm (the regularizers/auxiliary losses that encourage low average policy divergence & help the algorithm learn from unlabeled demonstrations, respectively).\n- **[S3]** Empirical results are strong, especially for Adroit Pen and Kitchen. The experiments section also adequately explains why these specific benchmark environments were chosen over others in the D4RL suite, which is appreciated.", "weaknesses": "My main concern with this paper is that the baselines might be too weak, especially given that the method looks fairly similar to BC. It would also be nice to have more analysis of the policy deviation-based loss function. In more detail:\n\n- **[W1]** The results in Table 1 and Table 2 (Gym and Kitchen only) seem relatively weak. Table 1 of the [RvS paper](https://arxiv.org/pdf/2112.10751.pdf) shows a method that is essentially BC (RvS) getting comparable or better performance to DPPO on most of these tasks (see RvS-R column and RvS-G column—the former is reward-conditioned and the latter is goal-conditioned). Are these results applicable in this setting, and if so, why are CQL and IQL used instead of these stronger and simpler baselines?\n  \n  The regularizer in Equation (4) seems to have the effect of encouraging the policy to have lower average divergence from the labeled trajectories in general, which is vaguely BC-like, and so I think it’s particularly important to separate this method from BC-style methods (filtered BC is going to be one of the leading competitors to any PBRL method like this).\n- **[W2]** The loss function in Equation (2) is a significant departure from the traditional Boltzmann PBRL loss. It seems to be saying that demonstrators prefer trajectories that deviate less, on average, from the optimal policy, or something of that nature. The paper doesn’t spend much time exploring the implications of this change, or the degree to which the new algorithm is or isn’t equivalent to traditional PBRL. Having more explanation of this would be helpful—when is this the “right thing” to do? Are there any situations in which it’s obviously the wrong thing to do? (due to how the labeller might be judging trajectory fragments) etc. \n\nOn the whole I’d be fairly comfortable accepting this paper if the concern about baselines is addressed. I’m marking as a weak accept pending the response to the points above, but I’m leaning towards acceptance overall.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for their positive remarks and insightful feedback. We would like to address your comments below.\n\n**1. ... examples include LLMs and real robots—although neither are used in this paper ...**\n\nWe concur with the reviewer that domains such as LLMs and real robots would specifically benefit from PbRL. Accordingly, we have added an LLM RLHF experiment with DPPO in Appendix Section C of our paper. This experiment has yielded encouraging outcomes, indicating that DPPO can effectively supplant the reward-modeling plus PPO method typically employed in RLHF.\n\n**2. Comparison with BC-based offline RL algorithms.**\n\nThank you for the suggestion. We first note that RvS is not directly applicable to the PbRL context as RvS requires explicit reward or goal information, which is not provided in conventional PbRL scenarios. Additionally, our initial selection of IQL (and CQL) was based on the value-based methods adopted by the baseline PT. That said, we present the results of BC-based offline algorithms in the common response 4. These results demonstrate that our method markedly surpasses the BC-based techniques.\n\n**3. Clarification on Equation (2).**\n\nThank you for the valuable discussion on our learning objective. We would like to clarify that we continue to employ the Boltzmann PbRL loss as shown in Equation (5) (Preference Correctness term). The preference model optimizes the Boltzmann PbRL loss (supplemented with a smoothness regularizing term) with regard to the given preference labels. The core distinction between the reward-modeling baselines and our approach is that we do not model the preference as a weighted sum of rewards. Rather, we model the preference directly. Equation (2) represents the policy optimization objective used after earning the preference model.\n\n**4. Clarification on Figure 2.**\n\nFigure 2 illustrates the predicted reward vs. the true reward on the hopper-medium-replay dataset. Aligning with the reviewer's observation that the hopper environment is relatively simplistic and may be represented by numerous alternative reward functions, we performed a similar experiment on the much more complex pen-human dataset. The results are presented in Figure 1 (please check out the pdf file in the common response). The new results reveal that the baseline reward model remains ineffective at accurately capturing the true reward structure.\n\n\n**5. What do the ranges and error bars in the tables represent?**\n\nThe ranges and the error bars indicate the standard deviation regarding the random seeds (Table 1, 2, Figure 6, 7) or the environments (Figure 8). We will make this clearer in the updated manuscript."}, {"review_id": "uMaqPCHF0Y", "reviewer": "Reviewer_93eM", "summary": "This paper presents DPPO, a method for offline Preference-based Reinforcement Learning (PbRL). Rather than using an Offline RL algorithm, DPPO uses a supervised learning objective inspired from contrastive learning. A distance metric is defined, and the policy is trained to have lower distance with preferred segments than with un-preferred segments. To attain good performance, the authors learn a preference predictor (analogous to a reward model, but with the whole segment as input) with a smoothness regularizer and introduce a scale penalty to policy-segment distances, both of which correlate with performance. Their approach is evaluated on D4RL with preference labels collected from real humans.\n", "questions": "Major:\n1. **There appear to be inconsistent results in comparison to Preference Transformer**, the primary paper with which the authors compare. For example, Preference Transformer achieves a score of 84.5 on Hopper Medium Replay in their publication (higher than DPPO), but in this paper has a score of 59.7. The results are also off by 30 points for Walker Medium Replay! Why is this? This feels important to answer and address in the paper, especially since the authors in the appendix state to use the same codebase as PT.\n2. How would DPPO perform on the antmaze benchmark? (See weaknesses)\n3. How would other preference-based RL algorithms perform when using the preference smoothness regularizer (see Weaknesses)?\n\nMinor:\n1. Figure 2: “Reward model from PbRL” Which model is this? Is it the preference transformer or a Markovian model?\n2. Is it true that $d_{sa}$ and AGG must be differentiable? I believe it to be the case given the method. This should be explicitly stated!\n", "limitations": "The authors have very little discussion on the limitations of their approach. I would have liked to see an explicit section on limitations with some of the experiments in the Weaknesses section included. \n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper was very easy to read (I appreciate that!) Explanations of major topics were largely complete.\n\nThe idea of reformulating the PbRL problem using ideas from contrastive learning is interesting, and appears to perform well in practice for the chosen benchmarks. \n\nThat brings me to experimentation. I believe the authors did a good job providing results on a number of applicable benchmarks, and I appreciate the use of real human feedback. That being said, there are a few points I wish they addressed (see weaknesses), mainly related to increased understanding of why the proposed contrastive objective works well.\n\nThe authors also ablated both hyper-parameters they introduced. \n\nI believe that this has the potential to be a very strong paper. However, there are a few oversights that I would like to see addressed listed in the weaknesses section. If these concerns are addressed and questions answered, I would be happy to revise my score. \n", "weaknesses": "**Is Contrastive Learning Important?**\nI wish the authors included an ablation showing the importance of the contrastive learning part of DPPO versus the segment smoothing objective. The smoothness regularization introduced in Eq. 5 could be used with *any* preference based RL algorithm, and does not appear to be unique to “reward” modeling vs. “preference” modeling. It would be great if the authors could include an ablation that trains PT-IQL with the preference smoothness regularizer. That would show that the performance of DPPO is due to the unique choice of contrastive objective, instead of the smoothness regularizer which could be applied to any reward-modeling algorithm.\n\n**Limitations of DPPO**\nI found the discussion of limitations and experiments surrounding limitations to be lacking. The authors opt to use a contrastive objective instead of approximate dynamic programming, and did not evaluate on any datasets where stitching is necessary. Supervised RL objectives (AWR, contrastive RL, GCSL etc.) do well when great policies can be learned with a single step of policy improvement, or when trajectory stitching isn’t necessary. On most of the chosen benchmarks, particularly Androit and Franka Kitchen, BC can do surprisingly well! For example, a score of 54 can be achieved on Androit pen cloned with just BC. This trend holds for many datasets (See Table 2 of https://arxiv.org/pdf/2004.07219.pdf). At the same time, the authors do not include results on AntMaze, which is known to require trajectory stitching, and was also included in the PreferenceTransformer paper to which the authors directly compare. \n\nI want to emphasize that it is OK if DPPO does not do as well as IQL-based PbRL algorithms on AntMaze. Including these results and an enlightening discussion on when DPPO will perform better or worse would incline me to consider raising my score, not lower it.\n\n**Modeling Rewards vs. Preferences**\n\nPersonally, I think the framing of the paper to be potentially mis-leading, specifically the part about it being “without reward modeling”. Yes, the authors do not have a “reward” model, but they still use a preference predictor with the same architecture as Preference Transformer (Appendix B). In fact the distinction between modeling a non-Markovian Reward (CITATION) and “modeling preferences” seems small: both have a neural network that takes in a state action segment and produces scores – the difference being that one is at each step (reward) and one is at the end (preference). It would be great if the authors could place more emphasis on why this difference matters. \n\n**Related Work**\nI found the related work section to be lacking. In particular, the authors omit several foundational works in preference-based RL [1,2,3,4,5], and more recent works as well [6,7,8]. I’ve included an incomplete number of examples.\n\nSecond, I will not penalize the authors for this since these works were made public after submission, but two highly related works have recently been released. [9], published at ICML also takes inspiration from contrastive learning and applies it to offline PbRL. [10], a preprint, removes the reward network from preference-based RL algorithms. \n\n[1] A. Wilson, A. Fern, and P. Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://proceedings.neurips.cc/paper/2012/file/ 16c222aa19898e5058938167c8ab6c57-Paper.pdf\n\n[2]  D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference-based learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), July 2017. doi: 10.15607/RSS.2017.XIII.053.\n\n[3]  J. R. Lepird, M. P. Owen, and M. J. Kochenderfer. Bayesian preference elicitation for multiobjective engineering design optimization. Journal of Aerospace Information Systems, 12(10): 634–645, 2015.\n\n[4] E. Biyik, N. Huynh, M. J. Kochenderfer, and D. Sadigh. Active preference-based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.\n\n[5] W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning, pages 292–297. IEEE, 2008.\n\n[6] Joey Hejna and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning, 2022\n\n [7] Liu, Yi, et al. \"Efficient preference-based reinforcement learning using learned dynamics models.\" arXiv preprint arXiv:2301.04741 (2023).\n\n[8] Daniel Shin and Daniel S Brown. Offline preference-based apprenticeship learning. arXiv preprint arXiv:2107.09251, 2021 \n\n[9] Kang, Yachen, et al. \"Beyond reward: Offline preference-guided policy optimization.\" arXiv preprint arXiv:2305.16217 (2023).\n\n[10] Hejna, Joey, and Dorsa Sadigh. \"Inverse Preference Learning: Preference-based RL without a Reward Function.\" arXiv preprint arXiv:2305.15363 (2023).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our appreciation to the reviewer for the positive remarks and insightful feedback. We would like to address your comments below.\n\n**1. Is contrastive learning important?**\n\nIn common response 3, we incorporated the baseline algorithms' evaluation results when combined with our smoothness regularizer. Although our smoothness regularizer elevates the baselines' performance, DPPO still outperforms them by a large margin. This observation underscores that the process of direct policy optimization by leveraging preference information is pivotal to achieving high performance.\n\n**2. Experiments on Antmaze**\n\nWe acknowledge that many offline RL studies highlight Antmaze results to demonstrate the efficacy of their methods in more challenging environments. Yet, it must be noted that the official implementation of Antmaze suffers from a serious flaw in the procedure of goal setting (https://github.com/Farama-Foundation/D4RL/issues/142). Concretely, the critical bug in Antmaze is that the offline dataset assumes a multi-goal setting, where a random goal location is chosen at the start of each episode, while the actual environment (unintentionally) uses a single fixed goal for every episode. Consequently, the optimal policies with regard to the offline dataset and the environment differ: According to the offline dataset, the optimal policy is to fastly sweep through all the goal candidates, while within the \n environment context, the optimal policy is to move directly to the solitary fixed goal.\n\nTo fix this, we can modify the Antmaze environment to choose a random goal location for each episode, thereby aligning with the provided offline dataset. Surprisingly, after this modification, we observe that offline RL algorithms struggle to achieve meaningful performance, even when provided with true reward information:\n\n|  | filtered BC | CQL | IQL |\n|---|---|---|---|\n| antmaze-medium-play-v2 | 10.5 $\\pm$ 2.9 | 9.8 $\\pm$ 4.8 | 1.5 $\\pm$ 1.5 |\n| antmaze-medium-diverse-v2 | 10.4 $\\pm$ 3.2 | 3.1 $\\pm$ 3.3 | 1.8 $\\pm$ 0.8 |\n| antmaze-large-play-v2 | 11.1 $\\pm$ 1.4 | 5.5 $\\pm$ 3.1 | 14.5 $\\pm$ 4.7 |\n| antmaze-large-diverse-v2 | 8.0 $\\pm$ 0.8  | 4.8 $\\pm$ 1.3 | 11.2 $\\pm$ 1.6 |\n\nThis is the reason why we did not include the Antmaze results in our paper. We perceive this flaw in the benchmark as a substantial concern for the offline RL community, especially considering that many existing offline RL studies continue to utilize this erroneous version of Antmaze. We plan to incorporate this discussion in our revised manuscript.\n\nThat said, we provide the offline PbRL results on the fixed version of Antmaze below:\n\n|  | PT+CQL | PT+IQL | DPPO (Ours) |\n|---|---|---|---|\n| antmaze-medium-play-v2 | 3.3 $\\pm$ 4.7 | 3.2 $\\pm$ 4.4 | 1.8 $\\pm$ 1.8 |\n| antmaze-medium-diverse-v2 | 0.0 $\\pm$ 0.0 | 1.5 $\\pm$ 1.5 | 0.0 $\\pm$ 0.0 |\n| antmaze-large-play-v2 | 0.0 $\\pm$ 0.0 | 13.5 $\\pm$ 5.4 | 10.0 $\\pm$ 9.4 |\n| antmaze-large-diverse-v2 | 0.0 $\\pm$ 0.0 | 6.7 $\\pm$ 5.8 | 7.5 $\\pm$ 9.6 |\n\n**3. Modeling Rewards vs. Preferences.**\n\nThank you for the feedback. Please refer to the common response 1.\n\n**4. Related work.**\n\nThank you for the feedback. Please refer to the common response 2.\n\n**5. Inconsistent results on PT.**\n\nWe did use the authors' official implementation (https://github.com/csmile-1006/PreferenceTransformer) to obtain the PT results for our paper. However, as highlighted in this issue (https://github.com/csmile-1006/PreferenceTransformer/issues/3), reproducing the performance of PT is challenging. Furthermore, as per the author's remark, some other less significant problems exist, such as sensitivity to the selection of random seeds. We conjecture that these factors contribute to the inconsistent results on PT when compared to the original paper. We will include this information and additionally provide the original paper's results in the updated manuscript.\n\n**6. Which model was used for Figure 2?**\n\nAs noted in the figure's caption, we used PT to produce the results.\n\n**7. Is it true that $d_{\\text{sa}}$ and $\\text{AGG}$ must be differentiable?**\n\nAs we currently use a gradient descent-style optimizer (Adam) to optimize the policy model, $d_{\\text{sa}}$ and $\\text{AGG}$ should be differentiable. However, should we adopt $d_{\\text{sa}}$ and $\\text{AGG}$ such that differentiation is impossible, we could utilize other gradient-free optimization methods (e.g., evolutionary algorithms, gradient approximation, ...). We believe that this extension would be a valuable direction for future research! We will include this discussion in the updated manuscript.\n"}, {"review_id": "8ABsv1hYZQ", "reviewer": "Reviewer_MTAP", "summary": "In the context of Preference-based RL (PbRL), the authors propose to learn a preference model that ranks trajectory segments without assuming that the model decomposes into a sum of rewards in an MDP sense. The motivation behind this choice is that the human teachers themselves might not follow a model of the sort, and hence modeling these preferences can be made simpler and more accurate without the usual reward decomposition assumption in PbRL. \n\nUsing this preference prediction model, the authors define a non-learned policy scoring function for ranking policies based on average similarity in action with trajectory segments in a dataset. The overall method is evaluated in an offline setting with a mix of labeled and unlabeled data. The preference model is trained on the labeled data, and used to predict preferences over the unlabeled dataset. Using these predictions, a policy optimizing the scoring function on the unlabeled dataset is returned. Comparisons with a recent baseline was provided on several robotics tasks.\n", "questions": "- If hat{P} is based on PreferenceTransformer, is it still learning a (history dependent) reward model? Why hat{P} was not formally defined in the paper?\n- How would you perform policy optimization in the absence of offline data? How will this offline data be generated in a given RL problem?\n- If your approach and the baseline use the same preference model, what contributes the most to the improved performance? The added regularizers or the optimization of the scoring function instead of IQL/CQL\n- In the appendix you state that policy optimization is \"following the experimental protocol of IQL\". What does this mean?", "limitations": "A limitation section is included but the paper does not discuss the counter part of not having a reward model and not being able to use RL for policy optimization, which seems important to me. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The hypothesis of many PbRL papers that humans provide preferences according to a Markovian reward might be too strong, and it's important to have papers exploring the validity of this hypothesis\n- The method shows improvements over a recent baseline on several tasks by using a very simple policy scoring function\n", "weaknesses": "- Some references are missing, for example in line 80 the authors state that \"While prior PbRL approaches adopt a two-step procedure involving the construction of a reward (...)\", whereas there are in fact PbRL papers that follow a very similar idea of using trajectory distances to compare policies without a reward model, e.g. Wilson et al., A Bayesian Approach for Policy Learning from Trajectory Preference Queries, NeurIPS2012.\n\n- While the main claim of the paper is to introduce a new model for predicting trajectory preferences that is not based on rewards, I was very surprised that a formal definition of this model is never provided in the paper! In the appendix, we learn that the implementation is based on PreferenceTransformer [25] that is in fact decomposing the score as a sum of (history dependent) rewards, so this is a bit confusing. \n\n- The experiments are limited to the offline setting, which is fine, but then it becomes unfair to compare to methods that assume a reward decomposition structure of the preferences because while this assumption might be restrictive, in exchange RL can be used to collect new data by optimizing the current reward model. In contrast, it is unclear how well the Direct Preference-based Policy Optimization method would work in the absence of offline data, since simply maximizing the scoring function on the labeled data will probably amount to behavioral cloning and offer very little in terms of exploration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer for the inspiring remarks and valuable feedback. We would like to address your comments below.\n\n**1. Some references are missing.**\n\nThank you for pointing out an important previous work related to our paper. For a detailed discussion on related works, please refer to common response 2.\n\n**2. Clarification of the preference model.**\n\nThank you for the feedback. For an in-depth explanation of our preference model, please refer to common response 1.\n\n**3. it is unclear how well the Direct Preference-based Policy Optimization method would work in the absence of offline data.**\n\nDrawing from the work of [1], our algorithm can be adapted to the online context by incorporating entropy-maximization terms [2]. More specifically, we can modify our policy optimization objective in Equation (2) by introducing a regularizing term that maximizes the entropy of the output distribution of the current policy. We evaluated this modified version of DPPO on the offline training-online finetuning setting. To elaborate, we run DPPO using only the offline data for 1 million gradient updates and proceed with online fine-tuning through 0.2 million online interactions. The outcomes for both DPPO and PT+IQL are presented below:\n\n|  | IQL (offline) | IQL (0.2m) | DPPO (offline) | DPPO (0.2m) |\n|---|---|---|---|---|\n| pen-human | 53.0 $\\pm$ 31.7 | 60.1 $\\pm$ 15.6 | 76.3 $\\pm$ 14.4 | **82.5 $\\pm$ 5.3** |\n| pen-cloned | 42.9 $\\pm$ 24.4 | 40.4 $\\pm$ 30.2 | 75.1 $\\pm$ 7.7 | **85.3 $\\pm$ 8.4** |\n| Average | 48.0 | 50.3 | 75.7 | **83.9** |\n| kitchen-mixed | 48.0 $\\pm$ 11.9 | 46.7 $\\pm$ 7.8 | 52.5 $\\pm$ 3.1 | **54.2 $\\pm$ 1.4** |\n| kitchen-cloned | 40.2 $\\pm$ 12.3 | **49.7 $\\pm$ 5.4** | **49.4 $\\pm$ 5.7** | **48.3 $\\pm$ 6.2** |\n| Average | 44.1 | 48.2 | **51.0** | **51.3** |\n\nThe findings indicate that DPPO is well-suited for deployment in online fine-tuning environments. Additionally, we'd like to highlight that experiments on LLM RLHF are included in Appendix Section C, representing another instance of an online fine-tuning context.\n\n**4. what contributes the most to the improved performance? The added regularizers or the optimization of the scoring function instead of IQL/CQL**\n\nThank you for the feedback. As analyzed in common response 3, we found that our smoothness regularizer also positively influences all the considered methods. However, the highest performance is attained through direct policy optimization that incorporates preference information.\n\n**5. \"following the experimental protocol of IQL\". What does this mean?**\n\nTo clarify, this statement note that we adopted the same neural network architecture and the optimizer from IQL. We will make this clearer in the updated manuscript.\n\n[1] Zheng et al., Online Decision Transformer, ICML 2022.\n\n[2] Haarnoja et al., Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, ICML 2018."}, {"review_id": "yQPkUjFMP2", "reviewer": "Reviewer_E68m", "summary": "This paper investigates learning a policy directly from preferences, i.e., pairwise comparison between trajectory segments. By defining a metric between policy and trajectory segments, the authors show that this problem resembles contrastive learning problems, and they propose to learn a policy by minimizing the distance with preferred trajectory segments. The author verified the proposed approach using D4RL benchmarks.", "questions": "1. I would suggest the authors compare the proposed approach with [1] to show the efficacy of their method for bypassing reward learning.\n\n2. The comparison with [2,3] is also relevant, especially for datasets with high quality.\n\n3. While the connection between PbRL and contrastive learning is interesting, the objective (eq. (4)) is an extension of the Bradley—Terry model. In other words, the output policy is not pulled away from MULTIPLE negative samples. Since the authors made use of unlabeled trajectories in a preference labeler (eq (5)), I think it is insightful to discuss a more close utilization of contrastive learning for PbRL.\n\n4. I would suggest the authors discuss the connection between their approach and behavior cloning. ", "limitations": "The authors discussed some technical limitations of their approach, but there lacks a discussion for broader impact or social impact.", "rating": 7, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The storytelling is concise and relevant. Materials are organized in a way that facilitates reading.\n2. The proposed policy scoring metric is intuitive and novel. ", "weaknesses": "1. Discussion and comparison with some very important alternative methods are missing, which weakens the significance of this paper.\nFor example, the problem of learning policies from preferences directly was studied by Liu et al. [1] before. Although they carried out their experiments in an online PbRL setting, it is straightforward to use their approach in the setting of this paper. Moreover, the proposed approach utilizes both offline data and preferences in learning. Two studies in the literature [2,3] also work in the same setting. Although the authors mentioned [2] in the related work section, they did not include results for the approach proposed in [2] or [3]. Importantly, the proposed approach outperforms the preference transformer mainly on medium-expert datasets, which are of high quality. Missing comparison with the imitation-based approach [2,3] undermines the significance of experiments.\n\n2. There lack of discussion on the connection between the proposed approach and behavior cloning. In fact, it was not straightforward for me to understand why minimizing a distance metric leads to a policy when reading sec 3.2 for the first time. It became clear after I realized the connection between $$d(\\pi,\\sigma^i)$$ and the objective of behavior cloning.\n\n\n[1] Meta-Reward-Net: Implicitly Differentiable Reward Learning for Preference-based Reinforcement Learning, NeurIPS 2022.\n\n[2] Reward learning from human preferences and demonstrations in Atari, NeurIPS 2018.\n\n[3] Conﬁdence-Aware Imitation Learning from Demonstrations with Varying Optimality, NeurIPS 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the encouraging comments and constructive feedback. We would like to address your comments below.\n\n**1. Discussion and comparison with some very important alternative methods are missing**\n\nThank you for pointing out some relevant baselines for our work. While [1] does model the preference directly (using the Q-value as the substitute for the preference score), it maintains an assumption that an underlying per-step reward structure is present and leans on this structure to facilitate preference learning. We made some minor modifications to [1, 2] to adapt them to the offline context (e.g., changing the policy optimization algorithm from SAC to IQL on [1]), and present the results below:\n\n|  | [1] | [2] | DPPO (Ours) |\n|---|---|---|---|\n| pen-human | 57.5 $\\pm$ 18.3 | 38.6 $\\pm$ 12.8 | **76.3 $\\pm$ 14.4** |\n| pen-cloned | 45.3 $\\pm$ 26.5 | 30.8 $\\pm$ 24.3 | **75.1 $\\pm$ 7.7** |\n| Average | 51.4 | 34.7 | **75.7** |\n| kitchen-mixed | 38.8 $\\pm$ 13.8 | 23.8 $\\pm$ 2.5 | **52.5 $\\pm$ 3.1** |\n| kitchen-cloned | 33.8 $\\pm$ 7.5 | 38.8 $\\pm$ 18.5 | **49.4 $\\pm$ 5.7** |\n| Average | 36.3 | 31.3 | **51.0** |\n\nThe results show that DPPO significantly outperforms the considered baselines.\n\n(Note: Regarding [3], we discovered that the official implementation's package configuration is not compatible with the Adroit and Kitchen environments. We plan to address this matter during the rolling discussion phase.)\n\n**2. lack of discussion on the connection between the proposed approach and behavior cloning**\n\nThank you for the feedback. Broadly speaking, our proposed method can be thought of as imitating the behavior from more desirable trajectory segments while distancing itself from the behavior in less desirable segments. We will make this clearer in the updated manuscript. In addition, we have conducted an empirical comparison with some of the BC-based methods, such as RvS and filtered BC, in common response 4. The findings reveal that DPPO substantially excels over the BC-based PbRL approaches.\n\n**3. In other words, the output policy is not pulled away from MULTIPLE negative samples**\n\nThank you for the insightful comment. We would like to point out that the decision to use a single negative sample was driven by computational efficiency. Our findings indicate that the number of negative samples doesn't significantly influence empirical performance (we plan to include the full comparison results as soon as possible). We will expand our discussion on the contrastive learning formulation in the updated manuscript. Meanwhile, could the reviewer be so kind as to elaborate further on what you were referring to with the statement `Since the authors made use of unlabeled trajectories using a preference labeler (eq (5))`?\n\n**4. lacks a discussion for broader impact or social impact.**\n\nThank you for your observation. Due to the constraints on page length, we placed the broader impact section in the appendix (section E). We will incorporate this section into the main body of the updated manuscript."}]

Author rebuttal:
We are grateful to the reviewers for their supportive and helpful feedback. We plan to address some of the shared concerns raised by multiple reviewers in this section, while individual responses will deal with more specific issues. Moreover, we'd like to emphasize that section C of our appendix showcases an LLM RLHF experiment with DPPO. This experiment reveals an encouraging outcome, demonstrating that our algorithm can successfully fine-tune LLMs, replacing the traditionally employed PPO algorithm.

**1. Clarification on the difference between the preference model from DPPO and the reward model from PT (PreferenceTransformer).**

We clarify that our preference model does not calculate the preference score (model output) as a weighted sum of some scalar value for each timestamp. Concretely, we design our transformer to output a hidden embedding for each timestamp, compute the mean of these hidden embeddings, and then pass the mean embedding through an MLP layer to generate a scalar preference score. Note that this kind of structure is common for tasks such as text classification. The only major resemblance between our preference model and the reward model from PT is that we use GPT-2 as the foundational transformer architecture to ensure a fair comparison with the baseline regarding model capacity. 

**2. Some related works are missing.**

We thank the reviewers for highlighting some important works aligning with our method. In particular, [1] bears similarity to our work as it leverages the preference information to directly optimize the policy using the idea of trajectory distance. However, a limitation of [1] is that it requires the two trajectories being compared to (1) start from the same initial state and (2) roll out from the current policy being trained, which makes it challenging to utilize extensive pre-collected offline data. We will also incorporate other related works recommended by the reviewers in our revised manuscript, thereby enriching the related works section to adequately address the relevant literature.

**3. Applying the smoothness regularizer to the baseline methods.**

In recognition of the fact that our novel smoothness regularizer can be integrated into any reward-modeling baselines, we assess how its application to the baselines would influence the overall performance.

|  | PT+CQL | PT+CQL+reg | PT+IQL | PT+IQL+reg | DPPO (Ours) |
|---|---|---|---|---|---|
| pen-human | 31.6 $\pm$ 3.3 | 18.3 $\pm$ 17.2 | 53.0 $\pm$ 31.7 | 53.7 $\pm$ 42.3 | **76.3 $\pm$ 14.4** |
| pen-cloned | 18.3 $\pm$ 10.6 | 32.7 $\pm$ 11.2 | 42.9 $\pm$ 24.4 | 49.8 $\pm$ 32.2 | **75.1 $\pm$ 7.7** |
| Average | 25.0 | 25.5 | 48.0 | 51.8 | **75.7** |
| kitchen-mixed | 12.3 $\pm$ 7.7 | 12.0 $\pm$ 5.0 | 48.0 $\pm$ 11.9 | 49.4 $\pm$ 5.2 | **52.5 $\pm$ 3.1** |
| kitchen-partial | 14.1 $\pm$ 13.0 | 11.4 $\pm$ 11.2 | 40.2 $\pm$ 12.3 | **49.4 $\pm$ 5.2** | **49.4 $\pm$ 5.7** |
| Average | 13.2 | 11.7 | 44.1 | 49.4 | **51.0** |

Although our smoothness regularizer does enhance the baseline methods' performance, this improved performance still lags behind our proposed method. This outcome suggests that the process of directly optimizing the policy through the use of preference information plays a critical role in the overall performance.

**4. Comparison with BC-based offline RL algorithms.**

Here, we compare our approach with certain BC-based offline algorithms, such as RvS [2] and filtered BC. Drawing from PT+IQL or PT+CQL, we modify the RvS algorithm to utilize the reward model from PT for predicting rewards for each state-action pair. These predicted rewards are then used to run RvS, referred to as PT+RvS. Along with PT+RvS, we also implement PT+filtered BC by using the predicted rewards from PT to rank the full trajectories and perform BC on the top 10% of the trajectories.

|  | PT+RvS | PT+filtered BC | DPPO (Ours) |
|---|---|---|---|
| pen-human | -1.8 $\pm$ 0.5 | 19.4 $\pm$ 6.7 | **76.3 $\pm$ 14.4** |
| pen-cloned | -2.2 $\pm$ 0.3 | 37.4 $\pm$ 7.5 | **75.1 $\pm$ 7.7** |
| Average | -2.0 | 28.4 | **75.7** |
| kitchen-mixed | 27.5 $\pm$ 9.4 | 40.9 $\pm$ 9.0 | **52.5 $\pm$ 3.1** |
| kitchen-partial | 26.0 $\pm$ 4.9 | **53.4 $\pm$ 9.0** | **49.4 $\pm$ 5.7** |
| Average | 26.8 | 47.2 | **51.0** |

The results show that DPPO outperforms the BC-based baselines on all tasks, with the performance gap being especially large on the challenging pen tasks.

[1] Wilson et al., A Bayesian approach for policy learning from trajectory preference queries, NeurIPS 2012.

[2] Emmons et al., What is Essential for Offline RL via Supervised Learning?, ICLR 2022.


Meta-review: The meta-reviewer has reviewed the paper, the reviews, the responses, and agrees with the majority of the reviewers that this paper meets the NeurIPS standard.
There is a large number of Preference-based RL papers coming out recently. This paper is a borderline accept in terms of novelty and significance.
Decision: Accept (poster)