Preference Transformer: Modeling Human Preferences using Transformers for RL
Authors: Changyeon Kim
Jongjin Park
Jinwoo Shin
Honglak Lee
Pieter Abbeel
Kimin Lee
Venue: ICLR 2023 poster (2023)
Keywords: preference-based reinforcement learning
human-in-the-loop reinforcement learning
deep reinforcement learning

Abstract: Preference-based reinforcement learning (RL) provides a framework to train agents using human preferences between two behaviors. However, preference-based RL has been challenging to scale since it requires a large amount of human feedback to learn a reward function aligned with human intent. In this paper, we present Preference Transformer, a neural architecture that models human preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non-Markovian rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention layers. We demonstrate that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer can induce a well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal dependencies in human decision-making. Code is available on the project website: https://sites.google.com/view/preference-transformer.

Published as a conference paper at ICLR 2023
PREFERENCE TRANSFORMER: MODELING HUMAN PREFERENCES USING TRANSFORMERS FOR RL
Changyeon Kim Jongjin Park Jinwoo Shin' Honglak Lee Pieter Abbeel Kimin Lee
KAIST Uniiversity of Michigan SL.G AT Research 'UC Berkeley "Google Research
ABSTRACT
Preference-basel reinforcement learning (RL) provides a framework to train agents using human preferences between behaviors. However, preference- based RL has been challenging to scale since requires a large amount of hu- man feedback learn a reward function aligned with human intent. In this pa- per, we present Preference Transformer, a neural architecture that models hu- man preferences using transformers. Unlike prior approaches assuming human judgment is based on the Markovian rewards which contribute to the decision equally, we introduce a new preference model based on the weighted sum of non Mar-vaako rewards. We then design the proposed preference model using a transformer architecture that stacks causal and bidirectional self-attention lay- ers. We demonstrate that Preference Transformer tasks using real human preferences, while prior approaches fail to work. We also show that Preference Transformer induce well-specified reward and attend to critical events in the trajectory by automatically capturing the temporal de- pendencies in human decision-making. Code is available on the project website: https:: /tttss///sstsseeee google. com/ciee/////e www.w/ewwwweeeeeeeeeeeeeeeeeeeee wwtp://eefeeeeeeeeeeeee trans former.
solve variety of control
INTRODUCTION
Reinforcement learning (RL) has been successful in solving sequential decision-making problems in various domains where a suitable reward function is available (Mnih al., 2015; Silver et al., 2017: Berner et al., 2019; Vinyals et al., 2019). However, reward engineering poses number of challenges. It often requires extensive instrumentation (e.g., thermal cameras (Schenck Fox, 2017), accelerometers (Yahya et al., 2017), or motion trackers (Peng et al., 2020)) to design dense and precise reward. Also, is hard to evaluate the quality of outcomes in a single scalar since many problems have multiple objectives. For example, need to care about many objectives like velocity, energy spent, and torso verticality to achieve stable locomotion (Tassa et al., 2012; Faust al., 2019). It requires substantial human effort and extensive task knowledge aggregate multiple objectives into a single scalar.
To avoid reward engincering, there various ways to learn the reward function from human data, such as real-valued feedhack (Knox & Stone, 2009; Daniel et al., 2014), expert demonstrations (Ng al., 2000; Abbeel & Ng. 2004), preferences (Akrour et al., 2011; Wilson al., 2012; Sadigh et al., 2017) and language instructions (Fu et al., 2019; Nair et al., 2022). Especially, research interest in preference-based RL (Akrour al., 2012: Christiano et al., 2017: Lee et al., 2021b) has increased recently since making relative judgments (e.g., pairwise comparison) is easy to provide yet information-rich By learning the reward function from human preferences between trajectories, recent work has shown that the agent can learn novel behaviors (Christiano et al., 2017; Stiennon al., 2020) or avoid reward exploitation (Lee et al., 2021b).
However, existing approaches still require large amount of human feedhack, making it hard to scale up preference-based RI various applications. We hypothesize this difficulty originated from common underlying assumptions in preference modeling used in most prior work. Specifically, prior work commonly assumes that (a) the reward function is Markovian (i.e., depending only on the current state and action), and (b) human evaluates the quality of a trajectory (agent's behavior) based on the sum of rewards with equal weights. These assumptions can be flawed due to the
"Equal contribution. Correspondence to: (changyeon.c kim, jongj -park] (kk at..

---

Published as a conference paper at ICLR 2023
Preference model: Pla' > 0d)
Preference
Preference Transformer
Preference Transformer
s
a
2
s
a
S
a
Figure 1: Illustration of framework. Given preference between two trajectory segments (o", oo)) Preference Transformer generates non-Markovian rewards é, and their importance weights Wy over each segment. We then model the preference predictor based on the weighted sum of non- Markovian rewards (i.e., >; ) and align it with human preference.
following reasons. First, there are various tasks where rewards depend on the visited states (i.e., non- Markovian) since hard to encode all task-relevant information into the state (Bacchus et al., 1996; 1997). This can be especially true in preference-based learning since the trajectory segment provided to the human sequentially (e.g., video clip (Christiano et al., 2017; Lee et al., 2021b)), enabling earlier events to influence the ratings of later ones. In addition, since humans are highly sensitive to remarkable moments (Kahneman, 2000), credit assignment within the trajectory is required. For example, in the study of human attention video games using eye trackers (Zhang et al., 2020), the human player requires a longer reaction time and multiple eye movements on important states that can lead to large reward penalty, in order to make decision.
In this paper, we aim to propose an alternative preference model that can overcome the limitations of common assumptions in prior work. To this end, introduce the new preference model based on the weighted sum of non-Markovian rewards, which can capture the temporal dependencies in human decisions and infer critical events in the trajectory. Inspired by the recent success of trans- formers ( (Vaswani et al., 2017) in modeling sequential data (Brown et al., 2020; Chen et al., 2021), we present Preference Transformer, transformer-bas based architecture for designing the proposed preference model (see Figure 1). Preference Transformer takes trajectory segment as input, which allows extracting task-relevant historical information. By stacking bidirectional and causal self- attention layers, Preference Transformer generates non-Markovian rewards and importance weights as outputs. We then utilize them to define the preference model.
We highlight the main contributions of this paper below:
We propose a more generalized framework for modeling human preferences based on a weighted sum of non-Markovian rewards.
We present Preference Transformer, a transformer basedd architecture that consists of our novel preference attention layer designed for the proposed framework.
Preference Transformer enable RL agents to solve complex navigation, locomotion tasks from D4RL (Fu al., 2020) benchmarks and robotic manipulation tasks from Robomimic (Man- dlekar et al., 2021) benchmarks by learning a reward function from real human preferences.
We analyze the learned reward function and importance weights, showing that Preference Transformer can induce a well-specified reward and capture critical events within a trajectory.
RELATED WORK
Preference-basee reinforcement learning. Recently, various methods have utilized human prefer- ences to train RL agents without reward engineering (Akrour et al., 2012; Christiano et al., 2017; Ibarz. et al., 2018; Stiennon et al., 2020; Lee al., 2021b;c; Nakano et al., 2021; Wu et al., 2021: III & Sadigh, 2022; Knox et al., 2022; Ouyang et al., 2022; Park et al., 2022; Verma & Metcalf, 2022). Christiano et al. (2017) showed that breference-based RL can effectively solve complex con- trol tasks using deep neural networks. To improve the feedback-efffcienccy several methods, such as pre-training (Ibarz et al., 2018: Lee et al., 2021b), data augmentation (Park et al., 2022), explo- ration (Liang et al., 2022), and meta-learning (III & Sadigh, 2022), have been proposed. Preference- based RL also has been successful in fine-tuning large-scale language models (such as GPT-3; Brown al. 2020) for hard tasks (Stiennon et al., 2020; Wu et al., 2021; Nakano et al., 2021; Ouyang al.,
2

---

Published as a conference paper at ICLR 2023
2022). Transformer- bassrd models (i..., pre-trained language models) used reward function in these approaches due to partial observability of language inputs, but we utilize transformers with new preference modeling for control tasks.
Transformer for reinforcement learning and imitation learning. Transformers (Vaswani et al., 2017) have been studied for various purposes in RL (Vinyals al., 2019; Zambaldi et al., 2019; Parisotto et al., 2020; Chen et al., 2021; Janner et al., 2021). It has been observed that sample- efficiency and generalization ability can be improved by modeling RL agents using transformers in complex environments, such as StarCraft (Vinyals et al., 2019; Zambaldi et al., 2019) and DMLab- 30 (Parisotto al., 2020) benchmarks. For offline RL, Chen al. (2021) and Janner al. (2021) also leveraged the transformers by formulating RL problems in the context of the sequential modeling problem. Additionally, transformers have been applied successfully in imitation learning (Dasari & Gupta, 2020; Mandi et al., 2022; Reed et al., 2022). Dasari & Gupta (2020) and Mandi et al. (2022) demonstrated the generalization ability of transformers in one-shot imitation learning (Duan et al., 2017), and Reed et al. (2022) utilized the transformers for multi-domain and multi-task imi- tation learning. In this work, we demonstrate that transformers also can be useful in improving the efficiency of preference- based learning.
Non-Markovian reward learning, Non-Markovian reward (Bacchus et al., 1996; 1997) has been studied for dealing with the realistic reward setting in which reward depends the visited states, or reward is delayed or even given at the end of each episode. Several recent work focuses on return decomposition where an additional model is trained under the objective of predicting trajec- tory return with a given state-action sequence and used for reward redistribution. Arjona-Medina al. (2019) and Early et al. (2022) used LSTM (Hochreiter & Schmidhuber, 1997) for capturing sequential information in reward learning. Gangwani et al. (2020) and Ren et al. (2022) proposed simplified approaches without any prior task knowledge, under the assumption that episodic return is distributed uniformly in the timestep. In this work, we adopt transformer based reward model for learning Markovaov rewards, which can capture the temporal dependencies in human decisions.
PRELIMINARIES
We consider the reinforcement learning (RL) framework where an agent interacts with an environ- ment in discrete time (Sutton & Barto, 2018). Formally, at each timestep I, the agent receives the current state s, from the environment and chooses action a, based its policy π. The environ- ment gives a reward r(s;, at), and the agent transitions to the next state St+1- The goal of RL is to learn policy that maximizes the expected return, R₂ A== r at+k), which is defined as a discounted cumulative sum of the reward with discount factor y..
In many applications, it is difficult to design a suitable reward function capturing human intent. Preference-based RL (Akrour et al., 2011; Pilarski et al., 201 I; Wilson et al., 2012: Christiano et al., 2017) addresses this issue by learning reward function from human preferences. Similar to Wilson et al. (2012) and Christiano et al. (2017), we consider preferences over two trajectory segments of length H, σ {(s1, a1); ..., (SH, a Given a pair of segments o"),a"), (human) teacher indicates which segment is preferred, i.c., E 1, 1, The preference label y = 1 indicates a! > 0o, y = 0 indicates の > I², and y = 0.5 implies an equally preferable case, where o² > o² denotes the event that the segment is preferable to the segment j. Each feedback is stored in dataset of preferences as a triple (a", (.)))))
To obtain reward function é parameterized by v. most prior work (Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021b;c; & Sadigh, 2022; Park et al., 2022) defines a preference predictor following the Bradley-Terry model (Bradley & Terry, 1952):
exp (E, (() at:
Pla o
exp (E, f(s), (b)) + exp (E, f(s], al: +))"
Then, given dataset of preferences D. the reward function f is updated by minimizing the cross- entropy loss between this preference predictor and the actual human labels:
CE
E
[1 v) log P(a" > a o'; + y log Pla" > 06 :i]
(2)
(o WWDD
One can update a policy π using any RL algorithm such that it maximizes the expected returns with respect the learned reward.
3

---

Published as a conference paper at ICLR 2023
PI-
Σ w
MatMul
SottMax
Scale
Preference Allention Layer (bidirectional)
MalMul
{qt
# h--1
{Kk
[KeTeml
Causal Transformer
Linear
a
Figure 2: Overview of Preference Transformer. We first construct hidden embeddings fx,} through the causal transformer, where each represents the context information from the initial timestep to timestep 1. The preference attention layer with a bidirectional self-attention computes the non- Markovian rewards {fr} and their convex combinations {&} from those hidden embeddings, then aggregate } for modeling the weighted sum of non-Markovian rewards Σ. Wyfr.
PREFERENCE TRANSFORMER
In this section, we present Preference Transformer (PT), the transformer architecture for modeling human preferences (see Figure 2 for the overview). First, we introduce new preference predictor Plo' > d0] based on a weighted sum of non- Mor-Maakovian rewards Section 4.1, which can reflect the long-term context of the agent's behaviors and capture the critical events in the trajectory segment. We then describe a novel transformer-basee architecture model the proposed preference predictor in Section 4.2.
4.1 PREFERENCE MODELING
As mentioned in Section most prior work assumes that the reward Markovian (i.e., depending only current state and action), and human evaluates the quality of trajectory segment based on the sum of rewards with equal weight. Based on these assumptions, preference predictor Pla > o"] defined equation 1. However, this formulation has several limitations in modeling real human preferences. First, in many cases, is hard to specify the tasks using the Markovian reward (Bacchus al., 1996; 1997; Early al., 2022). Furthermore, credit assignment within the trajectory can be required since human is sensitive to remarkable moments (Kahneman, 2000).
To address the above issues, introduce new preference predictor that assumes the probability of preferring segment depends exponentially on the weighted sum of non Mon-MMokovinn rewards:
exp (Ee w ({(s), a v 7 (1(s) a))))))))))))
Pla" > d'i
(3)
Σje(o,1 exp Ee w ((s)), a()) ( a()))
To capture the temporal dependencies, we consider a Markoviovi reward function f, which re- ceives the full preceding sub-trap sub-trajectory {(s,, a.)) as inputs. The importance weight is also introduced as function of the entire trajectory segment ((s, 1 which makes our preference predictor able to perform the credit assignment within the segment. We remark that our formula- tion is a generalized version of the conventional design; if the reward function only depends on the current state-action pair and the importance weight is always 1, our preference predictor would be equivalent to the standard model in equation 1.
4.2 ARCHITECTURE
In order to model the preference predictor as in equation 3, we propose a new transformer architec- ture, called Preference Transformer, that consists of the following learned components:
Causal transformer. We use the transformer network (Vaswani et al., 2017) as the backbone in- spired by its superiority in modeling sequential data (Brown et al., 2020; Ramesh al., 2021;
4

---

Published as a conference paper at ICLR 2023
Janner et al., 2021; Chen al., 2021). Specifically, use the GPT architecture (Radford al., 2018), i.e., the transformer architecture with causally masked self-attention. Given trajectory seg- ment {(s), a. ... (S } of length H. generate 2H imput embeddings (for state and action), which are learned by a linear layer followed by layer normalization (Ba et al., 2016). A shared positional embedding (ie., state and action the same timestep share the same positional embedding) is learned and added to each input embedding. The input embeddings are then fed into the causal transformer network, and produces output embeddings {x,]== such that t-th output depends on input embeddings up to t.
To model the preference predictor using the weighted sum of the non- Markovian rewards as defined in equation 3, we introduce preference attention layer. As shown in Figure 2, the preference attention layer receives the hidden embeddings from the causal transformer (xt) and generates rewards é and importance weights linear transformations
Preference attention layer.
Formally, t-th input x is mapped via key k, E Rd, query q: E RR,, and value Êg E R. where d is the embedding dimension. We remark that t-th value in self-attention, i.e., fr. is considered
modeling non- Markovian reward e ({(s; a.)) since hidden embedding X₁ only depends on previous inputs in trajectory segment. Specifically, each timestep approximating the reward f during the training.
use t state-action pairs {(s; a,)), for
Following the self-atte self-attention (Vaswani et al., 2017). the i-th output 2; is defined convex combi- nation of the values with attention weights from the i-th query and keys:
2 softmax(((q;))) (e) ke))?" fz.
Then, the weighted sum of rewards can be computed by the average of outputs { as follows:
H
Σ
2
so softmax (( k2)
wHft
where w/ = म H so tafffmax(ffffmmmmmmmmmmm (q:, ke)) Here, w, corresponds to modeling importance weight w([(s,, a.) )=) _ in equation because this preference attention is not causally masked and thus depends on the full sequence (i.e., bidirectional self-attention). In summary, we model the weighted sum of non-Markovian rewards by taking the average of outputs from transformer net- works and assume that the probability of preferring the segment is proportional to it. The complete architecture of Preference Transformer is shown Figure 2.
4.3 TRAINING AND INFERENCE
Training. We train Preference Transformer by minimizing the cross-entropy loss in equation 2 given a dataset of preferences D. By aligning a preference predictor modeled by Preference Transformer with human labels, we find that Preference Transformer induce suitable reward function and capture important events in the trajectory segments (see Figure 3 for supporting results).
Inference. For RL training, all state-action pairs labeled with the learned reward func- tion. Because we train the non-Markovian reward function, we provide H past transitions 11 a #+11" " St, a,) to Preference Transformer and use t-th value f; from the preference attention layer as a reward at timestep L.
EXPERIMENTS
We design our experiments to investigate the following:
Can Preference Transformer solve complex control tasks using real human preferences?
Can Preference Transformer induce a well-aligned reward and attend to critical events?
How well does Preference Transformer perform with synthetic preferences (i.e. scripted teacher settings)?
5

---

Published as a conference paper at ICLR 2023
Table Averaged normalized scores of IQL on AntMaze, Gym-Mujoco locomotion tasks, and suc- cess rate on Robosuite manipulation tasks with different reward functions. Using the same dataset of preferences from real human teachers, we train Preference Transformer (PT), MLPP-baed Marko- vian reward (MR; Christiano et al. 2017; Lee et al. 2021b), and LSTM-based Mon-Markovian reward (NMR; Early et al. 2022). The result shows the average and standard deviation averaged over runs.
IQL with task reward
IQL with preference learning
Dataset
MR
NMR
PT (ours)
maemaaze-medummpllaye
73.88 4.49
31.13
#: 16.96
62.88 5.99
70.13 1.76
anmaaremmddium-diveee---
68. 10.15
19.38
+9234
20.13
17.12
65.25 +3309
untmaze-large-piae paargge-play-22
48.75 4.35
24.25
14.13 3.60
42.38 9:99
anttaae-large--diveee--e
44.38 4.47
5,88 6.94
0.00 0.00
19.63 3.70
antmaze-v2 total
58.79
20.16
24.29
49.35
hopper-meciuu-rrepeee
83.06
11,56
30.27
57.88
40.63
84,54
4.07
hopper-meedium--exeett->
73.55 41.47
57.75
4: 23.70
38.63
35.58
68,96
4: 33.86
wallerr24-mcddium-rpplayyy
73.11 + 8.07
72.07
+ 1.96
77.00 + 3.03
71.27
10.30
walker2d-medium -xxpettt-t-mee--2222222 expert-v2
107.75 +: 2.02
108.32
+: 3.87
110.39
4: 0.93
110.13 1: 0.21
locomotion-v2 total
84.37
62.43
70.98
83.72
lift-ph
96.75 +: 1.83
84.75
6.23
91.50
5:42
91.75
1: 5.90
lift-mh
86.75
+ 2,82
91.00 4.00
90.75
5.75
86,75
:: 5.95
can-ph
74.50
d 6.82
68.00 9.13
62.00
à 10.90
69.67
4: 5.89
can-mh
56.25
47.50
30.50
8.73
50.50
+
robosuite total
78.56
72.81
68.69
74.66
5.1 SETUPS
Similar to Shin & Brown (2021), we evaluate Preference Transformer (PT) on several complex con- trol tasks in the offline setting using D4RL (Fu al., 2020) benchmarks and Robomimic (Man- dlekar et al., 2021) benchmarks.' Specifically, we consider three different domains (AntMaze, Gym-Mujoco locomotion (Todorov et al., 2012; Brockman al., 2016) from D4RL benchmarks and Robosuite robotic manipulation (Zhu et al., 2020) from Robomimic benchmarks) with differ- ent data collection schemes. For reward learning, we select queries (pairs of trajectory segments) uniformly at random from offline datasets and collect preferences from real human trainers (the au- t Then, using the collected datasets of human preferences, we learn reward function and train RL agents using Implicit Q-Learning (IQL; Kostrikov et al. 2022), recent offline RI algo- rithm which achieves strong performances on D4RL benchmarks. We consider Markovian policy and value functions by following the original implementation (e.g., architecture, hyperparameters) of IQL. We additionally provide the result offline RL experiments with the non-MDP models in Appendix G.
For evaluation, we measure noreet-zora scores respecting underlying task reward from the original benchmark (D4RL)* and success rate (Robomimic). As baselines, consider the standard preference modeling based on Markovian reward (MR) or non-Markovian reward (NMR). For the
exp F(s) [lss]a/:s/ a/a)))
MR-based model, we define the preference predictor as Pia > o²; v/val = Σ; expl E, f(s].n] abs))
using reward function modeled with MLP similar to Christiano al. (2017); Lee et al. (2021b). For the NMR- -based model, we use the LSTM-based architecture (Early et al., 2022) as a reward function
'To focus evaluating the performance of reward learning, we consider the offline setting. Note that similar setting assuming expert demonstrations available is also considered in prior work (Ibarz al., 2018).
We collected 100 (medium) 1000 (large) queries in AntMaze, 500 (medium- replay)) / 100 (medium- expert) queries in Muj-Mujoco Jocomotion, and 100 (PH) 500 (MH) queries in Robosuite robotic manip- ulations. A maximum of 10 minutes of human time required for feedback collection in all cases except Gyj-Mujoco caamontion-mdium--ee
and antmaze-large, which required hetween and hours of human
time.
SSinee preferences collected by experts who are familiar with robotics and RI. domains, measure the nerformance with respect task reward similar Christiano al. (2017).

---

Published as a conference paper at ICLR 2023
Mal
16 ON SOEEOO
Tinesapa
Timeskos
Agents movemnnt
(a) Successful trajectory
(b) Failure trajectory
Figure 3: Time series of learned reward function (red curve) and importance weight (blue curve) on (a) successful trajectory segment and (b) failure trajectory segment from untmaze-large-ple
For both cases, spikes in the importance weight correspond to crit- ical events: turning right to reach the goal (point 2), or flipping (point 4). The learned reward is also well-aligned with human intent: reward increases the agent gets close to the goal, while decreases when agent is flipped.
exp E. F [(s), [()) 11
and define the preference predictor as Pla > s/wNR] =
Note that both baselines use the same preference modeling based on the sum of rewards with equal weight, while method is based on the weighted sum of non-Markovian rewards modeled by transformers. We also report the performance of IQI. trained with task reward from benchmarks as a reference.
Σ, expl E, [f((s( . (-
For all experiments, details (e.g., task descriptions, feedback collection, and reward learning) are in Appendix B and C.
report the mean and standard deviation across 8 runs. More experimental
5.2 BENCHMARK TASKS WITH REAL HUMAN TEACHERS
Table shows the performances of IQL. with dif- ferent reward functions. Preference Transformer consistently outperforms all baselines in almost all tasks. Especially, only method almost matches the performance of IQL with the task reward, while baselines fail work in hard tasks. This implies that PT can induce a suitable reward function from real human preferences and teach meaningful behav- iors. In particular, there is big gap between PT and baselines complex tasks (i.e., AntMaze) since capturing the long-term context of the agent's 'ss behav- iors (e.g., direction of the agent) and critical events (e.g., goal location) is important in this task. These results show that our transformer-ben based preference model is very effective in reward learning.
PT vs MR
vs NMR
71.2%
16.3%
Profer
Equally Good
baseline
Figure 4: Averaged human evaluation results on 4 AntMaze tasks. Numbers denote the statistics of the evaluators' responses over 40 trials. PT received higher ratings compared to both MR and NMR.
Human evaluation. To check whether learned rewards are indeed aligned with human preferences, also conduct human evaluation. We generate query (i.c., two trajectories) from agents trained with two different rewards and human evaluator (authors) decides which trajectory is better. Fig- 4 shows the results comparing Preference Transformer and baselines averaged over 40 sets of
Original work (Farly al., 2022) only considered the return prediction but we utilize the proposed L.STM- lcarning.
architecture in preference-
'Trjecctories anonymized for fair comparisons. Also, the evaluators skip the query if both agents equally fail to solve the task for meaningful comparisons.
7

---

Published as a conference paper at ICLR 2023
Averaged Normalized Score
100
(imppte)) NMR (5 ptedi PT (Bolltted)
75
Mummnn)
NM4 (Human)
grumwn)
50
25
AntMaze
Hopper
Walker2d
Robosuite
Figure 5: Averaged normalized scores of IQL with various reward functions trained from human and synthetic preferences on AntMaze, Gym-Mujoco locomotion (Hopper and Walker2d), and Ro- bosuite robotic manipulation tasks. The result shows the mean and standard deviation averaged over runs. Our method (PT) achieves strong performances on both scripted and human teachers, while the performances of baselines (MR and NMR) are significantly reduced on human teachers.
queries. 6 We observe that human evaluators prefer agents from PT compared to agents from MR or NMR, showing PT more aligned with human preferences.
5.3 REWARD AND WEIGHT ANALYSIS
We evaluate whether Preference Transformer can induce a well-specified reward and capture the critical events from human preferences. Figure 3 shows the learned reward function (red curve) and importance weight (blue curve) on successful and failure trajectory segments from antmaze aatmazeelarge-play-ee
First, we find that the reward function well-aligned with hu- man intent. In the successful trajectory (Figure 3a), reward value increases as the agent gets close the goal, while the failure trajectory shows low rewards since the agent struggles escape the corner, then flips as shown in Figure 3b. This shows that Preference Transformer can capture the the reward function. We also find that importance weight shows
context of the agent's 's behaviors a different trend compared to reward function. Interestingly. for both successful and failure trajec- tories, spikes importance weights correspond critical events such as turning right reach the goal or flipping. More video examples are available in the supplementary material.
5.4 BENCHMARK TASKS WITH SCRIPTED TEACHERS
Similar prior work (Christiano al., 2017; Lee al., 2021b;c), also evaluate Prefer- ence Transformer using synthetic preferences from scripted teachers. We consider a determinis- tic teacher, which generates preferences based on task reward from the benchmark follows: y = i, where = arg max; Σ/ r(s), a²). We remark that this scripted teacher a special case of the preference model introduced in equation 1.7
Figure 5 shows the performances of IQL with different reward functions from both human and scripted teachers (see Appendix D for full results). Preference Transformer achieves strong per- formances on scripted teachers even though synthetic preferences are based on Markovian rewards which contribute to the decision equally. We expect this is because Preference Transformer can duce hetter-shaped reward by utilizing historical information. Also, non-Markovian formulation (in equation 3) can be interpreted as a generalized version of Markovian formulation (in equation 1) since it can learn Markovian rewards by ignoring the history in the inputs. We also remark that baselines achieve better performances with scripted teachers compared to real human teachers on some easy tasks (e.g., locomotion tasks). This implies that scripted teacher does not model real human behavior exactly, and evaluation on scripted teachers can generate misleading information. To investigate this in detail, measure the agreement rates between human and scripted teach- ers. Figure 6b shows that disagreement rates are quite high since the scripted teacher can not catch
6 'ach set consists of 5 rollouts from 8 models trained with different random seeds. II we introduce rationality constant (or temperature) 3 the exponential term in equation and set as oc, the preference model hecomes deterministic.
8

---

Published as a conference paper at ICLR 2023
12.2
Ardnazo-ted
2033
Arorece--d
71.5
28.7
83.0
WiRer2l-in
46.
56.4
trajectory
trajectory
Litgh
$1.0
35.0
Utbeh
01
00
Cangh
R6.3
20.3
Canveh
31.2
Human
Dicagran
Aginn
(a) Examples of trajectories
(b) Agreement rates (%)
Figure 6: Difference between the human and scripted teacher. (a) Examples of trajectories shown to the human and scripted teacher on AntMaze task. The human teacher provides the correct label by catching the context of behavior (i.e. direction) while the scripted teacher does not. (b) Agreement between human teachers and scripted teachers. We find that disagreement rates are quite high across all tasks, implying that evaluation on scripted teacher can generate misleading information.
the context of the agent's behavior correctly (see Figure 6a). Note that the disagreement between human and scripted teachers also has been observed in simple grid world domains (Knox et al., 2022). These results imply that existing peffrence-bbssed RL benchmarks (Lee al., 2021c) based on synthetic preferences may not be enough, calling for a new benchmark specially designed for preference-based RI.
5.5 LEARNING COMPLEX NOVEL BEHAVIORS
Figure 7: Six frames from a double backflip of Hopper. The agent is trained to perform a sequence of backflips using 300 queries of human feedback.
We present the effectiveness of Preference Transformer for enabling agents to learn complex and novel behaviors where suitable reward function is difficult to design. Specifically, demon- strate Hopper in Gym-Mujoco locomotion performing multiple backflips at each jump. This task challenging compared
single backflip considered in previous preference-based RI. approaches (Christiano et al., 2017; Lee et al., 2021b), as the reward function must capture non- Markovian contexts including the number of rotations. We observe that the Hopper agent learns to perform double backflip as shown in Figure 7, while the agent with Markovian reward function fails learn it. The implementation details of Hopper backflip are provided in Appendix B, and videos of all behaviors (including triple backflip) are available on the project website.
DISCUSSION
In this paper, we present new framework for modeling human preferences based on the weighted sum of non-Markovian rewards, and design the proposed framework using a transformer- chitecture. We propose a novel preference attention layer for learning the weighted sum of the non-Markovian reward function, which enables inferring the importance weight of each reward via the self-attention mechanism. Our experiments demonstrate that Preference Transformer signif- icantly outperforms the current preference modeling methods on complex navigation, locomotion, and robotic manipulation tasks from offline RL benchmarks. In addition, we observe that the learned preference attention layer indeed capture the events critical to the human decision. We believe that Preference Transformer essential to scale preference- baseed RL (and other human-in -iaan----tee--ooot the-loop learning) to various applications.
ar-

---

Published as a conference paper at ICLR 2023
FUTURE DIRECTIONS
There are several future directions in Preference Transformer. One is to utilize the importance weights in reinforcement learning preference-based reward learning. For example, importance weights can be utilized for sampling more informative queries, which can improve the feedback- efficiency of preference-based reward learning (Sadigh et al., 2017; Lee et al., 2021c). It is also an interesting direction to use the importance weights for stabilizing Q-learning via weighted up- dates (Kumar et al., 2020; Lee al., 2021a). Combination with other preference models is another important direction for future research. For example, Knox et al. (2022) proposed a new preference model based on each segment's regret simple grid world environments. Even though their pro- posed method is based on several assumptions (e.g., generating successor features (Dayan, 1993; Barreto et al., 2017)), combination with the regret-based model would be interesting.
ETHICS STATEMENT
Unlike other domains (e.g., language), control tasks require more high-quality human feedback from domain experts. Even though the quality of the dataset can be improved by training labelers, such training requires substantial effort. addition, feedback from crowd-soureing platforms, such as Amazon Mechanical Turk, can be noisy. We addressed these concerns by collecting human feedback from domain experts (authors) who are very familiar with roboties, RL, and target tasks. We expect that our datasets are labeled with high-quality and clean labels, and our collection strategy is closer to practice. However, the same time, we think that evaluations using public crowd-sourcing platforms would be interesting and leave future work.
REPRODUCIBILITY STATEMENT
We describe the implementation details of Preference Transformer in Appendix B, and our source code available on the project website in the abstract. We will also publicly release the collected offline dataset with real human preferences for benchmarks.
ACKNOWLEDGMENT AND DISCLOSURE OF FUNDING
We would like to thank Sihyun Yu, Subin Kim, Younggyo Seo, and anonymous reviewers for pro- viding helpful feedback and suggestions for improving our paper. This research is supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) NNo.2022-0-0099 Self-directed AI Agents with Problem-solving Capability: No.2019-0-000 9-0-00075, Artificial Intelligence Graduate School Program (KAIST)). This ma- terial is based upon work supported by the Google Cloud Research Credits program with the award
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2011.
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference learning-basee rein- forcement learning. In Joint European Conference on Machine Learning and Knowledge Discov- ery in Databases, 2012.
Jose A Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brand- stetter, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. In Advances in Neural Information Processing Systems, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 607.004500 2016.
10

---

Published as a conference paper at ICLR 2023
Fahiem Bacchus, Craig Boutilier, and Adam Grove. Rewarding behaviors, In National Conference on Artificial Intelligence, 1996.
Fahiem Bacchus, Craig Boutilier, and Adam Grove. Structured solution methods for non-markovian decision processes. In AAAI Conference on Artificial Intelligence, 1997.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 2017.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota 2 with large scale deep reinforcement learning. arXiv preprint 1912::1922.066 2.06800, 2019.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01. 000.001000.. 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Advances Neural Information Processing Systems, 2021.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
C. Daniel, M. Viering, J. Metz, O. Kroemer, and J. Peters. Active reward learning. In Robotics: Science and Systems, 2014.
Sudeep Dasari and Abhinav Gupta. Transformers for one-shot visual imitation. In Conference on Robot Learning, 2020.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural computation, 5(4):613-624, 1993.
Yan Duan, Marcin Andrychowicz, Bradly Stadie, OpenAl Jonathan Ho, Jonas Schneider, Ilya Sutskever, Pieter Abbeel, and Wojciech Zaremba. One-shot imitation learning. Advances in Neural Information Processing Systems, 2017.
Joseph Early, Tom Bewley, Christine Evers, and Sarvapali Ramchurn. Non-markovian reward mod- elling from trajectory labels via interpretable multiple instance learning. In Advances in Neural Information Processing Systems, 2022.
Aleksandra Faust, Anthony Francis, and Dar Mehta. Evolving rewards automate reinforcement 2019.
learning. arXiv preprint 1rXiv:1905.007
Roy Frostig, Matthew James Johnson, and Chris Leary. Compiling machine learning programs via high-level tracing. Systems for Machine Learning. 4(9), 2018.
Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint 19220:1902.07742, 2019.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Tanmay Gangwani. Yuan Zhou, and Jian Peng. Learning guidance rewards with trajectory-space smoothing. In Advances in Neural Information Processing Systems, 2020.

---

Published as a conference paper at ICLR 2023
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8): 1735-1780, 1997.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, 2018.
Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference Robot Learning, 2022.
Michael Janner, Qiyang Li, and Sergey Levine. Reinforcement learning as one big sequence mod- eling problem. In Advances Neural Information Processing Systems, 2021.
Daniel Kahneman. Evaluation by moments: Past and future. Choices, values, and frames, pp- 693-708, 2000.
W Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The tamer framework. In International Conference on Knowledge Capture, 2009.
W Bradley Knox, Stephane Hatgis- Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessan- dro Allievi. Models of human preference for learning reward functions. arXiv preprint 22ti:222066022311, 2022.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In International Conference Learning Representations, 2022.
Aviral Kumar, Abhishek Gupta, and Sergey Levine. Discor: Corrective feedhack in reinforcement learning via distribution correction. In Advances in Neural Information Processing Systems, 2020.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Picter Abbeel. Sunrise: A simple unified frame- work for ensemble learning in deep reinforcement learning. In International Conference Ma- chine Learning, 2021a.
Kimin Lee, Laura Smith, and Picter Abbeel. Pebble: tredback-efficiene interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, 2021b.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference-based reinforcement learning. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track (round 1), 2021c
Xinran Liang, Katherine Shu, Kimin Lee, and Pieter Abbeel. Reward uncertainty for exploration in preference-based reinforcement learning. In International Conference on Learning Representa- tions, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Zhao Mandi, Fangchen Liu, Kimin Lee, and Pieter Abbeel. Towards more generalizable one-shot visual imitation learning. International Conference on Robotics and Automation, 2022.
Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei- Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín-Martin, What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint 108.22800,.03298, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Suraj Nair, Eric Mitchell, Kevin Chen, Brian Ichter, Silvio Savarese, and Chelsea Finn. Learning language-condition robot behavior from offline data and crowd-sourced annotation. In Confer- on Robot Learning, 2022.
12

---

Published as a conference paper at ICLR 2023
Reiichiro Nakano, Jacob Hilton, Suchir Balaji. Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted sueestion-answere with human feedback. arXiv preprint 2xXIv:2112.03332,20000 2021.
Andrew Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. International Conference Machine Learning, 2000.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, 2020.
Jongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. Surf: preference-baee
Semi-supervised reward learning with data augmentation for reinforcement learning. In International Conference Learning Representations, 2022.
Xue Bin Peng. Erwin Coumans, Tingnan Zhang, Tsang- Wei Lee, Jie Tan, and Sergey Levine. Learn ing agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems, 2020.
Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason Carey, and Richard S Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In International Conference on Rehabilitation Robotics, 2011.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training, 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, 2021.
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.0677
2022.
Zhizhou Ren, Ruihan Guo, Yuan Zhou, and Jian Peng. Learning long-term reward redistribution via randomized return decomposition. In International Conference on Learning Representations, 2022.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference-based learn- ing of reward functions. In Roboties: Science and Systems, 2017.
C. Schenck and D. Fox. Visual closed-loop control for pouring liquids. In International Conference on Robotics and Automation, 2017.
Daniel Shin and Daniel S Brown. Offline preference-basee basedd apprenticeship learning. arXiv preprint arXiv:2107.09251, 2021.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676) :554, 2017.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. arXiv preprint TTT::2000.01325 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.
Yuval Tassa, Tom Erez, and Emanuel Todorov. Synthesis and stabilization of complex behaviors through online trajectory optimization. In International Conference on Intelligent Robots and Systems, 2012.
13

---

Published as a conference paper at ICLR 2023
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems, 2017.
Mudit Verma and Katherine Metcalf. Symbol guided hindsight priors for reward learning from human preferences. arXiv preprint arXiv:2210.01551, 2022.
Oriol Vinyals, Igor Babuschkin, Wojeiech M Czameeki, Michael Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782):350 2019.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances in Neural Information Processing Systems, 2012.
Jeff Wu, Long Ouyang, Daniel Ziegler, Nissan Stiennon, Ryan Lowe, Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2 10862, 2021.
A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In International Conference Intelligent Robots and Systems, 2017.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep re- inforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019.
Ruohan Zhang, Calen Walshe, Zhuode Liu, Lin Guan, Karl Muller, Jake Whritner, Luxin Zhang, Mary Hayhoe, and Dana Ballard. Atari-head: Atari human eye-tracking and demonstration dataset. In AAAI Conference on Artificial Intelligence, 2020.
Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: modular simu- lation framework and benchmark for robot learning, arXiv preprint 1222: 2020.
14

---

the same Published as a conference paper at ICLR 2023
A TASKS AND DATASETS
In this section, we describe the details of control tasks from D4RL benchmarks (Fu et al., 2020).
AntMaze. AntMaze is navigation task requiring Mujoco Ant robot to reach goal location. The datasets are generated by pre-trained policy designed to reach goal on different maze layouts. We consider two maze layouts: medium and large. Datasets are generated by two strategies: diverse and play. The diverse datasets are generated from the pre-trained policy with randomized start and goal locations. The play datasets are generated from the pre-trained policy with specific hand-picked goal locations. Task reward is constructed in a sparse setting that gives rewards only when the distance to the goal location is less than a fixed threshold, or all of them give zero.
Gym-Mujoco locomotion. The goal of Gym-Mujoco locomotion tasks is to control the simu- lated robots (Walker2d, Hopper) such that they can move forward while minimizing the energy cost (action norm) for safe behaviors. We consider two data generation strategies: medium- expert and medium-replay. The medium-exper datasets are generated by mixing equal amounts of expert demonstrations, suboptimal (partially-triined) demonstrations, and mediun-replay datasets corre- spond to the replay buffer collected by a partially-trained policy. The task reward is defined as the forward velocity of the torso, control penalty, and survival bonus.
Robosuite robotic manipulation. Robosuite robotic manipulation (Zhu et al., 2020) includes dif- ferent types of tasks with various 7-DoF simulated Hand robots. We use simulated environments with Panda by Franka Emika for the robot in our experiments. We choose the task of lifting a cube object (lift) and placing a coke can from table the target bin (can). Datasets are collected by either one proficient telcoperators (ph) or 6 teleoperators with varying proficiency (m/). Task reward is defined as a sparse reward, and we defer this detail to the original paper.
B EXPERIMENTAL DETAILS
Training details. Our implementation of Preference Transformer is available at:
https: //gtthb.
Our model is implemented based on publicly available re-implementation GPT JAX (Frostig et al., 2018).. We use the hyperparameters in Table 2 for all experiments. We use segments of length 100 in all experiments. To improve the stability in training IQL with learned reward function, we normalize the learned reward as follows:
predi reward max timestep x learned reward (max returns)
(max returns) - (min returns)
where max timestep is the maximum episode length, and max timestep and min timestep de- note returns of best and worst trajectories in the dataset, respectively. Training time varies depending on the environment, but it takes less than 10 minutes for reward learning in AntMaze environment with 1000 human feedback and takes about an hour to train IQL with learned PT. For training and evaluating our model, we use a single NVIDIA GeForce RTX 2080 Ti GPU and 8 CPU cores (Intel Xeon CPU E5-2630 v4 @ 2.20GHz). We train both reward function and IQL over 8 random seeds.
Implementation details of PT. In all experiments, we use causal transformers with one layer and four self-attention heads followed by a bidirectional self-attention layer with single self-attention head. PT is trained using Adam W optimizer (Loshchilov & Hutter, 2019) with a learning rate of 1 x 10-4 including linear warmup steps of 5% of total gradient steps, cosine learning rate decay, weight decay of 1 x 10-4, and batch size of 256. For RL training, we use publicly released implementations of IQL and follow the original hyper-paramend hyperparameters used by the original IQL.
settings. For all experiments,
Evaluation details. We follow the original settings of IQL in evaluation. We evaluate IQL over 100 rollouts in AntMaze, 10 rollouts in Gym-Mujoco locomotion, and 50 rollouts in Robosuite robotic manipulation. For evaluation metries, we measure expert-normaliza scores introduced in D4RL
$nttps: pttpps:/Ae http://gww..b.cc//taaaaaaaggggg/ Phttps: / wtt...com/kkottrtiiioci
15

---

Published as a conference paper at ICLR 2023
normalized score 100 locomotion tasks and success rate for Robosuite robotic manipulation tasks.
benchmark:
expart score randon for AntMaze and Gym-Mujoco
rraddor
Hyperparameters. Hyperparameters for PT shown in Table 2. We remark that PT with more attention layers and more gradient updates can boost the model's performance, but we use the current version with small layers for faster training and evaluation.
Table 2: Hyperparameters of Preference Transformer.
Hyperparameter
Value
Number of layers
1
Number of attention heads
4
Emhedding dimension
256
(Casual transformer, Preference attention layer)
Batch size
256
Dropout rate (embedding, attention, residual connection)
0.1
Learning rate
0.0001
Optimizer
AdamW (Loshchilov & Hutter, 2019)
Optimizer momentum
81 0.9, .3 0.99
Weight decay
0.0001
Warmup steps
500
Total gradient steps
10K
Baselines. Implementation details of our baselines follows:
Markovian reward model (MR). We use two-layer MLPs with 256 hidden dimensions each. We use ReLUS for the activation function between layers, and we do not use the activation function for the output. Each model is trained by optimizing the cross-entropy loss defined in equation with the learning rate of 0.0003.
Non-Markovian reward model (NMR). For the non-Markovian reward model, re- implement CSC Instance Space LSTM (Early et al., 2022) following the architecture specified in the original paper using JAX. We double the hidden dimensions to match the NMR and PT.
Implementation details of Hopper backflip. For enabling Hopper backflip, we train the agent in online setup following Lee et al. (2021b). We pre-train the policy и using an intrinsic reward for the first 10,000 timesteps, to explore and collect diverse experiences. Then, we train PT using collected behaviors and relabel past experiences in the replay buffer using the learned reward model. This reward learning and relaheling stage is repeated every 10,000 timesteps, and we give 100 queries of human feedback at each stage. We observe that 3 reward learning stages (i.e., 300 queries) are enough to perform double backflips. We use 3 layers and attention heads for PT, and the model trained for 100 epochs each stage. The other details are the same as in Table 2.
C HUMAN PREFERENCES
Preference collection. We collect feedback from actual human subjects (the authors) familiar with robotic tasks. In detail, the human teacher who is given instruction each task watches video rendering each segment and chooses which of the two is more helpful in achieving the objective of the agent. Each trajectory segment is 3 seconds long (100 timesteps). If the human teacher cannot decide about the preference over segments, it is allowed to select a neutral option that gives the same preference to each of the segments.
Instruction given to human teacher.
AntMaze: The first priority is for the ant robot to reach the goal location as soon as possible without wandering or falling. If the ant robot is either left falling, hovering, moving in the
16

---

Published as a conference paper at ICLR 2023
opposite direction to the goal location, lower your priority to the segment even if the distance to the goal direction is closer than that of the other segment. If the two robots are almost tied this metric, choose the segment by the distance that the robot has moved.
Hopper: The hopper robot aims to move to the right as far as possible while minimizing energy costs. If the hopper robot lands unsteadily, lower your priority even if the distance traveled moved during segment longer than the other. If the two robots are almost tied on this metric, choose the segment by the distance that the robot has moved.
Walker2d: The goal of the walker robot is to move to the right as far as possible while min- imizing energy costs. If the walker is about fall or walks abnormally (e.g., walking using only one leg, slipping, etc.), lower your priority to the segment even if the distance traveled moved during a segment longer than the other. If the two robots are almost tied on this metric, choose the segment by the distance the robot has moved.
Robosuite: Panda robot arm first grasps the object and carries out actions for specific purpose (lift cube move coke can to the target bin). Prioritize catching the object between the two. If the two robots are almost tied on this metric, choose by the extent to which the target object has moved.
D FULL EXPERIMENTAL RESULTS WITH SCRIPTED TEACHER
Table 3: Averaged normalized scores of IQL on AntMaze, Gym-Mujoco locomotion tasks, and success rate on Robosuite manipulation tasks with different reward functions. We train Preference Transformer (PT) and standard preference modeling with Markovian reward (MR) modeled by MLP or non-Markovian reward (NMR) modeled by LSTM using the same dataset of preferences from scripted teachers and real human teachers. The result shows the average and standard deviation averaged over 8 runs.
IQL with preference learning
IQL. task
Dataset
Scripted Teacher
Human Teacher
MR
NMR
PT (ours)
MR
NMR
PT (ours)
antmane-medium-pllge
73.88 4.00
64.75 5:2
66.13
66.13 1.36
31,13
16.66
62.88 4.99
70.13 #37
antmsao-medium-ddieess
68.13 1015
4.25
627
67.00 5:000
68.13 #:488
19.38 9:244
20.13 17.2
65,25 #33
animaze large play-v2
48.75 14436
21.00
14:13
10.75 199
23.13 15:00
24.25
14m
14.13
1.000
42,38 199
antmaze-large- baruuu--arge-diveee---e
44.38 4.47
15.75 4:5
12.00 333
44.50 2:500
5.88 6.9
0.00 2:00
19.63 3.0
antmazz-vv v2 average
$8.79
26.31
38.97
50.00
20.16
24.29
49.35
hopper medium replay-v2 x2
83.06 15.00
62.77 1:366
72.33 amm
94.19 1:e
11.56 2:122
57.88
4185
84.54 ::
hoppx--mcdiun--cpeett2222
73.55 44147
80.00
3306
85.97
#3.51
39.14 20.33
57.75 23.70
38.63
3538
68.96 33.6
wa walker2d-mmddum--opllae
73.11 =800
65.69 8.17
73.63 #:35
77.08 9:9.4
72.07 196
77.00
1/0
71.27 10.30
wal wwller22-mmdiiu--oppeeyy
107.75 mm
109.95 6.5
110.41 4-0m
109.99 0/53
108. 32 3.0
110.39
110.13 6.0
locomotion-v2 average
84,37
79.60
85,56
80.10
62,43
70.98
83.72
lift-ph
96.75 1:8
95.75 277
80.00
10.15
92.50 +:424
84.75 6:633
91.50 +:442
91,75 =:5
lift mb
86.75 1.2
92.25 14435
91.50 1.542
93.00 1:55
91.00 1:100
90.75 1:775
86.75 1.595
can-ph
74.50 6.6
62.25
12:02
67.75
1:600
69.75
1:04
68.00
2:133
62.00
10.0
69.75 5:89
cun-mih
56.25 2.8
59.00 1.00
54.25 135
53.25 1:508
47.50 1:31
30.50 k.73
50.50 =
robosuite average
78.56
77.31
73.37
77.63
72.81
68.69
74.68
17

---

Published as a conference paper at ICLR 2023
E ADDITIONAL EXAMPLES OF QUERIES
In Figure 8, we provide additional examples of queries to highlight the difference between human and scripted teachers. the case of the example Figure 8a, the scripted teacher prefers trajectory since ant is closer to the goal. However, the human teacher prefers trajectory 0 since ant is flipped in trajectory
the case of the example in Figure 8b, the scripted teacher shows very myopic decision because hand hand-designed reward does not capture the context of behavior (e.g., falling down). These examples show the lack of ability to consider multiple objectives, while humans can provide more reasonable feedback by balancing the multiple objectives.
VS
trajectory 0
ø1 trajectory
dg Inajectoyy 0
01 trajectory
> 00
00 >
σ1
>
00
Human
Scripted
Human
Scripted
(a) AntMaze
(b) Walker2d
and (b) ww-relly-22 Since hand-designed task reward does not contain all task- relevant information, scripted teacher ignores important events such as flipping and falling down.
Figure 8: Illustrations of trajectories from (a) antmaze-large-play-t
DESCRIPTION OF THE VIDEO EXAMPLES
We provide video examples with visualization of the learned importance weight in the supplemen- tary material. For better visibility, we represent the weight as color map the rim of the video, ie., we highlight frames with higher weights using more bright colors. We observe that the learned importance weights are well-aligned with human intent, in Figure 3.
18

---

Published as a conference paper at ICLR 2023
G EXPERIMENTAL RESULTS USING NON-MARKONE MODELS
We investigate whether our non-Markovian reward function shows better performance with non- Markovian policy and value functions. For each timestep t, we augment state sr by concatenating historical information". 10 and use it as imputs to train policy and value functions. Table 4 shows the results comparing Preference Transformer and NMR. We observe that PT outperforms NMR, which again shows the superiority of our method. However, compared to the results with Markovian policy and value functions Table 1, the overall performances of all methods are degraded. We expect that this is because the original offline RI. algorithm (IQL) assumed Markovian setup and tuned all hyper-panee syper-parameters under this assumption.
Table 4: Averaged normalized scores of non-MDP IQL with different reward functions on AntMaze and Gym-Mujoco locomotion tasks. Using the same dataset of preferences from real human teach- ers, we train Preference Transformer (PT) and LSTM-based non- ManMMaaoovinn reward (NMR; Early et al. 2022). The result shows the average and standard deviation averaged over 8 runs.
IQI. with preference learning
Dataset
NMR
PT (ours)
cuttmaze-mediun medium--paa--22
54,50 4.99
64.38 3:36
untmaze--ediun--iivee
10,50 13.60
53.00 2155
untmaze-lange-play-v2
14.83 5.78
25.63 4.788
antmaaz-large-divees---22
0.00 0.00
8.13 a 3.27
antmaze-v2 total
19.96
37.79
hopper-meddium--rplly-222
15,80 2621.11
55.77
#1.00
happer-mcddum--expprtt2
55.80 4:34.3
79.84
+ 29.32
walker2d
41.91 32.58
67.06
13.39
waller22lmmccium--xeper----
94.20 +:29.11
104.97
locomotion-y v2 total
51.92
76.91
"Specificallly provide predicted reward of the previous timestep Âr-1 additional input.
19


Reviews:
[{"review_id": "m9Gazucaslw", "reviewer": "Reviewer_EobQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper describes an approach for learning a reward function in preference-based reinforcement learning based on transformers.  This provides the advantages that the model allows credit assignment within the behavior trajectory to correctly weight significant state/actions, and that less feedback samples that has typically been required to learn the reward.", "summary_of_the_review": "Reward learning for preference-based RL is receiving increasing interest, so this is timely.  Transformers are a natural way to accommodate the problem of credit assignment over a trajectory.", "strength_and_weaknesses": "+The paper is well written\n\n+The transformer approach naturally lends itself to credit assignment, which is important for human preferences as human are significantly influenced by surprising events.\n\n-Given an advantage of this approach is feedback-efficient, it would have been nice to have compared against PEBBLE given that is designed for efficiency.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Sh0Fw9IMO_i", "reviewer": "Reviewer_gNsz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Given: just preferences over agent trajectories (and not rewards),\nand the \"underlying\" reward being non-Markovian and weighted.\n\nApproach: the authors propose the Preference Transformer model which learns the \"latent reward\" based on the whole past being fed into a Transformer to output current reward and weight. Then from this reward they can classically train an agent.\n\nThe contribution includes this architecture as well as experiments\n", "summary_of_the_review": "Overall, it is an interesting idea to use Transformers for non-Markovian reward and weighting, as part of preference RL. The writing overall is OK-good, though some clarifications are needed. \n\nThe main weaknesses for me are that the contribution is interesting but rather limited compared to (Early 2022) (replace LSTM by Transformer and add weighting). Additionally, I'm not sure how often actually non-Markovian rewards are needed, since they can be mitigated by good state representations. And, in the experiment, if the reward is Markovian, I didn't understand why nonetheless the non-Markovian method performs better.\n\nWhen ignoring significance/size of contribution, I lean accept; when not ignoring it, I lean reject. In favor of the doubt, overall I lean accept.", "strength_and_weaknesses": "**Strengths.**\n\nExcept for details, overall the paper is well written.\n\nThe preference RL setting is interesting,\nand,\nif in deed the rewards are non-Markovian,\nthe idea of Transformers is well motivated.\n\nSo there is a conceptual as well as empirical contribution.\n\n\n**Weaknesses and improvement points:**\n\nI think the precise contribution over previous work needs to be clarified:\n* From a first look at the paper, I had these questions: does a preference transformer already exist, but just for Markovian, unweighted rewards, and the contribution is just to extend it to the non-Markov, weighted setting? Or is the contribution, to introduce the preference transformer thing altogether, and directly in the general from (non-M. ...)?\n* When looking into it in more detail, I understand it as follows: it seems (Early 2022) is closest, since they also address preference RL with non-Markov rewards, and with a similar relation between rewards and preferences, but the difference is that (Early 2022) (1) use LSTM instead of Transformers and (2) don't allow for weighing. This, in particular, should be made more clear.\n\nRegarding experiments, there need to be some clarifications, and some results are counterintuitive:\n* For the tasks that are studied, are the true reward functions actually Markovian or not? I don't find that info. Or is there no true reward for some tasks?\n* Sec 5.4: if the reward is actually Markovian, then the much simpler MR baseline model (which assumes a Markovian reward via an MLP) actually has *no approximation error in that sense (i.e., up to functional approximation, the dependencies between the variables can in principle perfectly be captured by this model class)*, but is much simpler in terms of capacity (so, should be more data efficient). I'm really surprised that nonetheless PT outperforms it. Can you explain why PT outperforms it?\n\n**Minor points:**\n\n* Note that non-Markovian rewards can be turned into markovian my changing the state.\n* How does it relate to V/Q value function? It's not necessary, but 1-2 sentences could be interesting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P1Ri3kLoGm", "reviewer": "Reviewer_WbtG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a transformer-based neural architecture, Preference Transformer, to model human preferences in preference-based reinforcement learning. Specifically, this work adopts the assumption of non-Markovian rewards and utilizes transformer-based architecture in modeling sequential data to capture temporal dependencies in human decisions and infer critical events in a trajectory.", "summary_of_the_review": "This paper introduces a novel transformer-based architecture to model human preferences under the non-Markovian assumption in preference-based reinforcement learning. The overall framework seems to be effective and performs well according to the experiment results. Though some of the claims have minor issues, the overall paper is well-written and easy to follow.", "strength_and_weaknesses": "Strengths:\n1. This paper is well-written and easy to understand.\n2. The proposed new preference predictor that depends exponentially on the weighted sum of non-Markovian rewards is reasonable and effective.\n3. Conducted Experiments are well-designed. Preference Transformer are evaluated both on real human preferences and synthetic preferences. The authors demonstrate the relationship between them and show that existing benchmarks based on synthetic preferences may not be enough.\nWeaknesses:\n1. It would be interesting to see justifications for the architecture selection. Did you compare the GPT with other transformer architectures in experiments?\n2. In Section 5.2, the authors demonstrate that capturing long-term information is the reason why PT is more competitive than other baseline models, but LSTM-based NMR could also capture temporal dependencies. There should be detailed explanations to clarify the demonstration.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ItjfYT3dtK5", "reviewer": "Reviewer_v17x", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce a novel method for Preference-based Reinforcement Learning, which targets the problem of non-Markovian rewards. Namely, that human trajectory preferences cannot be assumed to be based on a sum of Markovian rewards, as expected in common RL and PbRL settings. The method learns non-Markovian rewards, based on the assumptions, that trajectory segment returns can be defined as weighted sum over history dependent state, action rewards. The learning algorithm uses a transformer architecture to these ends. Besides the novel PbRL algorithm, the authors contribute an empirical evaluation, demonstrating the advantages on 5 different domains (with 2-4 variants each).", "summary_of_the_review": "The paper is a nice read and the method is novel and interesting. However, the stated issues with a main property of the algorithm prevent an accept. Although, to be fair, the authors claims are not directly related to the property and only refer to \"real human preferences\" and are therefore mostly substantiated. Some issues concerning the evaluation metrics and clarity remain, but can likely be resolved without substantial time effort. ", "strength_and_weaknesses": "The consideration of non-Markovian trajectory preferences is an important issue in PbRL, because the Markov property is usually not assumed to hold in this setting. Therefore, non-Markovian PbRL is important to move from synthethic evaluations to real world applications. The paper is mostly well written, but some assumptions are hidden and some clarifications are required. (See next Section) The evaluation covers a sufficiently divers set of domains and real, human evaluations are usually to be preferred over pure synthethic trials. However, the evaluation is lacking in several parts:\n\n- Evaluation is only done in terms of reward. However, user preference may not be aligned with the given reward function. In fact, if it is, the domain does not require a non-Markovian approach. Therefore, evaluation should be performed in terms of preference agreement of the resulting policy. Furthermore, the number of queries is usually also deemed an important metric, because human evaluations are costly.\n- The scripted evaluation uses a Markovian reward, but the NMR and PT approaches are still able to outperform the MR variant in several domains. This needs to be better explain and evaluated. In fact, a non Markovian reward should be used for the evaluation.\n- The RL algorithm used to maximize the reward assumes a common MDP structure. This means, it will likely not correctly consider the non-Markovian property or the weighting factors. Considering it works, it may be that the learned reward \"collapses\" to an Markovian variant (also see next Section concerning this issue). To resolve this issue, it should be validated that the learned reward cannot be approximated with a Markovian definition and/or a PO-MDP or N-state MDP RL algorithm should be used.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster