Benchmarks and Algorithms for Offline Preference-Based Reward Learning
Authors: Daniel Shin
Anca Dragan
Daniel S. Brown
Venue: Accepted by TMLR ()

Abstract: Learning a reward function from human preferences is challenging as it typically requires having a high-fidelity simulator or using expensive and potentially unsafe actual physical rollouts in the environment. However, in many tasks the agent might have access to offline data from related tasks in the same target environment. While offline data is increasingly being used to aid policy optimization via offline RL, our observation is that it can be a surprisingly rich source of information for preference learning as well. We propose an approach that uses an offline dataset to craft preference queries via pool-based active learning, learns a distribution over reward functions, and optimizes a corresponding policy via offline RL. Crucially, our proposed approach does not require actual physical rollouts or an accurate simulator for either the reward learning or policy optimization steps. To test our approach, we first evaluate existing offline RL benchmarks for their suitability for offline reward learning. Surprisingly, for many offline RL domains, we find that simply using a trivial reward function results good policy performance, making these domains ill-suited for evaluating learned rewards. To address this, we identify a subset of existing offline RL benchmarks that are well suited for offline reward learning and also propose new offline apprenticeship learning benchmarks which allow for more open-ended behaviors. When evaluated on this curated set of domains, our empirical results suggest that combining offline RL with learned human preferences can enable an agent to learn to perform novel tasks that were not explicitly shown in the offline data. 

Published in Transactions on Machine Learning Research (01/2023)
Benchmarks and Algorithms for Offline Preference-Based Reward Learning
Daniel Shin
Computer Science Department
Stanford University
Anca D. Dragan
anca&Wberkeley.commm edu
EECS Department University of Colifornia, Berkeley
Daniel S. Brown
dsbrown@cs. utah. edu
School of Computing
University of Utah
Reviewed on OpenReview: /ttps://openneee www.reveeep.....e..om net/ forum? TGuXXldKen
Abstract
Learning a reward function from human preferences is challenging as it typically requires having a high-fidelity simulator or using expensive and potentially unsafe actual physical rollouts the environment. However, in many tasks the agent might have access offline data from related lasks in the same target environment. While offline data increasingly being used to aid policy optimization via offline RL, our observation is that it can be surprisingly rich source of information for preference lcarning well. We propose approach that uses oflline dalaset to craft preference queries via pool-based active learning, learns distribution over reward functions, and optimizes a corresponding policy via offline RL. Crucially, our proposed approach does not require actual physical rollouts or an accurate simulator for either the reward learning or policy optimization steps. To test our approach, we first evaluate existing offline RT. benchmarks for their suitability for offline reward learning. Surprisingly, for many offline RL domains, we find that simply using a trivial reward function results in good policy performance, making these domains ill-suited for evaluating learned rewards (even those learned by non preference-bart based methods). To address this, we identify subset of existing offline RL benchmarks that are well suited for reward learning and also propose new oflline reward learning benchmarks which allow for more open-ended behaviors allowing us to better test offline reward learning from preferences. When evaluated on this curated set of domains, our empirical results suggest that combining offline RL with learned human preferences can enable an agent learn perform novel tasks that were not explicitly shown in the oflline data.
1 Introduction
For automated sequential decision making systems to effectively interact with humans in the real world, these systems need to be able to safely and efficiently adapt to and learn from different users. Apprenticeship lcarning (Abbeel & Ng 2004)-also called learning from demonstrations (Argall et al. 2009) or imitation
learning (Osa et al. 2018 seeks to allow robots and other aulonomous systems to learn how to perform a
task through human feedback. Even though traditional apprenticeship learning uses expert demonstrations
(Argall et al.
2009 Osa et al. 20188 Arora Doshi 2021), preference shories-where. user asked
compare two trajectory segments- have been shown to more accurately identify the reward (Jeon al.) 2020
while reducing the burden the user to generate near-optimal actions (Wirth et al. 2017 Christiano el. al.

---

Published in Transactions on Machine Learning Research (01/2023)
Collect
X
Preference Labels
(s, a, s')
(8,a, a, s',5)
Reward Learning
Offline Dataset
RL Policy
Figure 1: Offline Preference-Basse Reward Learning (OPRL) enables safe and efficient preference-basee policy customization without any envirommental interactions. Given an oflline database consisting of trajectories, OPRL queries for pairwise preference labels over trajectory segments from the database, learns a reward function from these preference labels, and then performs offline RL using the learned reward function.
2017). However, one challenge is that asking these queries requires either executing them in the physical world (which can be unsafe), or executing them in simulator and showing them to the human (which requires simulating the world with high enough fidelity).
Learning a policy via RL has the same challenges of needing a simulator or physical rollouts (Sutton Barto 2018). To deal with these problems, offline RL has emerged as a promising way to do policy optimization with access to offline data only (Levine et al. 2020). Our key observation in this paper is that an analogous approach can be applied to reward learning: rather than synthesizing and executing preference queries in simulator in the physical world, draw an offline dataset to extract segments that would be informative for the user to compare. Since these segments have already been exccuted and recorded by the agent, all we need to do is replay them to the human, and ask for a comparison.
We call this approach Offline Preference-baset Reward Learning (OPRL), a novel approach that consists of offline preference- based reward learning combined with offline RL. Our approach is summarized in Figure [ OPRL has two appealing and desirable properties: safety and efficiency. No interactions with the environment are needed for either reward learning policy learning, removing the sample complexity and safety concerns that come from trial and error learning in the enviromment.
Although the individual components of our framework are not novel by themselves, our novel contribution lies in the fundamentally new and unexplored capability of lcarning unseon tasks during test time. For example as seen in Figure 2 even if all demoustrations are short segments of random point to point navigation, we demonstrate that OPRL can recover a policy that is able to go in a counter-clockwise orbit around the entire maze infinitely. The key to achieving this is the ability to stitch together incomplete segments from the original dataset to create one long trajectory for new task during test time.
By actively querying for comparisons between segments (or snippets) from variety of different rollouts, the agent can gain valuable information about how to customize it's behavior based on a user's preferences. This is true even when none of the data in the offline dataset explicitly demonstrates the desired behavior. For example, in Figure 2 an agent learns an orbiting behavior from human preferences, despite never having seen a complete orbit in the offline data.
To effectively study offline reward learning, we require set of benchmark domains to evaluate different approaches. However, while there are many standard RL benchmarks, there are fewer benchmarks for reward lcarning or even imitation learning more broadly. Recent work has shown that simply using standard RL benchmarks and masking the rewards is not sufficiently challenging since often learning a +1 or -1 reward everywhere is sufficient for imitating RL policies (Freire et al. 2020). While there has been progress in developing imitation learning benchmarks (Toyer et al.] [2020] Freire et al. 2020), existing domains assume an online reward learning and policy optimization setting, with full access to the environment, which is only realistic in simulation due to efficiency and safety concerns.
2

---

Published in Transactions on Machine Learning Research (01/2023)
Samples from offline data distribution
Examples of pairwise preference queries selected from offline data
States (position velocity) with highest learned reward.
Trajectories from learned policy
Figure 2: OPRL selects active queries from offline dataset consisting of random point to point navigation. These active queries elicit preferences about a human's preferences (the human wants the robot to perform counter-clockwise orbits and prefers blue over red trajectories). Note that none of the segments in the offline dataset consists of a complete counter-clockwise orbit. However, OPRL is able to recover a counter-clockwise orbiting policy given human preferences. The learned reward function matches the human's preferences and, when combined with offline RL, leads to an appropriate policy that looks significantly different from the original offline data distribution. Because offline data is often not collected for the specific task human wants, but for other tasks, being able to repurpose data from variety of sources is important for generalizing to different user preferences in offline settings where we cannot easily just gather lots of new online data.
One of our contributions is to evaluate a variety of existing offline RL benchmarks (Fu et al. 2020 in the oflline reward learning setting, where we remove access to the true reward function. Surprisingly, we find that many offline RL benchmarks are ill-suited for studying and comparing different reward learning approaches- simply replacing all actual rewards the offline dataset with zeros, or a constant, results in performance similar lo, or better than, the performance obtained using the true rewards! This is problematic since means that high performance these domaïns is not always indicative of a better learned reward rather appears that performance in many domains in mainly affected by the quality of the data (expert suboptimal) and the actual reward value has little impact offiine RL. To better isolate and study the effect of different reward learning approaches, we identify a subset of environments where simply using a trivial reward function guess results in significant degradation in policy performance, motivating the use of these environments when evaluating reward learning methods. We identify high oflline data variability and multi-tast data as two settings where reward learning is necessary for good policy learning.
Because most cxisting offline RL domains are not suitable for studying the effects of reward lcarning, we also propose several new lasks designed for open-ended reward learning in offline settings. Figure 2 shows example from experiments where the learning agent has access to dataset composed of trajectories generated offline by force-controlled pointmass robot navigating between random start and goal locations chosen along a 5x3 discrete grid. OPRL uses this dataset to learn to perform counter-clockwise orbits, a behavior never explicitly seen in the offline data. Figure shows a sequence of active queries over trajectory snippets taken from the offline dataset. The blue trajectories were labeled by the human as preferable to the red trajectories. We also visualize the learned reward function. Using a small number of queries and a dataset. generated by very different. process than the desired task, able to learn the novel orbiting behavior in completely offline manner.
We summarize the contribations of this paper as follows:
We propose and formalize the novel problem of offline preference- based reward learning.
2. We evaluate a large set of existing offline RL benchmarks, identify subset that is well-suited for ovaluating offline reward learning, and propose new benchmarks that allow for more open-ended behavior.
We perform an empirical evaluation of OPRL, comparing different methods for maintaining uncertainty and comparing different acquisition functions that use the uncertainty estimates to actively select informative queries. We provide evidence that ansemble-based disagreement queries outperform other approaches.
3

---

Published in Transactions on Machine Learning Research (01/2023)
4. Our results suggest that there is surprising value in offiine datasets. The combination of offine reward learning and offline RL leads to highly efficient reward inference and enables agents to learn to perform tasks not explicitly demonstrated in the offline dataset.
2 Related Work
Prior work on preference-based reward learning typically focuses on online methods that require actual rollouts in the environment or access to an accurate simulator or model (Wirth et al. 2017 Christiano et al, 2017 Sadigh et al. 2017 Biyik & Sadigh 2018 Brown et al. 2019 2020b; Lee et al. 2021). However, accurate simulations and model-based planning are often not possible. Furthermore, even when accurate simulator or model is available, there is the added burden of solving a difficult sim-to-real transfer problem (Tobin
let al. 2017 Peng et al. 2018: Chebotar et al. 2019). In most real-world scenarios, the standard preference learning approach involving many episodes of trial and error in the environment is likely to be unacceptable due to safety and efficiency concerns.
Prior work on safe apprenticeship learning either enables learners estimate risky actions (Zhang & Cho 2016) and request human assistance (Hoque et al., 2021), optimizes policies for tail risk rather than expected return (Lacotte et al. 2019 Javed et al. 2021), provides high-confidence bounds on the performance of the agent's s policy when learning from demonstrations (Brown & Niekum 2018 Brown et al. 2020a however, these methods all rely on either an accurate dynamics model or direct interactions with the environment. By contrast, our approach towards safety is to develop a fully offline apprenticeship learning algorithm to avoid costly and potcntially unsafe physical data collection during reward and policy lcarning.
While there has been some work on offline apprenticeship learning, prior work focuses on simple environments with discrete actions and hand-crafted reward features (Klein et al. 2011 Bica et al. datasets that consist of expert demonstrations that are optimal for a particular task (Lee et al. 2019
2021 and requires
Other
work has considered higher-dinenssonall continuous tasks, but assumes access to expert demonst rations or
requires experts to label trajectories with explicit reward values (Cabi et al. 2019 Zolna et al. 2020 By contrast, we focus on fully offline reward learning via small numbers of qualitative preference queries which are known to be much easier to provide than fine- ane-grained reward labels or near-optimal demonstrations (Kendall, 1948: Stewart et al., 2005; Saaay 2008 Wirth et al., 2017). . Prior work by Castro al. (2019) is similar to ours in that they learn reward function from offline ranked demonstrations of varying qualitics. However, their proposed approach is only evaluated on MDPs with finite state spaces where they learn unique reward for each state by solving a quadratic program. By contrast, our approach learns reward features from low-level, continuous state information and uses a diffcrentiable pairwisc preference loss function rather than quadratic programmingg
Other prior work has considered offline imitation learning. Methods such as behavioral cloning are trained offline expert demonstrations, but suffer from compounding errors (Pomerlcaul 1988). DemoDICE (Kim) et al. 2021 seeks to imitate expert demonstrations that are provided offiine and improves stability by also leveraging dataset of suboptimal demonstrations. By contrast, our approach does not require expert demonstrations, just an offline dataset of state transitions and learns an explicit reward function from preferences. IQ-Learn (Garg et al.) 2021) is capable of both offline and online imitation learning. Rather than learning a reward function, they learn a parameterized Q-function. Similar to DemoDICF, TQ-Learn requires access to expert demonstrations, whercas we only require human preference labels over offline data of any quality some of our experiments use data generated from a completely random policy. The main downside to imitation learning methods like DemoDICE and IQ-Learn is the requirement of having expert demonstrations. We avoid this problem by learning rewards from preference labels over pairs of trajectory segments, something that is easily provided, even for humans who cannot provide expert demonstrations.
There has also been previous work that focused on apprenticeship learning from heterogeneous human demonstrations in resource schoduling and coordination problems (Paleja et al. 2019). Our approach also works by learning from heterogeneous data, but uses preference queries to learn a reward function which is then used for offline RL algorithms rather than learning scheduling algorithm from human remonstrations.
4

---

Published in Transactions on Machine Learning Research (01/2023)
3 Problem Definition
We model our problem as a Markov decision process (MDP), defined by the tuple A,, P. Po:o7), 7), where $ denotes the state space, A denotes the action space, S: x + R denotes the reward, P(s"|s, [s, a) denotes the transition dynamics, Po(s) denotes the initial state distribution, and y E (0,1) denotes the discount factor.
In contrast to standard RL, do not assume access to the reward function Furthermore, do not assume access to the MDP during training. Instead, are provided a static dataset, D, consisting of trajectories, == {To,..., TN), where each trajectory тi consists of contiguous sequence of state, action, next-state transitions tuples T; {[sL,, ag,0, (..,),... (St. ((M)) Unlike imitation learning, we do not assume that this dataset comes from a single expert attempting to optimize specific reward function r(s, a). Instead, the dataset may contain data collected randomly, data collected from a variety of policies, or even from a variety of demonstrations for variety of tasks.
Instead of having access to the reward function, r, assume access to expert that can provide small number of pairwise preferences over trajectory snippets from offline dataset D. Given these preferences, the goal is to find a policy x <(ass) that maximizes the expected cumulative discounted rewards (also known the discounted returns), (T) = P,,0 2 7'r(s,, dz)], under the unknown true reward function in a fully offiine fashion we assume no access to the MDP other than the offline trajectories contained in D.
4 Offline Preference-Basede Reward Learning
We now discuss proposed algorithm, Offline Preference-basee based Reward Learning (OPRL). OPRL is an offline, active preference- based learning approach which first searches over offline dataset of trajectories and soquentially selects new queries to be labeled by the human expert in order to learn a reward function that models the user's preferences and can be used to generate a customized policy based on a user's preferences via offline RL.
As established by large body of prior work (Christiano et al. 2017 Ibarz al. 2018 Brown et al. 2019 Palan et al., 2019), the Bradley-Terry pairwise preference model (Bradley & Terry 1952) is an appropriate model for learning reward functions from user preferences over trajectories. Given preference over trajectories, Ti Tj. we seek to maximize the probability of the preference label:
Σ fo(s)
exp
P(T. 0T)
SETy
(1)
exp
fa(s) exp
Σ fo(s)
SET,
by approximating the reward at state 8 using neural network, fo(s), such that Even < Ever, Po(s) when T; Ty-
OPRL actively queries to obtain informative preference labels over trajectory snippets sampled from the offline dataset. In order to perform active learning, OPRL needs a way to estimate the uncertainty over the predicted preference label for unlabeled trajectory pairs, in order to determine which queries would be most informative (result the largest reduction uncertainty over the true reward function). this paper, we investigate two different methods represent uncertainty (ensembles and Bayesian dropout) along with different acquisiiiinn functions (disagreement and information gain). We describe these approaches below.
4.1 Representing Reward Uncertainty
We compare two of the most popular methods for obtaining uncertainty estimates when using deep learning: eusembbles (Lakshminarayanat et al. (0166) and Bayesian dropout (Gal & Ghahramani 2016). On line 4 of Algorithm we initialize an ensemble of reward models for ensemble queries or a single reward model for Bayesian dropout.
Ensemble Queries Following work on online active preference learning work by Christiano al. (2017). we test the effectiveness of training ensemble of reward models approximate the reward function posterior
5

---

Disagreement When using disagreement lo select active queries, we select pairs with the highest ensemble disagreement among the different return estimates obtained from either ensembling or Bayesian dropout. Following Christiano et al. (2017), caleulate disagreement as the variance in the binary comparison predictions: if fraction of the posterior samples predict T₁ > Ty while the other 1 - p ensemble models Published in Transactions on Machine Learning Research (01/2023)
using the Bradley- Terry preference model. Similar to prior work (Christiano et al. 2017 Reddy et al. 2020 we found that initializing the ensemble networks with different seeds was sufficient to produce diverse set of reward functions.
Bayesian Dropout We also test the effectiveness of using dropout to approximate the reward function posterior. As proposed by Gal & Ghahramani (2016), we train a reward network using pairwise preferences and apply dropout to the last layer of the network during training. To predict a return ributiio over candidate pairs trajectories, we still apply dropout and pass each trajectory through the network multiple times to obtain a distribution over the returns.
4.2 Active Learning Query Selection
In contrast to prior work on active preference learning, we do not require on-policy rollouts in the environ- ment. (Christiano et al. 2017: Tee et al.) 2021) nor require synthesizing queries using model (Sadigh et al. 2017). Instead, we generate candidate queries by searching over randomly chosen pairs of sub-trajectories obtained from the offline dataset. Given a distribution over likely returns for each trajectory snippet in a candidate preference query, we estimate the value of obtaining label for this candidate query. We consider two methods for computing the value of query: disagreement and information gain. We then take the trajectory pair with the highest predicted value and ask for pairwise preference label. On line of Algorithm | we can either use disagreement information gain the estimated value of information (VOI).
predict 7, I Tj then the variance of the query pair (Ti, is p(1 p)-
Information Gain Queries As an alternative to disagreement, we also consider the expected information gain (Cover) 1999) between the reward function parameters and the outcome Y of querying the human for a preference. We model our uncertainty using an approximate posterior p(0 DD, given by training an ensemble or dropout network on our offline dataset D. Houlsby el al. (2011) show that the information gain of potential query can be formulated as:
[ D) = H(Y D) - (a |6, D)).
(2)
Intuitively, the information gain will be maximized when the first term is high, meaning that the overall model has high entropy, but the second term is low, meaning that each individual hypothesis 0 from the posterior assigns low entropy to the outcome Y. This will happen when the individual hypot strongly disagree with each other and there is no clear majority. We approximate both terms in the information gain equation with samples obtained via ensemble or dropout. See Appendix D for further details.
Searching the Offline Dataset for Informative Queries We note that this process of searching for informative queries can be performed quite efficiently since the information gain or ensemble disagreement for each candidate query can be computed in parallel. Furthermore, we can leverage GPU parallelization by feeding all of the states in one more trajectories into the reward function network as a batch. Finally, we continue
note that scarching for the next active query can be formulated to sample random pairs of trajectories from the oflline dalaset to evaluate and when particular time computation limit is reached and then simply return the most informative query found so far.
any-time algorithm-
4.3 Policy Optimization
Given a lcarned reward function obtained via preference-learning, we can then use any existing offline or batch RL algorithm (Levine et al. 2020) learn policy without requiring knowledge of the transition dynamics rollouts in the actual environment. The entire OPRL pipeline is summarized in Algorithm
6

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm 1 OPRL
Require: A dataset D (To...., TNY, where each trajectory T; consists of a,,,
(8Lar, M; 8(,M)
2: // REWARD LEARNING
3: Generate dataset of pairs of snippets Danip
4: Initialize reward model fg
for each iteration do Compute estimate of VOI across all pairs in Danip
Find snippet pair (Ta,1, Ta.2) with highest estimated VOI Query preference label y
Store query pair and label Drew +- (Ts,1, Ta,2: v) 10: Train reward model with updated Drew
11: end for
12: Label transitions D with Pvp.
13: // POLICY LEARNING
14: Run selected offline RL algorithm DD
5 Experiments and Results
We first evaluate a variety of popular offline RL benchmarks from D4RL Fuu ct al. 2020) to determine which domains are most suited for evaluating oflline reward learning. Prior work on reward learning has shown that simply showing good imitation learning performance on an RL benchmark is not sufficient to demonstrate good reward learning (Freire al.j 2020). In particular, we soek domains where simply using an all zero or constant reward does not result good performance. After isolating several domains where learning a shaped reward actually matters (Section 5.1 : we evaluate OPRL on these selected benchmark domains and investigate the performance of different active query strategies (Section 5.2 Finally, propose and evaluate several tasks specifically designed for offline reward learning (Section 5.3 To further analyze our method,
we include sensitivity analyses on the number of initial queries, queries per round, ensemble models, and dropout samples (Appendix B
Videos of learned behavior and code is available in the Supplement.
5.1 Evaluating Offline RL Benchmarks
An important assumption made our problem statement is that we do not have access to the reward function. Before describing our approach for reward learning, we first investigate in what cases we actually need to learn reward function. We study a set of popular benchmarks used for offline RT (Fu et al. 2020 To test. the sensitivity of these methods to the reward function, we evaluate the performance of offline RL algorithms these benchmarks when set the reward equal to zero for each transition in the offline dataset and when we set all the rewards equal to constant (the average reward over the entire dataset).
We evaluated four of the most commonly used offline RL algorithms: Advantage Weighted Regression (AWR) (Pnng et al. 2019), Batch-Constrained deep Q-learning (BCQ) (Fujimoto et al. 2019 Bootstrapping Error Accumulation Reduction (BEAR) (Kumar et al., 2019), and Conservative Q-Learning (CQL. {Kunar ct 2020). Table [ shows the resulting performance across a large number of tasks from the D4RL benchmarks (Fu et al. 2020). We find that. for tasks with only expert dala, there is need to learn reward function since using trivial reward function often leads to expert performance. We attribute this to the fact that many offline RL algorithms are similar to behavioral cloning (Levine et al. 2020). Thus, when an offline RL benchmarks consists of only expert data, offline RL algorithms do not need reward information to perform well. In Appendix F we show that AWR reduces to behavioral cloning in the case of an all zero reward function.
Conversely, when the offline dataset contains a wide variety of data that is either random or multi-tase tast (e.g. the Maze2D environment consists of an agent traveling to randomly generated goal locations), we find
7

---

Published in Transactions on Machine Learning Research (01/2023)
Table Offline RL performance on D4RL benchmark tasks comparing the performance with the true reward performance using a constant reward equal the average reward over the dataset (Avg) and zero rewards everywhere (Zero). Even when all the rewards are set the average still perform surprisingly well. In this table, present the experiments over three random seeds. Bolded environments are ones where find the degradation percentage to be over chosen threshold and are used for later experiments with OPRL. The degradation percentage is calculated max(GT-max AVG. ZERO RANDOM).) 100% and the threshold is set to 20%. The degradation percentage is G'T-min( (A...A/AA.R.OO..... CAAERORRRADOON used to determine domains where trivial reward functions do not lead to good performance isolate the effect of reward learning in later experiments.
zero, many offine RL algorithms
with AWR. Results averages
TASK
GT
AVG
ZERO
RANDOM
DEGRADATION %
WOW-RING-RANDON
-42.5
-42.9
-44.3
-66.22
0.3
FLOW-MEHGE-RANDOM
160.3
85.2
85.6
117.0
57.7
MAZE2D-UMAZE
104.4
56.5
55.0
49.5
87.3
MAZE2D-MEDIUM
134.6
30.5
34.5
44.8
86.3
HALFCHEETAHH
11.0
-49.4
-48.3
-285.8
20.0
WALFCHEEETATH-MEDIMMMRRRR
4138.2
3934.7
3830.2
285.8
4.6
HALFCHERTAH--EEDII
4096.0
3984.4
3898.3
285.8
2.5
HALFCHEETAH--MEUS M-EXPERT
669.9
401.7
560.5
-85.88
11.4
EEPEET HALFCHEETAH- EXPERT
467.8
511.1
493.1
285.8
0.0
HOPPER- RANDOM
HOPPER-RANDOM
123.3
129.8
29.1
18.3
0.0
HOPPER- MEDIUM-REPPAY
1045.6
1127.4
954.6
18.3
0.0
SUPPER-MEDIUS
1152.1
1192.8
963.6
18.3
0.0
HOPPER--MEDDUM-EEPEIRY
623
615.8
587.3
18.3
1.2
ПОРPER-EXXPEиT
571.7
617.2
427.6
18.3
0.0
KITCHEN-COMPLETT
0.3
0.2
0.3
0.0
25.0
KITCHEN-MIXEN
0.3
0.5
0.3
16.7
KITCHEN-PARTIAL
0.3
0.2
0.3
0.0
0.0
that running offline RL algorithms on the dataset with an all zero constant reward function significantly degrades performance.
5.2 Reward Learning on Subset of D4RL
To evaluate the benefit of offline reward learning, we focus our atlention on the D4RL benchmark domains shown in Figure g which showed significant degradation in Table [ since these are domains where we can confident that good performance is due learning a good reward function. We define significant degradation to be degradation percentage over 20%, which chosen to give enough room for OPRL to improve in comparison to uninformative rewards like avg the Flow Merge traffic simulation, the Halfcheetah random environment, and the Franka Kitchen-Complee environment. To facilitate a better comparison across different methods for active learning, we use oracle preference labels, where one trajectory sequence is preferred over another if it has higher ground-truth return. We report the performance of OPRL using AWR (Peng et al. 2019), which we empirically found to work for policy optimization across the different tasks. In the appendix we also report results when using CQL (Kumar et al. 2020 for policy optimization.
masking. We selected both of the Maze environments,
5.2.1
Maze Navigation
We first consider the Maze2d domain Fuu et al. 2020). which involves moving a forco-actuated ball (along the and axis) to fixed target location. The observation dimensional, which consists of the (æ, v).8) location and velocities. The offline dataset consists of one continuous trajectory of the agent navigation to
8

---

Published in Transactions on Machine Learning Research (01/2023)
(a) U-Maze
(b) Medium Maze
(c) Flow Merge
(d) Halfcheetah
(e) Franka Kitchen
Figure 3: Experimental domains chosen from D4RL Fu et al.
2020 for use offline preference-based reward
learning.
150
round
100
20
round
90
Train Return
Train Return
GT Reword
75
-20
60
round
round 1
round
raund 5
50
GT Reward
GT Reward
-80
100
300
100
100
200 teeration
100
200
400
600
800
1000
(a) Maze2D-Umaze
(b) Maze2D-Meddimm
(c) Half-Cheetah-v2
Figure 4: (a) Ensemble disagreement after 10 rounds of active queries (1 query per round) Maze2d-Umaze. (b) Ensemble disagreement after 10 rounds of active queries (1 query per round) in Maze2d- (c) Ensemble disagrecment after 10 rounds of active queries (10 queries per round) in Halfchoctah.
random intermediate goal locations. The true reward, which we only use for providing synthetic preference labels, is the negative exponentiated distance to a held-out goal location.
For our experimental setup, we first randomly select 5 pairs of trajectory snippets and train 5 epochs with our models. After this initial training process, for each round, one additional pair of trajectories is queried be added to the training set and we train more epoch this augmented dataset. The learned reward model is then used to predict the reward labels for all the state transitions in the offline dataset, which is then used to train policy via offline RL (e.g. AWR (Peng et al. 2019)). As baseline, we also compare against a random, non-active query baselinc.
The results are provided Table 2 We found that ensemble disagreement the best performing approach for both Maze2D-Umaal and Mediuu and outperforms T-REX. The offline RL performance for ensemble disagreement
shown in Figure | We find that ensemble disagreement on Maze2D-Umans is able to quickly approach the performance when using the ground-trute reward function the full dataset of million state transitions after only 15 preference queries (5 initial + 10 active). Mazc2D-Medium is more difficult and we find that 15 preference queries is unable to recover a policy on par with the policy trained with ground truth rewards. We hypothesize that this due to the fact that Maze2D is goal oriented and learning a reward with trajectory comparisons might be difficult since the destination matters more than the path.
5.2.2 MuJoCo Tasks
We next tested OPRL on the MuJoCo environment Halfcheetah-v2. For the tasks, the agent is rewarded for making the Halfcheetah move forward as fast as possible. The MuJoCo tasks presents an additional challenge comparison to the Maze2d domain since they have higher dimensional observations spaces, for Halfcheetah-v2. The dataset contains 1 million transitions obtained by rolling out random policy. Our experimental setup is similar to Maze2D, except start with 50 pairs of trajectories instead of 5 and
9

---

Published in Transactions on Machine Learning Research (01/2023)
Table 2: OPRL Performance Using Reward Predicted After N Rounds of Querying: Offline RL performance using rewards predicted either using static number of randomly selected pairwise preferences (T- REX) Brown al. 2019) or active queries using Ensemble Disagreement (ineemDDs) Ensemble Information Gain (EnsemInfo), Dropout Disagreement (DropDis), and Dropout Information Gain (DropInfo). N is set 15 for Halfcheetah. N is selected
to for Mlaze2D-Umaze 10 for Maze2d-Medium and how-merge-randeee based on the size of the dimension of observation space and the complexity of the environment. Results are averages over three random seeds and are normalized so the performance of offline RI. using the ground-truth rewards 100.0 and performance a random policy 0.0. The results provide evidence that Ensembles are at least. as good and often much better than random queries and that Dropout often fails to perform as well as random queries.
QUERY AQUISITION METHOD ENSEMINFO
RANDOM
ENSEMDIS
DROPDIS
DROPINFO
MAZE2D- UMAZE
78.2
93.5
89.2
88.0
61.3
MAZE2D- MEDIUM
71.6
86.4
71.0
52.6
44.4
IALFCIEEETAI-IANNONM
96.1
113.7
100.6
84.4
91.7
FLOW-MERRGE-RANDON
110.1
89.0
92.1
86.2
84.2
KITCHEN-COMPLETE
79.6
105.0
158.8
48.5
65.4
MALECHEETTAA--EEUUAAAAA
98.8
100.0
99.7
96.2
94.0
HALFCHEETANH-MEDDIU
102.2
103.0
100.7
94.4
99.1
TALFCHHETTAE-MMRDDUM M-XXPRT
111.4
91.0
112.8
86.0
113.9
TAL-CHIEFTAI--XPPPRT
104.3
176.4
124.6
86.0
129.3
HOPPER- RANDOM
81.9
86.7
64.7
72.0
75.1
HOPPER-MEDIUM MEDIUM-REPLAN
97.8
98.4
103.5
94.6
97.3
HOPPER- MEDIUM
90.4
93.6
91.7
89.4
92.0
HOOFEER--EDDDMM-EEPPIII
81.2
81.4
81.0
90.0
84.3
HOPPER-EXPERR
69.0
93.2
122.4
94.0
100.4
KITCHEN-MIXEN
110.0
120.0
96.7
110.0
90.0
KITCHEN-PARTIA
91.2
111.8
117.6
126.5
97.1
add 10 trajectorics per round of active queries instead of 1 query per round. We found that increasing the number of queries was helpful due to the fact that MuJoCo tasks have higher dimensional observational spaces, which requires more data points to learn an accurate reward function. The learning curve is shown in Figure (c). We found ensemble disagroement
be the best performing approach for the Halfchootah task. We noticeable improvement the number of active queries increases, however, 10 rounds of active queries is not sufficient. to reach the performance as the policy learned with the ground truth reward.
5.2.3 Flow Merge
The Flow Merge environment involves directing traffic such that the flow of traffic is maximized in highway merging scenario. Results shown in Table 2 Random queries achieved the best performance, but all query methods were able to recover near ground truth performance.
5.2.4 Robotic Manipulation
The Franka Kitchen domain involves interacting with various objects in the kitchen to reach certain state configuration. The objects you can interact with include water kettle, light switch, microwave, and cabinets. Our results in Table [ show that setting the reward to all zero or a constant reward causes the performance to degrade significantly. Interestingly, the results in Table 2 show that OPRL using Ensemble Disagroement and Ensemble InfoGain are able to shape the reward in a way that actually leads to better performance than using the ground-trute manthttruthh reward for the task. In the appendix, we plot the learning curves and find that OPRL also converges to better performance faster.
10

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 5: Constrained Goal Navigation. The highlighted yellow region represents a constraint region that the human prefers the agent to avoid while also traveling to the goal position shown in red. OPRL produces trajectories that match this preference by laking a more round-about, but more preferred, path to the goal.
5.3 New Offline Preference-Basse Reward Learning Tasks
To explore the full benefits of learning a reward function from preferences, we propose and study several tasks to highlight the need for shaped reward function, rather than simply goal indicator. To create these environments we adapted common gym environments including several that are in the D4RL set of benchmarks. The following section shows that OPRL can learn novel behaviors despite the offiine data never containing successful or complete examples of these behaviors. We believe this is an exciting result that has not been previously demonstrated or recognized as possible in prior preference learning work.
5.3.1 Maze Navigation with Constraint Region
We created variant of the Maze2d-Medium task where there is constraint region in the middle of the maze that the supervisor does not want the robot to enter. Figure 5 shows this scenario where entering the highlighted yellow region is undesirable. After only 25 active queries using ensemble disagreement, we obtain the behavior shown in Figure where the offline RL policy has learned to reach the goal while avoiding the constraint region.
5.3.2 Open Maze Behaviors
Next, we took the D4RT, Open Maze enviromment and dataset and used it to teach an agent to patrol the domain in counter-clockwise orbits, clockwise orbits, approximating digital 2 shape, preferring to stay at the top, and approximating a "Z" shape. The open maze counter-olockwis orbiting uses human preferences provided by one of the authors by showing the human user trajectory snippets visualized as in 2 and querying preferences. The dataset only contains the agent moving randomly chosen goal locations, thus this domain highlights the benefits of stitching together data from an offline database as well as the benefits of learning a shaped reward function rather than simply using a goal classifier to use as the reward function which has been used in prior work offline RL work (Eysenbach et al., (021)). The original data distribution, samples of the preference queries, and the resulting learned policies are shown in figures 2 and 0 Our results involving the open maze behaviors in Figure G and our work on constrained maze in Figure can be seen as a mixture of experts since the data consists of a mixture of demonstrations for a variety of starts and goals. OPRL can leverage diverse data to achieve different preferences (including ones not explicitly demonstrated), such as taking longer rather than shorter path a goal because of a specific user constraint preference in the medium maze, various behaviors in the open maze.
5.3.3 Open Ended CartPole Behaviors
We created an oflline datasel of 1,000 random trajectories of length 200 in a modified CartPole domain where the trajectories only terminate at the end of 200 timesteps this allows the pole to swing below the track and also allows the cart to move off the visible track to the left or right. Given this dataset of behavior agnostic data, we wanted see whether we could optimize different policies, corresponding to different human preferences. We also this data from random controller as highly suboptimal data for the following
11

---

Published in Transactions on Machine Learning Research (01/2023)
Figure 6: Learned "Shape" Policies adapt particular user's preferences learn counterclockwise orbits, digital 2 shape, preferring stay at the top, and "Z" shape, respectively. Green circles represent random initial states and red circles represent the position of the agent at the end of rollout.
Table Quantitative Results for Open Ended Cartpole Behaviors: We evaluated the trained OPRL policies calculate how often the pole was balanced (0 < =24) and in view (cart position in [-2.4,2.4]) or in view and windmilling clockwise or counter-clockwise (by looking at the sign of the angular velocity). This table contains results showing the average (standard error of the mean) number of time steps in 200-length trajectory that the different behaviors are exhibited in the randomly collected Offline Data (first row) and when rolling out the trained OPRL policy (second row).
BALANCE AvG. STEPS
WINDMILL (CW)
WINDMILL (CCW)
SE
AvG. STEPS
SE
AvG. STEPS
SE
OFFLINE DATA
23.07
(0.37)
71.39
(2.01)
72.12
(2.06)
OPRL
111.35
(3.36)
128.20
(0.14)
190.37
(1.43)
tasks. We optimized the following policies: balance, where the supervisor prefers the pole to be balanced upright and prefers the cart to stay in the middle of the track; clockwise windmilll, where the supervisor prefers the cart to stay in the middle of the track but prefers the pole to swing around as fast as possible in the clockwise direction; and coumter-clockuisc windmill, which is identical to clockwise windmill except the preference is for the pole to swing in the counter-clockwise direction. OPRL is able to learn policies for all three behaviors. See the supplementary video for examples of the learned behaviors.
To quantitatively evaluate OPRL on these open-ended behaviors, we mcasure the average number of steps that OPRL policy is performing the task (e.g. balanced, windmill clockwise, windmill counter-clockwise) compared the average number of steps these behavior appear in the original dataset as seen in Table 3 We see that OPRL is able perform the task about 5 times more frequently for the balance task and around 2 times more frequently for the windmill tasks when compared to the original dataset. This gives evidence that OPRL is able to leverage offline data to optimize tasks not explicitly shown in the data.
5.3.4 Pilot User Study
We ran a small user study where we recruited participants and asked them give pairwise preference labels to teach an agent goal location in Mediun-Maze (Fig 3(b)) and a counter-clockwise orbiting behavior in OpenMaze (Sec 5.3.2). For Medium-Maze
quantitatively evaluated the performance of OPRL with respect to the GT reward from D4RL. Similar to simulation results, found that EnsemDis version of OPRL performed the best, achieving scores of 122.6 compared to GT performance of 134.6 and the remaining results are included in Table 10 All users were able to successfully use OPRL to teach an orbiting behavior nearly identical to Fig (left) as shown in Fig in the Appendix. In Table 10 we show results per user as well as results when aggregating all user preferences labels. We find that the combined data set (POOLED) achioved performance better than most of the individual users. We hypothesize this is because each user only answered 100 random queries which were used to create the pool for active learning. The POOLED results show the
12

---

Published in Transactions on Machine Learning Research (01/2023)
benefit of having large-enough active learning pool when performing offiine preference learning. Overall, we found that users were able to provide preference labels that enable offline RI. to generate behaviors aligned with human users.
6 Discussion and Future Work
Our goal is to learn policy from user preferences without requiring access to model, simulator, interactions with the environment. Toward this goal, propose OPRL: Offline Preference-based Reward Learning. We evaluate several different offline RL benchmarks and find that those that contain mainly random data or multi-task data are best suited for reward inference. While most prior work on reward learning has focused on learning from expert demonstrations, preference-based reward learning enables us to learn a user's intent even from random datasets by selecting informative sub-sequences of transitions and requesting pairwise preferences over these sub-sequences. Our results suggest that using ensemble disagreement as an acquisition function is the best choice since it leads to the best performance for 3 out of the 5 tasks and outperforms the random baseline for all tasks. Additionally, using ensemble disagreement with only 10-15 preference queries we are able to learn policies with similar performance to policies trained with full access to lens of thousands of ground-truth reward samples. Finally, our results suggest there is surprising value in offline datasets even if the datasets do not explicitly contain data that matches a user's preferences or if the datascts contain suboptimal data. We are excited about the potential of offline reward and policy learning and believe that our work provides stepping stone towards safer and more efficient autonomous systems that can better learn from and interact with humans. Preference-learning enables users to teach system without needing to be able to directly control the system demonstrate optimal behavior. We believe this makes preferences idcally suited for many tasks in as healthcare, finance, or robotics. Future work includes developing more complex and realistic datasets and benchmarks and developing oflline RL algorithms and active query strategies that are well-suited for reward inference and offline RI.
References
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In Advances in Neural Information Processing Systems, 2021.
Brenna D Argall, Sonia. Chernova, Manuela. Veloso, and Brett Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, 57(5):469-483, 2009.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, pp. 103500, 2021.
Toana Biea, Daniel Jarrett, Alihan Hüyük, and Mihaela van der Schaar. Learning' what-if" explanations for sequential macision-making In International Conference on Learning Representations, 2021.
Erdem Biyik and Dorsa Sadigh. Batch active preference-based learning of reward functions. In Conference on robot learning, pp. 519-528. PMLR, 2018
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrikn, 39(3/4):324-3M)) 1952.
Daniel Brown and Scott Niekum. Efficient, probabilistic performance bounds for inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Daniel Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Nickum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In International Conference on Machine Learning, pp. 783 792. PMLR, 2019.
13

---

Published in Transactions on Machine Learning Research (01/2023)
Daniel Brown, Russell Coleman, Ravi Srinivasan, and Scott Nickum. Safe imitation learning via fast bayesian reward inference from preferences. In International Conference on Machine Learning, pp. 1165- 1177. PMLR, 2020a.
Daniel Brown, Wonjoon Goo, and Scott Niekum. Better-than-demmonstaaaao imitation learning via automatically-ranbee demonstrations. In Conference on robot learning, pp. 330-359. PMLR, 2020b.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rac Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning, arXiv preprint Xri::1199 12200, 2019.
Pablo Sammel Castro, Shijian Li, and Daqing Zhang. Inverse reinforcement learning with multiple ranked experts. arXiv preprint arXiv:1907. 1941199977.111,, 2019.
Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In 2019 International Conference on Robotics and Automation (ICRA), pp. 8973-8979. IEEE, 2019.
Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Logg, and Dario Amodei. Deep reinforcement learning from human preferences. In NIPS, 2017.
Thomas M Cover. Elements of information theory. John Wiley Sons, 1999.
Benjamin Eysenbach, Sergey Levine, and Ruslan Salakhuldinov. Replacing rewards with examples: Example based policy search via recursive classification. arXiv preprint (rXie:2103.12636 2021.
Pedro Freire, Adam Gleave, Sam Toyer, and Stuart Russell. Derail: Diagnostic environments for reward and imitation learning. In Proceedings of the Workshop on Deep Reinforcement Learning at NeurIPS, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. XrXiv preprinl Xki:20044007111 2020.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International Conference on Machine Learning, pp.
PMLR, 2019.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Newral Information Processing Systems, 34:4028 4039, 2021.
Ryan Hoque, Ashwin Balakrishna, Ellen Novoseller, Albert Wilcox, Daniel Brown, and Ken Goldberg. Thriftydagger: Budget- aware novelty and risk gating for interactive imitation learning. In 5th Annual Conference Robot Learning, 2021.
Neil Houlsby, Ferene Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. aixiv preprint arXiv: 11t..111257754 2011.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning 2018.
from human preferences and demonstrations in atari. arXie preprint
Zaynah Javed, Daniel S Brown, Satvik Sharma, Jerry Zhu, Ashwin Balakrishna, Marek Petrik, Anca Dragan, and Ken Goldberg, Policy gradient bayesian robust optinization for imitation learning, In International Conference on Machine Learning, pp. 4785-4796. PMLR, 2021.
Hong Jun Jeon, Smitha Milli, and Anca D Dragan. Reward-rational (implicit) choice: unifying formalism for reward learning. Tn Neural Information Processing Systems, 2020.
Maurice George Kendall. Rank correlation methods. 1948.
14

---

Published in Transactions on Machine Learning Research (01/2023)
Geon-Hyeong Kim, Scokin Seo, Jongmin Lee, Wonseok Jeon, HycongJoo Hwang, Hongscok Yang, and Kee-Fang Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021.
Fdouard Klein, Matthieu Geist, and Olivier Pietquin. Batch, off-polic and model-free apprenticeship learning. In European Workshop on Reinforcement Learning, pp. 283-296. Springer, 2011.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 190:099900094, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. arXiv preprint ar fft::006004779, 2020.
Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, and Marco Pavone. Risk-sensitive generative adversarial imitation learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2154 2163. PMLR, 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint arXi:16112.0147,, 2016.
Donghum Lee, Srivatsan Srinivasan, and Finale Doshi Velez. Truly batch apprenticeship learning with deep successor features. ar preprint 0rXiv:1903110070 2019.
Kimin Loe, Laura Smith, and Picter Abbecl. Pebble: Feodback-cfficiett interactive reinforcement learning via relabeling experience and unsupervised pre-training. arXiv preprinl arXiv:2106.000 20000091,, 2021.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, reviow, and perspectives on open problems. arXiv preprint 2tl:20055001643 2020.
Farzan Memarian, Wonjoon Goo, Rudolf Lioutíkov, Ufuk Topcu, and Scott Niekum. Self-supervised online reward shaping sparse-reward environments. In International Conference on Intelligent Robots and Systems (IROS), 2021.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective on imitation learning. arXiv preprint drX1::181100611,, 2018.
Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. Roboties: Science and Systems, 2019.
Rohan Paleja, Andrew Silva, Letian Chen, and Matthew Gombolay. Interpretable and personalized appren- ticeship scheduling: Learning intorpretable scheduling policies from heterogeneous user demonstrations. arXiv preprint orXiv:196600633 2019.
Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pp. 3803 3810. IEEE, 2018.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. weiigtte regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint orXiv:1910.0011 2019.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information
processing systems, 1, 1988.
Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning human objectives by evaluating hypothetical behavior. In International Conference on Machine Learning, pp. 8020-8020 PMLR, 2020.
Thomas L Saaty. Relative measurement and its generalization in decision making why pairwise comparisons are central in mathematics for the measurement of intangible factors the analytic hierarchy/netee process. L.CSAM-Revisse de la Real Academia de Ciencias Eractos, Físicas y Naturales. Serie A. Matematicas, 102(2):251-31) 2008
15

---

Published in Transactions on Machine Learning Research (01/2023)
Dorsa Sadigh, Anca. D Dragan, Shankar Sastry, and Sanjit A Seshia, Active preference-basee learning of reward functions. In Robolics: Science and Systems, 2017.
Neil Stewart, Gordon DA Brown, and Nick Chater. Absolute identification by relative judgment. Psychological review, 112(4):881, 2005.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), 23 30. IEEE, 2017.
Sam Toyer, Rohin Shah, Andrew Critch, and Stuart Russell. The magical benchmark for robust imitation. arXiv preprint (:X:x:2011.04000 2020.
Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learving Research, 18(136):1 46, 2017.
Jiakai Zhang and Kyunghyun Cho. Query-efficient imitation learning for end-to-end autonomous driving. arXiv preprint
660500650, 2016.
Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. Xiv preprint ar arxix:2011.1880 2020.
16

---

Published in Transactions on Machine Learning Research (01/2023)
7 Appendix
A OPRL Performance with IQM
We present the results of methods using Interquartile MIa) instead of mean which was used in table 2 IQM a statistic that is more robust to outliers than the mean and suggested for offline reinforcement lcarning by Agarwal ct al. (2021). IQM is calculated by discarding the bottom and top 25% of the statistics and calculating the mean of the remaining 50%. For returns from seeds, the IQM calculated as 0.25R1 + R2 + 0.25Rg
where R1 < R2 < R3 are the returns sorted from lowest to highest.
1.5
Table 4: OPRL Performance Using Reward Predicted After N Rounds of Querying using IQM
QUERY ACQUISITION METHOD ENSEMINPO
RANDOM
ENSEMDIS
DROPDIS DROPINFO
MAZE2D-UMAZE
89.5
96.2
95.1
81.7
64.7
MAZE2D-MEDIUM
90.6
91.4
86.5
77.7
36.7
HALFCHEETAH RANDOOM
99.3
101.6
100.7
86.8
88.8
KITCHEN-COMPLETE
75.2
138.5
102.8
82.6
125.7
MAAUFCHEETAAI-MEDDIUM
99.6
100.3
99.7
96.7
93.5
HAJ-CCEERTAE-MMEDIUM
101.8
102.7
100.6
93.7
98.6
TAL.CCHETTAF-MMEDDUUM EXPERT
63.7
65.0
76.3
68.4
108.6
MALFCHEETAH-EXPPAE
81.8
127.3
66.4
56.3
87.0
HOPPER- RANDOM
72.8
74.1
69.2
55.2
57.4
IIOFFERE-MEDDDMMM-EEPP...
95.9
96.7
100.7
93.9
95.4
HOPPER--EDDUM
91.4
91.9
89.1
87.4
89.7
COPFEER--EEDIMM-EXPIII
73.0
84.2
92.9
87.4
90.1
HOPPER-EXPERT
84.0
95.5
100.3
104.9
94.6
KITCHEN-MIXED
99.1
105.3
86.7
86.7
61.9
KITCHEN-PARIE
77.9
108.0
82.9
98.0
55.3
B Sensitivity Analysis on Number of Queries
We present sensitivity analysis on the number of initial queries and number of queries for ensemble disagreement in the halfcheotab-randomm-y environment.
Table sensitivity analysis is run on the half-cheetab-randdomyy environment. The default uses 50 initial queries and includes 10 additional queries per round. this table, we vary the number of initial queries while koeping the additional number of queries per round fixod. The results are reported with sceds and aggregated using the interquartile
NUMBER OF INITIAL QUERIES
25
50
100
RANDOM
25.7
34.2
36.3
ENSEMDIS
26.4
41.6
36.9
We find that more initial queries improve performance for random queries. However, ensemble disagreement performs best with 50 initial queries. Ensemble disagrecment outperforms random queries for all numbers of initial queries. The improvement from using ensemble disagreemeett vs. random queries is less significant for 25 and 100 queries. This is likely due to the fact that 25 queries are not sufficient to train an initial ensemble estimate the disagrcement and 100 queries by itsclf is sufficient train a good reward model without additional active learning. 50 is likely strikes good balance where we have enough train good initial ensemble of models while not diminishing the effect of using active queries.
17

---

Published in Transactions on Machine Learning Research (01/2023)
Table sensitivity analysis is run on the calf-cheetal--aandoii environment. The default uses 50 initial queries and includes 10 additional queries per round. In this table, we vary the number of queries per round while keeping the initial number of queries fixed. The results are reported with seeds and aggregated using the interquartile mcan.
NUMBER OF QUERIES PER ROUND
5
10
20
RANDOM ENSEMDIS
19.9
34.2
26.9
31.1
41.6
32.5
We also vary the number of queries per round and found that ensemble disagreement outperforms random in all settings. The performance for ensemble disagreement is also fairly consistent across the different numbers of queries per round. We note that the more number of queries per round does not necessarily improve performance. However, this is likely due the fact that other typer-parametedt like the number of iterations are tuned for 10 queries per round.
C Sensitivity Analysis on Number of Ensemble Models and Number of Dropout Samples
Table 7: A sensitivity analysis is run on the hopper-medium--2 enviromment. The default uses 7 ensemble models. In this table, we vary the number of ensemble models for both ensemble disagreement and ensemble information gain. Results are aggregated with 3 seeds using interquartile mean.
NUMBER ENSEMBLE MODELS
14
ENSEMDIS ENSEMINFO
1024.5
1116.1
1099.8
1088.8
1092.2
1128.6
For ensemble disagroement, we notice an improvement in performance from increasing the number ensemble models from 3 to and no improvement and a slight dip in performance from 14. For ensemble information gain, we notice that the performance improves slightly with more ensemble models. However, ensemble information gain seems be less sensitive to the number of ensemble models compared to ensemble disagroement. This can also mcan that ensemble information gain be effective with fewer ensemble models. Although 14 ensemble models perform slightly better than the default in the case of ensemble disagreement, it does take significantly longer to train due to the additional number of ensemble models to train.
Table 8: A sensitivity analysis is run on the hopper-medium--2 environment. The default uses 30 dropout samples. In this table, we vary the number of dropout samples for both dropout disagreement and dropout information gain. Results are aggregated with seeds using interquartile mean.
NUMBER OF DROPOUT SAMPLES
15
30
60
DROPDIS DROPINFO
1034.3
1054.3
1062.3
1064.3
1116.6
1087.0
For dropout disagreement, we notice that with more dropout samples, the performance improves. Dropout info gain on the other hand works best with 30 dropout samples and worse with 15 dropout samples. In both variants, increasing the number dropout samples beyond 30 does not improve the performance significantly and even slightly degrades performance in the of dropout information gain.
18

---

Published in Transactions on Machine Learning Research (01/2023)
D Information Gain
As alternative to disagreement, also consider the expected information gain Cover (1999 between the reward function parameters 9 and the outcome Y of querying the human for a preference. We model our uncertainty using an approximate posterior p(0 'D), given by training an ensemble or dropout network on our offline dataset D. Houlsby et al. (2011) show that the information gain of a potential query can formulated as:
T(0; Y D) H( D) (H D)]-
Intuitively, the information gain will be maximized when the first term is high, meaning that the overall model has high entropy, but. the second term is low, meaning that each individual sample from the posterior has low entropy. This will happen when the individual hypotheses strongly disagree with each other and there is clear majority. We approximate both terms in the information gain equation with samples obtained via ensemble or dropout.
Given a pair of trajectorics (TA.TB), let Y = O denote that the human prefers trajectory A and Y denote that the human prefers trajectory B. Using the Bradley-Tere preference model Bradley & Terry 1952 we have that
exp(BR(EA))
P(Y = =0 = exp(3 +
(4)
where R(E) = ((s.)er. Po(s) and P(Y 1
= - P(Y = 0.
To perform queries using Information Gain, we evaluate a pool of potential trajectory pairs, evaluate the information gain for each pair and query the human for the pair that has the highest information gain.
M
p 0, D)]
~
Σ M
1
H(Y (0,
(5)
where 0,, i i...., M are M approximate posterior samples from p(0 D) obtained from each member of the ensemble or from Bayesian dropout sampling.
Because we only consider pairwise preference queries, computing the entropy corresponds to
H(Y D)
--PEPS P(Y 0,, D) log P(Y 0g, D)
(6)
compute the first entropy lerm, we simply take the entropy of the average over the posterior as:
1
H(YID)
Σ
Dy log Pv
(7)
==0
where b, 1 EM P(Y = 0,,
M
E Evaluating Offline RL Benchmarks
We find that using average and zero masking are very competitive with using the true rewards many of the benchmarks for D4RL and that these basclines often perform significantly better than purely random policy.
This a byproduct of offline reinforcement learning algorithms needing to learn in the presence of out- of-distribution actions. There are broadly two classes of methods towards solving the out-of-distributioo problem. Behavioral cloning based methods like Advantage Weighted Regression used in our experiments, train only actions observed in the dataset, which avoid OOD actions completely. Dynamic programming
19

---

Published in Transactions on Machine Learning Research (01/2023)
(DP) methods, like BCQ, constrains the trained policy distribution to lie close to the behavior policy that generated the dataset. As result of offline reinforcement learning algorithms constraining action to be similar to that of the static dataset, if the static dataset consists of exclusively expert actions, then the offline reinforcement learning algorithm will recover a policy that has expert like action regardless of the reward.
E.1
Serantagg-Weighter Regression
Consider Advantage- advantage-Weighted Regression with constant reward function r(s.a) = The algorithm is simple. We have a replay buffer from which we sample transitions and estimate the value function via regression
V* arg min E Be [I]R? [()]]]
(8)
then the policy is updated via supervised learning:
[og (ogg(als)eng exp -
π arg max E. anD
(9)
If the rewards are all zero, then we have V(s) O, Vs E S and
1 π arg max Bs log exp exp
(10)
max E DD [als)]
(11)
This is exactly the behavioral cloning loss which will learn to take the actions in the replay buffer. Thus, AWR is identical to BC when the rewards are all zero.
For non-zero, constant rewards, the advantage term will be zero (at least for deterministic tasks). If all trajectories from the state s have the same length, then we will have zero advantage. However, there are trajectorics that terminate carlier than others, then AWR will put weights on these trajectorics proportional to their length (for positive rewards > 0) and inversely proportional to their length (for negative rewards < 0). Thus, terminal state will leak information and good policy can be learned without informative reward function.
F Comparison of AWR and CQL for OPRL
In Table 9 we show results for OPRL when using AWR Peng et al. (2019) for policy optimization and when using CQL Kumar et al. (2020) for policy optimization. Interestingly, our results suggest that when using CQL, using dropout to represent uncertainty performs better than using an ensemble. However, when using AWR for policy optimization, ensembles perform better. We hypothesize that different query mechanisms can have complex interactions with the actual policy learning algorithm. Better understanding how active queries and offline reward learning affects the performance of different offline RL algorithms is an interesting area of future work.
G Franka Kitchen Learning Curves
The FrankaKitchen domain involves interacting with various objects in the kitchen reach certain state configuration. The objects you interact with include water kettle, light switch, microwave, and cabinets. Our results in Table ! show that setting the reward to all zero performance to degrade significantly. However, the results in Table 2 show that OPRL using Ensemble Disagreement and Ensemble InfoGain are able to shape the reward in a way that leads to better performance than using the ground-truth reward for the task. In Figure S) we plot the learning curves over time and find that OPRL also converges showing that the shaping reward learned from pairwise preferences can boost the performance of RL, even when the agent has access to the ground truth reward function Memarian et al. (0211.
constant reward causes the
better performance faster. While surprising, these results support recent work
20

---

Published in Transactions on Machine Learning Research (01/2023)
Table 9: OPRL Performance Using Reward Predicted After N Rounds of Querying: Offline RL performance using rewards predicted with T-REX using a static number of randomly selected pairwise preferences (T-REX), Ensemble Disagreement DiseemDss, Ensemble Information Gain (Enseminfo), Dropout Disagreement (DropDis), Dropout Information Gain (DropInfo) and Ground Truth Reward (GT). N is set to 5 for Maze2D-Umaze, 10 for Maze2d-Medium and flow-merge-random, 15 for Hopper and Halfcheetah. N is selected based on the size of the dimension of observation space and the complexity of the environment. Results are averages over three random seeds. Results are normalized such that the ground truth performance is 100.0 and the random policy performance 0.0. We report the performance of OPRL using AWR Peng (2020) for policy optimization.
et al. (2019) and CQL Kumar
AWR PENG ET
2019
QUERY ACQUISITION METHOD ENSEMINPO
T-REX
ENSEMDIS
DROPDIS
DROPINFO
MAZEZD-UMAZE
78.2
93.5
89.2
88.0
61.3
MAZE2D-MEDIUM
71.6
86.4
71.0
52.6
44.4
НOPPER
69.0
77.5
72.8
87.6
90.6
НALFCHEBTAH
113.7
100.6
84.4
91.7
MEROGEEGGE-ITO RANDOM
110.1
89.0
92.1
86.2
84.2
KITCHEN-COMPLETE
79.6
105.0
158.8
48.5
65.4
CQL KUMAR ET AL.
2020
QUERY ACQUISITION METHOD ENSEMINFO
T-REX
ENSEMDIS
DROPDIS DROPINFO
MAZE2D-UMAZE
87.0
54.8
100.4
102.3
95.2
MAZE2D-MEDINE
41.1
33.1
115.4
1.6
34.5
HOPPER
32.1
28.7
17.5
35.9
42.0
HALFCHEETAH
68.4
75.9
76.3
75.0
79.6
FLOW- MERGE- MERGE-RANDOM
-13.6
44.3
103.2
69.6
114.0
WITCHEN-COMPLETT
85.7
100.0
71.4
114.3
71.4
0.12
0.08
0.06
round 5
round 10
0.02
GT Reward
200
300
400
Iteratio
Figure 7: Kitchen LrrchenCoomplet-vll
Figure 8: Dropout disagroement after 15 rounds has similar performance ground truth reward in Kitchen- complete
H Pairwise Preference Accuracy
Pairwise preference accuracy on held-out set of trajectory pairs for each approach provided in Table II Notably, when using randomly chosen queries (T-REX) the accuracy barely improves after round in terms of ranking accuracy whereas all of our approaches continue to improve after round 5.
21

---

Published in Transactions on Machine Learning Research (01/2023)
Table 10: Quantitative Results for maze2t-meiimm-dens--- Using Human Labels: To evaluate quantitative performance with human data, we collected 100 query labels from different users in the maze2d-medium- meeeodd--mmiudddeee----e dense--e environment. Due the limitation that an ensemble of neural networks cannot be updated fast enough in between queries, elected to create pool of 100 queries per below, used 15 randomly selected preferences for Random and 15 active queries for all versions of OPRL. We notice that random queries slightly outperforms ensemble disagreement version of OPRL when trained on data from individual users but performs slightly worse in the pooled data which contained 600 queries. We attribute this to the observation that OPRL works better with larger datasets because the ability to pick informative queries becomes more important as the dataset gets larger. This also explains why ensemble disagreement version of OPRL consistently outperforms random queries on the original full dataset where the number of queries to choose from is order of magnitudes larger. This experiment demonstrates that humans are able to elfectively answer queries that can be used to recover policies on par with GT performance.
For the results
QUERY ACQUISITION METHOD ENSEMINFO
RANDOM
ENSEMDIS
DROPDIS
DROPINFO
GT
POOLED
122.5
122.6
103.6
106.2
106.9
134.6
USER
118.60
114.98
119.72
106.97
116.85
134.6
USER 2
119.81
114.40
114.48
93.68
107.32
134.6
USER
118.68
114.39
114.51
112.97
107.87
134.6
USER
123.21
121.97
120.54
112.07
115.07
134.6
USER
116.29
118.87
109.30
96.84
101.09
134.6
USER
121.88
121.89
122.48
112.05
109.23
134.6
USER AvG
119.75
117.25
116.84
105.76
109.57
134.6
Quantitative Results with Human Labels
J Real human feedback experiments
(a) User
(b) User
(c) User
(d) User 4
(c) User
(f) User
Figure 9: Counterclock wise orbit generated from human provided preferences
K
Reward Model, Hyperparameter, And Dataset Details
During active reward learning, lotal of models used in the ensemble variant. For the reward models, used standard MLP architecture using hidden sizes [512, 256, 128, 64, 32] with ReLU activations. For the dropout variant, we use 30 pass through of the model to calculate the posterior. Both the number of ensemble members and the number of pass-throughs were tuned to balance between speed and giving an accurate disagreement or information gain.
For policy learning with AWR, lower dimensional environments including Mazo2D- Umaze, Mazc2D-Modium, and Hopper are ran with 400 iterations. Higher dimensional environments including Halfcheetah, Flow- Mergo- Random, and Kitchen-Comples
with 1000 iterations. The number of iterations tuned that the
22

---

Published in Transactions on Machine Learning Research (01/2023)
policy converges before the maximum number of iterations. For CQL, policy learning rate is 1c-4, lagrange threshold is -1.0, min q weights is 5.0, min q version is 3, and policy eval start is O. These numbers are suggested by the author of the CQL paper.
All results are collected and averaged across 3 random seeds. All models are trained on an Azure Standard NC24 Promo instance, with 24 vCPUs, 224 GiB of RAM and K80 GPU (2 Physical Cards). Datasets used generate the customized behaviors (e.g. Constrained Goal Navigation, Orbit Policy, and Cartpole Windmill) will be released upon conference acceptance.
Table 11:
Pairwise preference accuracy on held out set trajectory pairs after rounds and 10 rounds of queries respectivcly for different types of query acquisition functions.
RANDOM
ENSEMDIS
ENSEMINFO
DROPDIS
DROPINFO
MAZE2D-UMAN
0.818, 0.867
0.817, 0.916
0.77, 0.862
0.874, 0.893
0.715, 0.755
MAZEE2-MEDIUM
0.646, 0.686
0.650, 0.783
0.638, 0.824
0.636, 0.795
0.692, 0.756
HOPPER
0.929, 0.922
0.943, 0.966
0.952, 0.966
0.946, 0.966
0.927, 0.960
HALFCHEBTAH
0.814, 0.873
0.852, 0.942
0.866, 0.944
0.875, 0.927
0.833, 0.913
THOW-MERGE-RANDRE
0.808, 0.841
0.829, 0.881
0.825, 0.861
0.846, 0.884
0.839, 0.893
KITCHEN
0.964, 0.978
0.981, 0.986
0.967, 0.984
0.973, 0.990
0.953, 0.986
L Table 1 Extended
Table 12: Offline RL. performance on D4RL benchmark tasks comparing the performance with the true
reward to performance using constant reward equal to the average reward over the dataset (Avg) and zero rewards everywhere (Zero). Even when all the rewards are set to the average or to zero, many offine RL algorithms still perform surprisingly well. In this table, we present the experiments ran with BCQ. The degradation percentage is calculated as (AA.GTTTmAAVVGGEEOOOO.
x 100%.
[GT]
TASK
GT
AvG
ZERO
RANDOM
DEGRADATION
PLON-BUNG-RAANDOM-VV
14.3
-41.8
-67.5
-166.2
392.3
KLON-MERGEE-LANDOO--Y
130.3
121.4
117
61.0
MAZE:2DUUAZEE
104.2
36.5
41.8
49.5
52.5
MAZE2D-MEDIUM
106.6
23.3
28.5
44.8
58.0
HALFCHEETAH RANDOOM
-0.6
-1.5
-1.9
-285.8
133.9
MEDIIEM REPLAY
3400.0
2659.3
3185.0
285.8
6.3
HALFCHEETAH MEDIUM
3057.7
2219.1
2651.7
285.8
13.3
CALICHEEETTAI--MEIDU--EEEE
10905.8
11146.9
11030.3
-25.88
0.0
IALFCHEEETAI--EXEETT
63.6
115.8
-32.4
225.88
0.0
HOPPER-HANDOM
199.8
186.0
199.4
18.3
0.2
HOPPER-MEDIUM-HHIEE
1188.1
1082.4
908.3
18.3
8.9
HOPPER-MEDIUM
922.3
745.4
842.2
18.3
8.7
HOPPER-MEDDUMMEEPERR
343.3
225.7
202.0
18.3
34.3
HOPPER-EXPERT
241.5
268.1
343.3
18.3
0.0
CITCHEN-COMPLETE
0.07
0.11
0.04
0.0
0.0
KITCHEN-MIXED
0.08
0.07
0.02
0.0
12.5
KITCHEN-PARTIE
0.16
0.17
0.13
0.0
0.0
23

---

Published in Transactions on Machine Learning Research (01/2023)
Offline RL performance on D4RL benchmark tasks comparing the performance with the true reward to performance using constant reward equal to the average reward over the dataset (Avg) and zero rewards everywhere (Zero). Even when all the rewards are set to the average to zero, many offline RL algorithms still perform surprisingly well. In this table, we present the experiments ran with BEAR. The degradation percentage is calculated
Table 13:
RANDOM),) X 100%
GTT]
TASK
GT
AvG
ZERO
RANDOM
DEGRADATION %
FION-BING-RANNDOM-VI RA
13.4
18.2
11.6
-166.2
0.0
75.9
62.1
68.3
117.0
0.0
MAZE2D-UMAZE
66.6
44.8
59.7
49.5
10.3
MAZE2D-MEDIUM
18.6
29.8
22.5
44.8
0.0
TALIFCHEEETTAI-RAADDOO
0.8
-0.9
-0.5
285.8
160.5
IALECHIBEEALIIMEDDAARRRRRYY
4997.5
3919.6
4282.7
-28..8
14.3
HALFCHEBBTAH-MEEDDIN
5260.9
4910.3
4941.4
285.8
6.1
TALFCCHERRTAF-MEEDDU---
5346.9
5033.4
4709.1
-25.88
5.9
HALFOHEETAH-EXPSHE
11329.1
11268.9
10627.6
285.8
0.5
HOPPER-RANDOM
221.9
215.6
86.3
18.3
2.9
ROPPER-MEDIUM-EET
1261.9
659.0
1311.2
18.3
0.0
COPPER-MEDIUX
1564.3
1749.3
1601.5
18.3
0.0
HOPPER-MEDIUX MEEDEE-DDEMMMMMM MEXPUR-
2269.7
2995.2
1387.9
18.3
0.0
2110.6
2925.9
2826.3
18.3
0.0
KITCHEN-COMPLETT
0.8
0.7
0.1
0.0
12.0
KITCHEN-MIXED KITCHEN-PARTIAL
0.7
0.6
0.2
0.0
14.3
0.2
0.0
Offline RL performance on D4RL benchmark tasks comparing the performance with the true reward to performance using constant reward equal to the average reward over the dataset (Avg) and zero rewards everywhere (Zero). Even when all the rewards are set to the average or to zero, many offline RL algorithms still perform surprisingly well. In this table, we present the experiments ran with CQL. The degradation percentage is calculated
Table 14:
BAATTTT-A(AGGG/B...... GT
x 100%.
TASK
GT
AvG
ZERO
RANDOM
DEGRADATION %
FLOW-RING-RANDOM--V
13.4
-49.5
-23.3
-166.2
274.1
FLOW-MERGE-RANDON
156.1
98.1
50.5
117.0
25.0
MAZZ2D-UMAZE:
83.4
46.0
38.2
49.5
40.7
MAZE2D-MEDIUM
107.9
34.3
19.6
44.8
58.5
HALFFCHERSTAHH-IAANOO
3140.7
-281.5
-15.55
-285.8
105.0
HALFFCHESSTA--MBEIICAAAA
5602.9
1145.4
-512.2
-285.8
79.6
HALFCCHERTTMH-MEEDDIIY
5546.9
5130.2
5037.2
-285.8
7.5
HALLFCHHEETTAA--MEEDEEMM
3544-6
4372.7
5505.3
-285.8
0.0
TALFCHEBTAH-EXPEE
258.2
45.5
3546.6
-28.88
0.0
HOOPERR-RNDOM
968.9
512.0
11.0
18.3
47.2
HOPPER-MEDIUM-ROO REPLLAY
2484.2
1970.3
563.7
18.3
20.7
HOPPER-MEDIUM
1709.9
2205.4
2098.7
18.3
0.0
HOPPER-MEDIUM-EENT
1156
1136.9
1210.9
18.3
0.0
HOPPER-EXPERT
919.1
904.1
2092.7
18.3
0.0
KITCHRN-COMPLETE
0.8
0.7
0.6
0.0
14.3
KITCHEN-MING MIXED
0.6
0.4
0.8
0.0
0.0
KITCHEN PARTIAL PURCHEN-PARTIAL
0.3
0.0
0.0
24


Reviews:
[{"review_id": "G8Dr9wIRit", "reviewer": "Reviewer_yMyt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall I think this paper is a nice contribution, and suggests an interesting problem which, as far as I know, hasn't been introduced before. I have a few (fixable) concerns which will make the paper stronger.\n\n## Strengths\n1. I like the idea of preference-based learning from offline datasets, and I think it shows promise for future research.\n1. I think the authors did a good job in evaluating some of the D4RL environments to determine which of these are well-suited for preference-based learning.\n1. The derivation in Appendix B is quite nice!\n\n## Weaknesses\n1. One aspect that recurs throughout the paper is the notion of estimating/representing uncertainty. While I can imagine reasons why this is important, I feel the paper could do a better job at justifying _why_ it is important to estimate/represent uncertainty. In its current form, it seems to take this for granted.\n1. Another recurring issue is that a number of design decisions for the proposed algorithm (number of queries, number of ensemble models, etc.) are specified but not well-justified. It would be good to provide sensitivity analyses for these, as it will help future researchers/users be better equipped for dealing with these types of problems methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a method for leveraging offline RL data for preference-based learning. That is, the offline trajectories can be used to query user preferences and learn a reward function for them. Their proposed method incorporates a measure of reward uncertainty, query selection, and policy optimization. The main contributions of the paper are in establishing offline preference-based learning as an interesting problem worty of study, evaluating a few natural baselines, and determining whether existing offline RL datasets prove challening enough for this type of problem.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "## Major changes / questions\n1. In the abstract, in the sentences \"To test our approach, we first ... which allow for more open-ended behaviors.\", it's not clear whether you're talking about preference learning or reward learning in general. I believe you're referring to preference-based, but worth making a bit more explicit.\n1. In the second-to-last paragraph in the second page, when discussing benchmarks that are \"ill-suited for reward learning\", could one consider this more indicative of the success of existing offline RL algorithms, as opposed to failures in the benchmarks?\n1. The caption of Figure 2 is repeated 3 times.\n1. The lines right before Section 4.1 saying \"We are the first to evaluate Dropout Disagreement...\" seems like a weird and unnecessary phrase to include, given that the two components listed are what make up OPAL; so the statement is akin to saying: \"We are the first to evaluate the method we are introducing\"... I'd suggest removing it.\n1. In FIgure 4 it is not clear what \"rounds\" represents.\n1. In Figure 4, how many seeds were run? What do the shaded areas represent?\n1. In section 5.2 the authors specify how many preference queries (initial + active) were used for the experiments in each domain. How were these selected? How sensitive is your method to this choice? It would be nice to provide some type of sensitivity analysis here, as this is an important decision. What is also lacking is some kind of insight/suggestion for future researchers. If someone uses a different offline dataset with a (possibly) different method, how would they know how many queries are sufficient?\n1. In Table 2, since all results are normalized, you should report more robust statistics as suggested in [2](https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html) (namely, Interquantile Mean)\n1. The trajectories in Figures 5 and 6 are hard to see, I would suggest removing the dark blue background.\n1. In Figure 6, it would be good to overlay multiple trajectories (instead of just one) so we can see the consistency of the method.\n1. In Appendix H it is specified that 7 models are used in the ensemble variant, and 30 pass-throughs used to calculate the posterior. How sensitive is the method to these choices? It would be good to provide a sensitivity analysis.\n\n\n## Minor / cosmetic changes:\n1. In abstract, should read \"using a trivial reward function results **in** good policy performance\"\n1. In page 2 in the second, third, and last paragraphs you refer to \"Figure 1\" but it should be \"Figure 2\"\n1. Page 2, second paragraph: \"components of our framework **are** not novel by **themselves**.\"\n1. Another paper that may be useful to discuss in related work is [1](https://arxiv.org/abs/1907.13411).\n1. In the last line of page 4: \"of demonstrations for a vari**e**ty of tasks.\"\n1. In the second-to-last sentence before section 5.1 it would read better as \"**Finally, we** propose and...\"\n1. In Section 5.2.2 it says that after 10 rounds of querying results are \"on par with the policy learned with the ground truth reward\", but it's not quite so, it seems a fair bit lower, according to Figure 4.\n1. In the third line of section 5.3.2 it should read \"The open maze **counter-clockwise**...\"\n1. In the line above equation (4) in the appendix, it should say \"**Using** the Bradley-Terry preference...\"\n\n[1] [Castro, Li, & Zhang, 2019; Inverse Reinforcement Learning with Multiple Ranked Experts.](https://arxiv.org/abs/1907.13411)\n\n[2] [Agarwal, Schwarzer, Castro, Courville, & Bellemare, 2021; Deep reinforcement learning at the edge of the statistical precipice.](https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html)", "broader_impact_concerns": "No concerns from me on this front.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qJouSp9MVj", "reviewer": "Reviewer_2gDE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work is reasonably careful on comparing different methods for surfacing queries to request labels for, eventually finding that when AWR is used, an ensemble disagreement approach works best. The authors also identify offline datasets where a reward-agnostic approach (all zero or all constant reward) performs well - these datasets are primarily datasets generated by expert policies, and are ones where a learned reward function is not too helpful for the underlying RL algorithm because offline RL algorithms tend to act BC-like when they do not have reward information.\n\nThe technical work seems solid and it refers to appropriate prior work for learning from human preferences - as noted in the paper, most of the approaches are known but this should be of interest for others, especially on which benchmarks are suitable for benchmarking learned reward functions.\n\nThe primary changes needed are general cleanup, see next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper describes an approach for learning a reward function based on human preferences. This is an approach with a long history in the literature, and this paper focuses specifically on learning a reward function assuming access to an offline dataset of interaction, and sampling episode segments from the offline dataset, rather than doing online interaction.\n\nIn the process of doing so, they consider the following axes:\n\n* How should paired trajectories be sampled? Different active learning techniques used.\n* How should uncertainty be estimated for the active learning techniques? Compared ensembles to Bayesian dropout\n* What environments are good ones to evaluate learned reward functions? Compared no-reward baselines to check which environments within D4RL seem more benefit from a good learned reward.\n\nThe work is primarily done using the ground truth reward to answer queries, with the last section focusing on more qualitative tasks with human feedback.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The caption for Figure 2 appears to be repeated 3 times in the submission on page 3, please clean this up. (critical)\n\n\"identify a subset environments\" -> subset of environments (nitpick)\n\nI do not think it is worth claiming that this is the first paper to evaluate dropout disagreement and ensembles for an info gain acquistion function. Although I am not sure of a specific learning from human preferences paper that does so, approaches like this are quite common for active learning in general, and if restricted to human preference learning, the claim is specific enough to not really be interesting to make. (decently important)\n\nIt would be good if the authors could add more algorithmic details on how they efficiently identify the best pair of trajectories to compare. A naive approach for IDing the best pair would be O(N^2) in dataset size, checking every pair, and I assume the authors did not do this. (decently important)\n\nThe paths in Figure 5 in black are hard to see against the blue background. The red color from Figure 5 would be better here. (nitpick)", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uI4fGr6M3m", "reviewer": "Reviewer_Ndnj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The method investigates how preference-based learning can work in offline RL settings where rewards are not present, which is an important and practical problem for the community.\n\n2. The empirical results of the paper clearly show that the method can perform relatively well in many scenarios compared to the approach with ground-truth rewards, which is surprising yet interesting.\n\n3. The paper is well-written and easy to follow.\n\n**Weaknesses**:\n\n1. I think the method of the approach is a bit incremental. It seems a bit derivative from the T-REX paper. Though the authors added the active query part with various uncertainty estimation techniques, the improvement over T-REX is a bit marginal and highly task/algorithm-dependent. I also wonder if the performance of T-REX could drastically improve with a large number of randomly selected pairwise preferences.\n\n2. It is a bit strange that the results are normalized w.r.t. the policy learned by offline RL with ground-truth rewards. The authors should use the expert score to make it clear how these approaches are performing. Moreover, The results of offline RL with ground-truth rewards are significantly worse than the numbers shown in the original papers, especially CQL. I think the authors should tune the backbone policy optimization algorithms better and make sure the results can match the results in the original papers. Otherwise, the results shown here could be misleading.\n\n3. I think the authors should also compare the method to other reward learning/imitation learning methods such as DemoDICE, IQ-Learn etc.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes to leverage human preference and active queries in offline RL without ground-truth reward labels. The work is built upon the previous work T-REX where the reward function is learned with pairwise preference provided by humans but instead of using randomly selected pairs and online rollouts in T-REX, the method picks pairs of trajectories using active queries with uncertainty estimation on static datasets. The authors show that the approach with various uncertainty estimation techniques for active queries can perform well in many offline RL benchmarks and sometimes matches or even outperforms the results of offline RL with true rewards. In the end, the authors also provided additional benchmarks where the proposed method can learn behaviors that are not shown in the offline dataset.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "1. Clarify the importance of using active queries and compare them to T-REX with more randomly sampled pairwise preferences per round.\n\n2. Normalize the results w.r.t. expert scores and tune the backbone offline RL algorithms to make sure the results can match the original paper numbers.\n\n3. Compare with other reward learning/imitation learning methods such as DemoDICE, IQ-Learn and etc.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is well-written and clear. The results are convincing and the experiments are well-conducted. The subject and the topic this paper is studying is interesting and important. The authors were happy about the paper and leaning accept due to some revisions that they requested. My comments are based on an earlier version of the paper, but it seems like the authors have already addressed some of the comments raised by the reviewers. As a result, I recommend this paper for acceptance with minor revisions. However, there are some small minor revisions that needs to be done as pointed out by the reviewers (if they are not already addressed). Some important ones are: 
1. Sensitivity analysis on the hyper-parameters chosen to justify as noted by the reviewer yMyt.
2. The phrase *"We are the first to evaluate Dropout Disagreement..."* is unnecessary. Both reviewers yMyt and 2gDE  suggested to remove it.
3. Efficiency analysis of identifying the best pair of trajectories to compare as pointed out by reviewer 2gDE.
4. Comparisons to T-REX and other reward learning baselines as suggested by reviewer Ndnj.
5. Some small typos, and cosmetic changes as pointed out by the reviewers.
Decision: Accept (poster)