Reward Uncertainty for Exploration in Preference-based Reinforcement Learning
Authors: Xinran Liang
Katherine Shu
Kimin Lee
Pieter Abbeel
Venue: ICLR 2022 Poster (2022)

Abstract: Conveying complex objectives to reinforcement learning (RL) agents often requires meticulous reward engineering. Preference-based RL methods are able to learn a more flexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher's preferences between two clips of behaviors. However, poor feedback-efficiency still remains as a problem in current preference-based RL algorithms, as tailored human feedback is very expensive. To handle this issue, previous methods have mainly focused on improving query selection and policy initialization. At the same time, recent exploration methods have proven to be a recipe for improving sample-efficiency in RL. We present an exploration method specifically for preference-based RL algorithms. Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward. Specifically, we utilize disagreement across ensemble of learned reward models. Our intuition is that disagreement in learned reward model reflects uncertainty in tailored human feedback and could be useful for exploration. Our experiments show that reward uncertainty exploration improves both feedback- and sample-efficiency of preference-based RL algorithms on complex robot manipulation tasks from Meta-World benchmarks, compared with other existing exploration methods that measure the novelty of state visitation.

Published as a conference paper at ICLR 2022
REWARD UNCERTAINTY FOR EXPLORATION IN PREFERENCE-BASED REINEORCEMENT LEARNING
Xinran Liang', Katherine Shu', Kimin Lee Pieter Abbeel!*
University of California, Berkeley
ABSTRACT
Conveying complex objectives to reinforcement learning (RL) agents often re- quires meticulous reward engineering. Preference- brrerence-bbsed RL methods are able learn a more flexible reward model based on human preferences by actively incorporating human feedback, i.e. teacher's preferences between two clips of behaviors. However, poor feedback-efficiency still remains a problem current preference-based RL algorithms, tailored human feedback is very expensive. To handle this issue, previous methods have mainly focused on improving query selection and policy initialization. At the same time, recent exploration methods have proven to be recipe for improving sample-efficinny in RL. We present an exploration method specifically for preference- RL algorithms. Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward. Specifically, we utilize disagreement across ensemble of learned reward models. Our intuition that disagreement in learned reward model reflects uncer- tainty in tailored human feedback and could be useful for exploration. Our exper- iments show that exploration bonus from uncertainty in learned reward improves both feedback- and sample-effciincy of preference based RI algorithms on com- plex robot manipulation tasks from World benchmarks, compared with other existing exploration methods that measure the novelty of state visitation.
INTRODUCTION
In reinforcement learning (RL), reward function specifies correct objectives RL agents. How- ever, it is difficult and time-consuming to carefully design suitable reward functions for a variety of complex behaviors (e.g., cooking or book summarization (Wu al., 2021)). Furthermore, if there are complicated social norms we want RL agents to understand and follow, conveying a reliable re- ward function include such information may remain be an open problem (Amodei al., 2016; Hadfield-Mende et al., 2017). Overall, engineering reward functions purely by human efforts for all tasks remains to be significant challenge.
An alternative to resolve the challenge of reward engineering is preference-basee RL (Christiano al., 2017: Ibarz et al., 2018; Lee et al., 2021b). Compared to traditional RI. setup, preference- based RI. algorithmss are able to teach agents without the necessity of designing reward functions. Instead, the agent uses feedback, usually in the form of (human) teacher preferences between two behaviors, to learn desired behaviors indicated by teacher. Therefore, instead of using carefully- designed rewards from the environment, the agent is able to learn a more flexible reward function suitably aligned to teacher feedback.
However, preference-basee based RL usually requires a large amount of teacher feedback, which may be timely sometimes infeasible to collect. To improve feedback-efficiency prior works have investigated several sampling strategies (Biyik & Sadigh, 2018; Sadigh et al., 2017; Biyik et al., 2020; Lee et al., 2021c). These methods aim select more informative queries to improve the quality of the learned reward function while asking for fewer feedback from teacher. Another line of works focus on policy initialization. Ibarz et al. (2018) initialized the agent's policy with imitation learning from the expert demonstrations, and Lee et al. (2021b) utilized unsupervised pre-training of RL agents before collecting for teacher preferences in the hope of learning diverse behaviors in self-supervised way to reduce total amount of human feedback.
Equal advising.

---

Published as a conference paper at ICLR 2022
Reward learning from preferences
Extrinsic reward (St, At) = mean (8c, 8t)
Intrinsic reward
paat (St, a2) Feed ax)
replay buffer
(a(s)))
poxe (St, ax) + Be r (s,, ar)
Figure 1: Illustrationn of RUNE. The agent interacts with the environment and learns an ensemble of reward functions based on teacher preferences. For each state-action pair, the total reward is combination of the extrinsic reward, the mean the ensemble's predicted values, and the intrinsic reward, the standard deviation between the ensemble's predicted values.
Exploration, the context of standard RL, has addressed the problems of samplc-efficiency (Stadie et al., 2015; Bellemare et al., 2016; Pathak et al., 2017; 2019; Liu & Abbeel, 2021; Seo et al., 2021b). When extrinsic rewards from the environment is limited, exploration has been demonstrated to allow RL agents to learn diverse behaviors. However, limited previous works have studied the effects of exploration in preference-based RL.
Inspired by recent success of exploration methods, present RUNE: Reward UNcertainty for Exploration, simple and efficient exploration method specifically for preference-based RI algo- rithms. Our main idea is incorporate uncertainty from learned reward function an exploration bonus. Specifically, we capture the novelty of human feedback by measuring the reward uncertainty (e.g., variance predictions of ensemble of reward functions). Since reward functions optimized and learned to align to human feedback, exploration based on reward uncertainty may also reflect high uncertainty information from teacher feedback. We hope that the proposed intrinsic reward contains information from teacher feedback and can guide exploration that better align to human preferences. Our experiment results show that RUNE can improve both sample- and feedback- efficiency of preference- based RL algorithms (Lee al., 2021b).
We highlight the main contributions of our paper below:
We propose a new exploration method based on uncertainty in learned reward functions for preference-based RL algorithms.
For the first time, we show that exploration can improve the sample- and feedback- efficiency of preference-based RL algorithms.
RELATED WORK
Human-in-the-loww reinforcement learning. We mainly focus on one promising direction that uti- lizes the human preferences (Akrour et al., 2011; Christiano al., 2017: Ibarz al., 2018: Lee al., 202 Leike et al., 2018; Pilarski et al., 2011; Wilson et al., 2012) to train RL agents. Christiano al. (2017) scaled preference-based learning to utilize modern deep learning techniques, and Ibarz et al. (2018) improved the efficiency of this method by introducing additional forms of feedback such as demonstrations, Recently, Lee al. (2021b) proposed seedback-efficieee RL algorithm by utilizing off-policy learning and pre-training.
To improve sample- and shedbback-fffciinge of moman-in-thheloop RL,, previous works (Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021b; Leike et al., 2018) mainly focus on methods such as selecting more informative queries Christiano et al. (2017) and pre-training of RL agents Ibarz et al. (2018); Lee et al. (2021b). We further investigate effects of different exploration methods in preference-batt based RL algorithm. We follow a common approach of exploration methods in RL: generating intrinsic rewards exploration bonus Pathak et al. (2019). Instead of only using learned reward function from human feedback as RL training objective, we alter the reward function to include combination of the extrinsic reward (the learned rewards) and intrinsic reward (explo-
2

---

Published as a conference paper at ICLR 2022
ration bonus). In particular, present an exploration method with intrinsic reward that measures the disagreement from learned reward models.
Exploration in reinforcement learning. The trade off between exploitation and exploration is critical topic in RL. If agents don't 't explore enough, then they may Iearn sub optimal actions. Explo- ration algorithms aim encourage the RL agent visit wide range of states in the environment. Thrun (1992) showed that exploration methods that utilize the agent's 's history has been shown to perform much better than random exploration. Hence, a common setup is to include an intrinsic re- ward as an exploration bonus. The intrinsic reward can be defined by Count-Based methods which keep count of previously visited states and rewards the agents for visiting new states Bellemare al. (2016); Tang et al. (2017); Ostrovski et al. (2017).
Another option is use a curiosity bonus for the intrinsic reward Houthooft et al. (2016); Pathak et al. (2017); Sekar et al. (2020). Curiosity represents how expected and unfamiliar the state is. One way to quantify curiosity is to predict the next state from current state and action Pathak al. (2017), then use prediction error an estimate of curiosity. If the error is high, that means the next state is unfamiliar and should explored more. Similarly, instead of predicting the next state, prediction errors from training a neural network to approximate a random function Burda et al. (2018) can serve as a valid estimate of curiosity. If there are multiple models, then curiosity can also be described as the disagreement between the models Pathak et al. (2019). A high disagreement means that the models are unsure about the prediction and need to explore in that direction more.
A different approach maximizes the entropy of visited states by incorporating state entropy into the intrinsic reward. State entropy can be estimated by approximating the state density distribution Hazan et al. (2019); Lee et al. (2019), approximating the k-nearest neighbor entropy of randomly initialized encoder Seo et al. (2021a), or using off-policy RL algorithms to maximize the k-nearest neighbor state entropy estimate contrastive representation space for unsupervised pre-training Srinivas et al. (2020); Liu & Abbeel (2021). These methods encourage agents to explore diverse states.
Our approach adds an intrinsic reward that drives exploration to preference-based RL algorithms. We take advantage of an ensemble of reward models preference- based RL algorithms, which is not available in other traditional RL settings. To estimate novelty of states and actions, we utilize the disagreement between reward models for our intrinsic reward, in hope of encouraging exploration aligned to directions of human preferences.
Trajectory generation in preference-based reinforcement learning. Previous works in preference-based reinforcement learning have investigated several methods to better explore diverse trajectories but close to current optimal policy Wirth et al. (2017).
One line of works computes agent's stochastic policies that are slightly deviated from optimal poli- cies. Christiano et al. (2017) uses Trust Region Policy Optimization (TRPO) Schulman et al. (2015) and synchronized A3C Mnih et al. (2016). These RL algorithms define stochastic policies to sure exploration of action space and deviations from optimal policies. However, these exploration methods based on stochastic RI. algorithms does not include information from human preferences to drive exploration.
Another line of works designs one multiple criterion to select from multiple possible stochastic policy candidates. Wilson et al. (2012) proposes to sample several policies from posterior distribu- tion of policy space after updating human preferences. However, such methods come a the cost of requiring many samples collected beforehand. While these methods similarly aims to reduce un- certainty in human preferences, RUNE uses a different metric estimate such uncertainty through reward functions ensemble. This is different from previous works and is simple, scalable, and easy to implement.
A different approach allows human to guide exploration by directly providing additional trajectories. Zucker et al. (2010) proposes user-guided exploration method that shows samples of trajectories to human. Human can provide additional feedback to guide exploration. While this method receives exact information from human, it requires additional human labels, which are usually expensive and time-consuming to collect. RUNE however tries to extract information from human feedback revealed in learned reward functions, which doesn" 't require additional human input.
3

---

Published as a conference paper at ICLR 2022
3 PRELIMINARIES
Preference-based reinforcement learning. We consider an agent interacting with an environment in discrete time Sutton & Barto (2018). At each timestep 4, the agent receives a state S, from the environment and chooses an action a, based on its policy n
In traditional reinforcement learning, the environment also returns reward r(s,, ax) that evaluates the quality agent's behavior at timestep t. The goal of agent is to maximize the discounted sum of rewards. However, in the preference-based RL framework, we don't have such reward function returned from the environment. Instead, a (human) teacher provides preferences between the agent's behaviors and the agent learns its policy from feedbacks (Christiano et al., 2017; Ibarz et al., 2018; Lee et al., 2021b; Leike et al., 2018).
is sequence of time-indexed observations and actions ((s), a1)..., (sH, an))- Given a pair of segments (a o²)) that describe two behaviors, a teacher indicates which segment is preferred, i.e., y (a a²) or (al o²), that the two segments are equally preferred = (o) )"), or that two segments are incomparable, i.e., discarding the query. The goal of preference-based RL is train an agent to perform behaviors desirable to a human teacher using as few feedback as possible.
Formally, segment
In preference-based RL algorithms, policy F and reward function Py are updated as follows:
Step I (agent learning):
The policy as interacts with environment to collect experiences and update it using existing RL algorithms to maximize the sum of the learned rewards f.p.
Step 2 (reward learning): The reward function F... is optimized based on the feedback received from teacher.
Repeat Step I and Step 2.
To incorporate human preferences into reward learning, we optimize reward function Py as follows. Following the Bradley-Terry model (Bradley & Terry, 1952), we first model preference predictor of pair of segments based reward function ig follows:
exp E, Fo a!)
Po > == - (ie exp E. Fre (8)
where o² > aa denotes event that segment is preferable to segment j. Here, the intuition that segments with more desirable behaviors should have higher predicted reward values from Pip- To align preference predictors of &. with labels received from human preferences, preference-bace based RL algorithms translate updating reward function binary classification problem. Specifically, the reward function Fg parametrized by & is updated to minimize the following cross-entropy loss:
[Beaard
E 10..o.0)00
[t = (a > oog Polo > + = (a > o0) Palo > =09]
(2)
where we are given a set D of segment pairs and corresponding human preferences.
Once reward function Ps has been optimized from human preferences, preference-based RL al- gorithms train RL agents with any existing RI. algorithms, treating predicted rewards from P. as reward function returned from the environment.
RUNE
In this section, present RUNE: Reward UNcertainty for Exploration (Figure 1), which encour- ages human-guides exploration for preference-baste RL. The key idea of RUNE is to incentivize exploration by providing an intrinsic reward based reward uncertainty. Our main hypothesis that the reward uncertainty captures the novelty of human feedback, which can lead to useful behaviors for preference-hasee RI.
4

---

Published as a conference paper at ICLR 2022
(a) Door Close
(b) Door Open
(c) Drawer Open
(d) Sweep Into (e) Door Unlock (I) Window Close
Figure 2:
Examples of rendered images of robotic manipulation tasks from Meta-world. We consider
learning several manipulation skills using preferences from a scripted teacher.
4.1 REWARD UNCERTAINTY FOR EXPLORATION
In preference-beee baseeence--baed RL, capturing the novelty of human feedback can be crucial for efficient re- ward learning. To this end, we propose to utilize an intrinsic reward based on ensemble of reward functions. Specifically, for each timestep, the intrinsic reward is defined as follows:
ant (st, ax) Fu (s: az),
where Fss is the empirical standard deviation of all reward functions (Fe. J. We initialize the model parameters of all reward functions with random parameter values for inducing an initial di- versity. Here, our intuition is high variance of reward functions indicates high uncertainty from human preferences, which means we collected less teacher preferences on those states and actions. Therefore, order to generate more informative queries and improve confidence learned ward functions, we encourage our agent to visit more uncertain state-action pairs with respect to the learned reward functions.
We remark that exploration based ensembles has been studied in the literature (Osband et al., 2016; Chen et al., 2017; Pathak al., 2019; Lee et al., 2021a). For example, Chen et al. (2017) proposed an exploration strategy that considers both best estimates (i.e., mean) and uncertainty (i.e., variance) of Q-functions and Pathak et al. (2019) utilized the disagreement between forward dynamics models. However, method is different in that propose alternative intrinsic reward based on reward ensembles, which can capture the uncertainty from human preferences.
4.2 TRAINING OBJECTIVE BASED ON INTRINSIC REWARDS
Once we learn reward functions (fx, IN from human preferences, agent is usually trained with RL algorithm guided by extrinsic reward:
axt (St, a:) Prean (Se, az);
where Fimem is the empirical mean of all reward functions (Pis, J.N To encourage exploration, train a policy to maximize the sum of both extrinsic reward and intrinsic reward in equation 3;
total
ext (St. a,) + Be (s)
where B, 2 0 a hyperparameter that determines the trade between exploration and exploitation at training time step Similar to Sco et al. (2021b), we use exponential decay schedule for 3 throughout training to encourage the agent to focus more on extrinsic reward from learned reward function predictions training proceeds, i.e., 3r 30 p)² where is decay rate.
While the proposed intrinsic reward would converge to 0 as more feedback queries are collected during training, believe our learned reward functions improve over training collect more feedback queries from teacher preferences. The full procedure of RUNE is summarized Algo- rithm
EXPERIMENTS
We designed our experiments answer the following questions:
Can exploration methods improve the sample- and feedback-efficiency of preference-bate
RL algorithms?
5

---

Published as a conference paper at ICLR 2022
100
100
100
Success Rate (%)
Success Rate (9%.
Success Rate (%)
FEBBLE StateEnt
Disagres
PEBBLE ICM
010
0.4
0.6
0.8
1.0
9.6 0.8 1.0
Environment Steps (x10f)
Environment Steps (x10%)
Environment Steps (x10%)
(a) Door Close (feedback = IK)
(b) Door Open (feedback = 5K) (c) Drawer Open (feedback 10K)
Figure 3: Learning curves on robotic manipulation tasks as measured on the success rate. Ex-
ploration methods consistently improves the sample-efficinncy of PEBBLE. In particular, RUNE provides larger gains than other existing exploration baselines, The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs.
How does RUNE compare other exploration schemes in preference-based RL setting?
How does RUNE influence reward learning in preference-based RL?
5.1 SETUP
In order to verify the efficacy of exploration in cifference-basee RL, we focus having agent solve a range of complex robotic manipulation skills from Meta- World (Yu et al., 2020) (see Fig- ure 2). Similar prior works (Christiano al., 2017; Lee al., 2021 the agent learns to perform a task only by getting feedback from an oracle scripted teacher that provides clean pref- erences between trajectory segments according to the sum of ground truth reward values for each trajectory segment. Because the simulated human teacher's preferences are generated by ground truth reward values, we measure the true average return of trained agents as evaluation metric.
We consider a combination of RUNE and PEBBLE (Lee et al., 2021b), an off-policy preference-based RL algorithm that utilizes unsupervised pre-training and soft actor-critic (SAC) method (Haarnoja et al., 2018) (see Appendix A for more details about algorithm procedure and Appendix B for more experimental details). To address the issue that there can multiple reward functions that align to the same set of teacher preferences, we bound all predicted reward values to a normalized range of (-1, 1) by adding tanh activation function after the output layer network architectures.
For all experiments including PEBBLE and RUNE, we train an ensemble of reward functions according Equation 2. Specifically, the ensemble. We use the same set of training data, i.c. sampled set feedback queries, but different random batches train each network in the ensemble. Parameters of each network independently optimized to minimize the cross entropy loss of their respective batch of training data according Equation 2. In addition, we use original hyperparameters of preference-based RL algorithm, which are specified in Appendix B and report mean and standard deviations across 5 runs respectively.
different random initialization for each network in
5.2 IMPROVING SAMPLE-EFFICIENE
To evaluate sample-efficiency of our method, we compare to the following exploration methods:
State entropy maximization (StateEnt; Mutti et al. 2021; Liu & Abbeel 2021): Maximizing
the entropy of state visitation distribution (s)). We utilize a particle estimator (Singh et al., 2003), which approximates the entropy by measuring the distance between k-nearest neighbors (k-NN) for each state.
Disagreement Pathak et al. (2019): Maximizing disagreement proportional to vari- ance in predictions from ensembles Var (s,, 2 We train

---

Published as a conference paper at ICLR 2022
semble of forward dynamics models to predict ground truth next state from cur- rent state and action 9s (5141/55, ar) by minimizing sum of prediction errors, i..
Σ (s 8s, az) - S1+1l12.
ICM Pathak al. (2017): Maximizing intrinsic reward proportional prediction error [1g($ [s,, av) - S-1ll22- We train a single dynamics model to predict ground truth next state from current state and action ( St, ar) via regression.
For all methods we consider, we carefully tune a range of hyperparameters and report the best results. In particular, we consider 3o 0.05 and E {0.001, 0.0001, 0.00001) for all exploration methods, and & (5, 10} for state entropy based exploration. We provide more details about training and implementation in Appendix B.
Figure shows the learning curves PEBBLE with various exploration methods. First, we remark that previous exploration methods (i.e., StateEnt, Disagree and ICM), which encourage agents to visit novel states, consistently improve sample-efficiency of PEBBLE on various tasks. This shows the potential and efficacy of exploration methods to further improve sample-efficiency of preference- based RL algorithms.
In particular, in Figure 3, compared to existing previous exploration methods, RUNE consistently exhibits superior sample efficiency in all tasks we consider. This suggests that exploration based on reward uncertainty is suitable for sifference-basee RL algorithms, such it encourages visitations to states and actions that are uncertain with respect to human feedback and thus can capture the novelty in distinctive perspective. We provide additional experiment results about better sample- efficiency in Figure (()) of Appendix C.
We also emphasize the simplicity and efficiency of RUNE compared other existing schemes (such as ICM and Disagree) because our method does not introduce additional model architectures (c.g., ensemble of forward dynamics models) to compute exploration bonus intrinsic rewards.
5.3 IMPROVING BEEDBACK-EFFICIBER
In this section, we also verify whether our proposed exploration method can improve the feedback- efficiency of both off-policy and on-policy preference- RI. algorithms. We consider PEB- BLE Lee et al. (2021b), an off-policy preference-basee based RI. algorithm that utilizes unsupervised pre- training and soft actor-critic (SAC) method (Haarnoja et al., 2018), and PreiPPO Lee et al. (2021c), an on-policy
sifferenne-bssee RL algorithm that utilizes unsupervised pre-training and proximal policy gradient (PPO) method (Schulman et al., 2017), respectively.
As shown in Table 1, compare performance of PEBBLE and PrefPPO with and without RUNE respectively using different budgets of feedback queries during training. With fewer total queries, we stop asking for human preferences earlier in the middle of training. We use asymptotic success rate converged at the end of training steps as evaluation metric. Table suggests that RUNE achieves consistently better asymptotic performance using fewer number of human feedback; additionally RUNE shows more robust performance with respect to different budgets of available human feed- backs. This shows potential of exploration in scaling preference-basee RL to real world scenarios where human feedback are usually expensive and time-consuming to obtain.
We report corresponding learning curves to show better sample efficiency of RUNE compared to PEBBLE or PrefPPO under wide variety of tasks environments in Figure 6, 7, 8, 5 of Appendix C.
5.4 ABLATION STUDY
Ensemble size. As intrinsic rewards of RUNE only depend on learned reward functions, we in- vestigate the effects of different number of reward function ensembles. In particular, we consider {3, 5, 7) number of reward function ensembles. In Figure 4(a), sample-efficiency of RUNE remains robust and stable with different number in the ensemble of reward functions. Additionally, RUNE can achieve better sample-effiiincy in preference- based RL while requiring fewer number of reward functions ensembles. This shows potentials of RUNE specifically suitable for cnfference-bssee RL in improving compute efficiency, as reduces the necessity of training additional reward functions architectures and still achieves comparable performance.
7

---

Published as a conference paper at ICLR 2022
TASK
FEEDBACK QUERIES
METHOD
CONVERGENT SUCCESS RATE
PEBBLE
0.98 1 0.08
10000
PEBBLE + RUNE
1 1
DRAWER OPEN
PEBBLE
0.94 = 0.08
5000
PEBBLE + RUNE
0.99 0.02
PEBBLE
0.8 0.4 1 0.8 = 0.08
10000
PEBBLE + RUNE
SWEEP INTO
PEBBLE
5000
PEBBLE + RUNE
0,9 1 0.14
PEBBLE
0.66 + 0.42
5000
PEBBLE RUNE
0.8 + 0.4
DOOR UNLOCK
PEBBLE
0.64 0.45
2500
PEBBLE + RUNE
0.8 + 0.4 I
PEBBLE
4000
PEBBLE + RUNE
1 1 0
DOOR OPEN
PEBBLE
0.9 = 0.2
2000
PEBBLE + RUNE
1
PEBBLE
I
1000
PEBBLE + RUNE
1 1
DOOR CLOSE
PEBBLE
0.8 = 0.4
500
PEBBLE RUNE
1 0
PEBBLE
0.94 = 0.08
1000
PEBBLE + RUNE
1 1
WINDOW CLOSE
PEBBLE
0.86 0.28
500
PEBBLE + RUNE
0.99 + 0.02
PREFPPO
0.46 - 0.20
20000
PREFPPO RUNE PREFPPO
0.64 + 0.18
BUTTON PRESS
0.35 1 0.31
10000
PREFPPO RUNE
0.51 0.27
Table 1: Success rate of off- and on-policy preference-based RL algorithms (e.g. PEBBLE and PrefPPO) in addition to RUNE with different budgets of feedback queries. The results show the mean averaged and standard deviation computed five and the best results indicated in bold. All learning curves (including means and standard deviations) are in Appendix C.
Number queries per feedback session. Human preferences are usually expensive and time- consuming collect. As shown in Table I, RUNE is able achieve better asymptotic performance under different budgets of total human preferences. We further investigate the effects of different queries in each feedback session performance of PEBBLE and RUNE. In particular, consider (10, 50} number of queries per feedback session equally spread out throughout training. In Figure 4(b), we indicate asymptotic performance PEBBLE baseline by dotted horizontal lines. shows that as 80% of feedbacks are eliminated, asymptotic success rate of PEBBLE baseline largely drops, while in contrast RUNE remains superior performance in both sample-efficiency and asymptotic success. Thus RUNE is robust to different number of available feedback queries constraints and thus a suitable exploration method specifically beneficial for preference-based RL.
Quality of learned reward functions. We use Equivalent-PPliccy Invariant Comparison (EPIC) distance Gleave et al. (2020) between learned reward functions and ground truth reward values as evaluation metrics. Gleave al. (2020) suggests that EPIC distance is robust to coverage distribu- tions of states and actions and is thus reliable metric to quantify distance between different reward functions under the same transition dynamics. We generate (s, a) from each task uniformly and independently at random order to obtain a wide coverage of state distributions.
As in Figure 4(c), compared PEBBLE, reward functions learned from RUNE are closer to ground truth reward evaluated based on EPIC distance and converges faster during training. We provide additional analysis different task environments in Figure 9(b) and (()) of Appendix C.
8

---

Published as a conference paper at ICLR 2022
100
100
PABBLE PEBBLE RUNE
Success Rate (%)
Mir
60
Succe ttate (54)
EPIC Distance
0.6
00-
0.4
ensamble
onsemble 5
per
Environment Steps (x109)
1.0
Endfronment Steps (x1df)
044 6.t
08
12
Environment Steps (x10f)
0.1
(a) Number of reward functions
(b) Number of queries per feedback (c) EPIC distance between learned reward and true reward
session
Figure 4: Ablation study on (a) Door Open, (b) Window Close, and (c) Door Close. We measure (a) Effects of different number of learned reward model ensembles on RUNE; (b) Effects of different number of queries received per feedback session on PEBBLE and RUNE; and (c) EPIC distance between the ensemble of learned reward functions and true reward in the task environment. The solid line and shaded regions represent the mean and standard deviation, respectively, across five runs. The dotted line in Figure 4(b) indicates asymptotic success rate of PEBBLE baseline with same hyperparameters.
We further analyze quality of reward functions learned by RUNE compared to true reward in Figure 10 of Appendix C. Learned rewards optimized by RUNE is well-aligned true reward, it can capture diverse patterns of true reward values from the environment.
DISCUSSION
In this paper, we present RUNE, a simple and efficient exploration method for preference-based RL algorithms. To improve sample- and feedback-effiiiency of preference-based RI, different from previous works, we investigate the benefits of exploration methods in preference-based RL. We show the significant potential of incorporating intrinsic rewards to drive exploration because it improves sample-efficiency preference-basee RL.
For our proposed exploration scheme RUNE, show that it is useful for preference-based RL because it showcases consistently superior performance both sample- and ceedback-efficiennyy compared to other existing exploration methods. Here emphasize that RUNE takes advantage of information in reward functions learned from human feedback, to measure the novelty of states and actions for exploration. This is different from existing estimates of uncertainty for exploration, as our method in particular encourages exploration aligned to teacher preferences.
In conclusion, we hope that our work could demonstrate the potential of exploration improve sample- and (eedback-efficiene, of preference-based RL, and to encourage future works to develop novel exploration methods guided by human feedback.
ACKNOWLEDGEMENTS
This research is supported in part by Open Philanthropy. We thank anonymous reviewers for criti- cally reading the manuscript and suggesting substantial improvements.
REFERENCES
Riad Akrour, Mare Schoenauer, and Michele Sebag. Preference-based policy learning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2011.
Dario Amodei, Chris Olah Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. In arXiv preprint arXiv: 1606606606665 2016.

---

Published as a conference paper at ICLR 2022
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, 2016.
Erdem Biyik and Dorsa Sadigh. Batch active preference- learning of reward functions. In Conference on Robot Learning, 2018.
Erdem Biyik, Nicolas Huynh, Mykel J Kochenderfer, and Dorsa Sadigh. Active preference-based gaussian process regression for reward learning. In Robotics: Science and Systemx, 2020.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345 1952.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018.
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via qr
ensembles. arXiv preprint arXiv:1706. 01502, 2017.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. arXiv preprint arXiv:2006. 13900, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning, 2018
Dylan Hadfield-Menell, Smitha Milli, Picter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, 2017.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference on Machine Learning, 2019.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. In Advances in Neural Information Processing Systems, 2016.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Kimin Lee, Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Sunrise: simple unified frame- work for ensemble learning in deep reinforcement learning. In International Conference on Ma- chine Learning, 2021a.
Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. International Conference on Machine Learning, 2021b.
Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel. B-pref: Benchmarking preference- based reinforcement learning. In Thirty-ffth Conference Neural Information Processing Sys- tems Datasets and Benchmarks Track (Round 1), 2021c. URL https: http::ps///ppeeeeeeeeee net/
forum? id -
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint arXiv: XXXx:/9066.0/2 2019.
10

---

Published as a conference paper at ICLR 2022
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.078) 2018.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. arXiv preprint arXiv:2103.04551 2021.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuogiu. Asynchronous methods for deep reinforcement learning. International Conference on Machine Learning. 2016.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. A policy gradient method for task-agnostic exploration. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, 2016.
Georg Ostrovski, Mare Bellemare, Aaron van den Oord, and Rémi Munos. Count-based explo- ration with neural density models. In International Conference on Machine Learning, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta Gupta. Self-supervised exploration via disagree- ment. In International Conference on Machine Learning, 2019.
Patrick M Pilarski, Michael R Dawson, Thomas Degris, Farbod Fahimi, Jason P Carey, and Richard S Sutton. Online human training of a myoelectric prosthesis controller via actor-critic reinforcement learning. In International Conference on Rehabilitation Robotics, 2011.
Dorsa Sadigh, Anca D Dragan, Shankar Sastry, and Sanjit A Seshia. Active preference- learn- ing of reward functions. In Roboties: Science and Systems, 2017.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 17Xiv:1707.06347 2017.
Ramanan Sekar. Oleh Rybkin, Kostas Danilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning explore via self-supervised world models. In International Conference on Machine Learning, 2020.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning (ICML), 2021a.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Picter Abbeel, and Kimin Lec. State entropy maximization with random encoders for efficient exploration. arXiv preprint arXiv:2102.09430. 2021b.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- est neighbor estimates entropy. American journal of mathematical and management sciences, 23(3-4):301-322 -211 2003.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl:
Contrastive unsupervised representa- tions for reinforcement learning. In International Conference on Machine Learning, 2020.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint
2015.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT Press, 2018.

---

Published as a conference paper at ICLR 2022
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. exploration: study of count-based exploration for deep reinforcement learning. In Advances in Neural Information Processing Systems, 2017.
Scbastian Thrun. The role of exploration in learning control. In Handbook of Intelligent Control: Neural, Fuzzy and Adaptive Models, 1992.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. A bayesian approach for policy learning from trajectory preference queries. In Advances Neural Information Processing Systems, 2012.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017. URL http: http:///wmooorr....eeeeoeee// wttpr//apper///leg
634. m...mmll..
Jeff Wu, Long Ouyang, Daniel M Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Chris- tiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109, 10862, 2021.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, 2020.
Matt Zucker, J Andrew Bagnell, Christopher G Atkeson, and James Kuffner. An optimization ap- proach to rough terrain locomotion. 2010 IEEE International Conference on Robotics and Automation, pp. 3589-3595. IEFE, 2010.
12

---

Published as a conference paper at ICLR 2022
Appendix
A
ALGORITHM DETAILS
RUNE. We provide the full procedure for RUNE with PEBBLE Lee et al. (2021b), an off-policy preference-based RI. algorithm, Algorithm 1.
Algorithm RUNE: Reward Uncertainty for Exploration
1: Initialize policy Fg and reward model Pr
2: Initialize replay buffer B
3: // UNSUPERVISED PRE-TRAINING
4: Maximize the state entropy H(s) - Eamp(s) logg(ss)
5: // POLICY LEARNING
6: for each training step /. do
7: // REWARD LEARNNNGG
8: if iteration % K == 0 then
Generate queries from replay buffer {(o", of)) B and corresponding human
9:
feedback (yr)) Nuzzy
10:
Update reward model F. according to equation 2
11:
// RELABEL BOTH EXTRINSIC AND INTRINSIC REWARD
12:
Relabel entire replay buffer B using Pe
Relabel extrinsic reward r (sj, a;) Fmenn (Sj, aj)
Relabel intrinsic reward y-iat (sj, aj) fsu (8j: aj)
end if
16:
// UPDATE POLICY
for each gradient step do Sample minibatch from replay buffer {($j, aj 85-1, r, f aj), r a;)) B
18:
19:
Update Be + 31 p) Let rrotal +- po + 8 .raut
20:
21:
Update é with transitions {(5,, aj. 85-1, total (sy, a1)1
22:
end for
INTERACTION WITH ENVIRONMENT
24:
for each timestep do
25:
Collect S: by taking a: ~ TT(aa(s))
26:
Store transitions B +- B U {(St, a St41, (p a:))
27:
end for
28:
end for
Hyperparameter
Value
Segment Length
50
Number of Unsupervised Pre-training Steps
9000
Interaction time
10000
Reward batch
50 (Door Close, Door Open), 100 (Drawer Open)
Total feedback
1000 (Door Close), 5000 (Door Open), 10000 (Drawer Open)
Sampling scheme
Disagreement Sampling
Reward model number of layers
3
Reward model number of hidden units
256
Reward model activation functions Reward model output activation function
LeakyRelu
TanH
Number of reward functions
3
Optimizer
Adam (Kingma & Ba, 2015)
Initial learning rate
0.0003
Table 2: Hyperparameters of the PEBBLE algorithm.
13

---

Published as a conference paper at ICLR 2022
Hyperparameter
Value
Hyperparameter
Value
Initial temperature
0,1
Hidden units per each layer
256
Learning rate
0,0003
Batch Size
512
Critic target update freq (31, (2) Optimizer
2
Critic EMA
0.005
(.9, .99))
Discount
.99
Adam (Kingma & Ba, 2015)
Table Hyperparameters of the SAC algorithm. Most hyperparameters values are unchanged across environments with the exception for learning rate.
B EXPERIMENTAL DETAILS
B. TRAINING DETAILS
For our method, we use the publicly released implementation repository of the PEBBLE algorithm (https: : //ggttttuu com/ com.pokaxpoka..
Pref) with the full list of hyperaarameters specified in Table and the SAC algorithm (https: / //github com dwe.dennnnaaaae wwwiiissrrtttspseae www.aaattttttttttttaaaoeoo sac) with the full list of hyperparameters specified in Table 3.
B.2 IMPLEMENTATION DETAILS OF EXPLORATION METHODS
For all exploration methods, we follow implementation of the publicly release repository of RE3 (https:/ / thub. coot:/yongggosso...oooooooo RRE3 to update policy with p = rext B pint
RUNE, We emphasize one simplicity in the implementation of RUNE. Following relabeling tech- nique in Lee et al. (2021b), after each update in reward learning step, we relabel the entire play buffer B to store both predicted reward functions learned from human feedback (extrinsic rewards ext)) and uncertainty in updated reward functions (intrinsic rewards yint in RUNE). This is because our learned reward functions parameters remain unchanged during RL training or agent interaction with the environment between subsequent feedback sessions. Thus, RL training, y potal rxt + 3r pant can be directly taken from replay buffer. As for hyperparameter related to exploration, we consider 3g 0.05, p (0.001, 0.0001, 0.00001).
State Entropy Maximization. Seo et al. (2021b) We use intrinsic reward (* = =||s; - k-
followed Seo al. (2021b) and Liu & Abbeel (2021). We use raw state space to compute entropy values as intrinsic rewards. We use code for unsupervised pre-training from the publicly release implementation repository of PEBBLE algorithm (attps / //gitttuu com/ pokaxpoka/ Pref) to compute k-NN and estimate state entropy. As the size of replay buffer grows signif- icantly as training time increases, we sample a random mini-batch B' from replay buffer and compute N--N with respect random mini-batch. We minibatch size 512, which is reported as a default hyperparameter in Seo et al. (2021b). As for hyperparameters related to exploration, we consider 30 0.05, p (0.001, 0.0001, 0.00001). and (5, 10).
Disagreement. Pathak et al. (2019) We train an ensemble of 5 forward dynamics model 9r, fol- lowing original results in Pathak et al. (2019). These dynamics models trained to predict next state S1+1 based current state and action (s,, (s,, minimize Σ (S a.) 8141l22- We use intrinsic reward proportional to variance in ensemble of predictions 7 (s,t az) (s444ss)))
For consistency we use same neural network architecture and opti- mization hyperparameters as reward functions specified in Table 2. We follow implementation of Disagreement agent in publicly released repository URLB (https : anonymous. 4oper. a science/ catncc//r/urla As for hyperparameter related to exploration, we consider So 0.05, E (0.001, 0.0001, 0.00001}.
ICM. Pathak et al. (2017) We train single forward dynamics model g to predict next state 8:41 based on current state and action (s,, az), i.e. minimize [|g(st+1|s; a,) - s141||2. We use intrinsic reward proportional to prediction error of trained dynamics model, i.e. pnn (s;, a,) [[g(s++1|s) a.) - 1441/12. For consistency we use same neural network architecture and optimization hyperparameters reward functions specified in Table 2. We follow implementation of ICM agent
14

---

Published as a conference paper at ICLR 2022
PESBLE PESOLE HUNE
800
606
026
310
6831 5e coN cin ERB 50RE-00
Stvrement 1129
Separdey
(a) Door Close (feedback (b) Door Close (feedback (c) Button Press 500)
(feed- (d) Button Press (feed-
250)
back=20K)
back=10K)
Figure 5: RUNE performs better than PEBBLE and PrefPPO in variety of manipulation tasks using different budgets of feedback queries. The solid/dashed line and shaded regions represent the mean and standard deviation, respectively, across five runs.
MMMM
8G-
Succics 82
Surces Rate
00-
Entironmmnn Staps (K10")
(a) Door Open (feedback 4K)
(b) Door Open (feedback = 3K) (c) Door Open (feedback = 2K)
Figure 6: RUNE performs better than PEBBLE in Door Open using different budgets of feedback
queries. The solid/dashed line and shaded regions represent the mean and standard deviation, spectively, across five runs.
in publicly released repository URLB (ntipss//anooyymuua hopennssciencee//urre.
As for hyperparameterr related exploration, we consider &n 0.05, p E
C ADDITIONAL EXPERIMENT RESULTS
Improving Feedback-Efficiennyy We provide additional experimental results various tasks from Meta-Worke benchmark Yu et al. (2020) using different budgets of feedback queries during training. We report asymptotic evaluation success rate at the end of training in Table 1. Here report corresponding learning curves and additional results of wider choices of total feedback queries. We observe RUNE performs consistently at least same or better than PEBBLE baseline in different tasks with different budgets of feedback queries.
Improving Efficieecf As shown in Figure 9a), RUNE outperforms other existing ploration baselines measured on the episode return of Pick Place. This shows applicability of our method harder tasks.
Reward Functions Learned by RUNE. We provide additional analysis of reward functions learned by RUNE, compared to true reward function defined by Meta-World environments. Figure 10 demonstrates in all tasks we consider, learned reward functions optimized by RUNE is well-aligned true reward from teacher feedback, because can capture diverse patterns of true reward values from the environment.
15

---

Published as a conference paper at ICLR 2022
330
300
160
Sucorta Pott (44)
MM
40
20
20-
22
0.4
0.5
0.7
7.2
0.5
1.0
Environment Staps D4 20P
Ensironmmnt Stopa (XLLPP)
Enstronment Staps (x20f)
(a) Drawer Open (Teedback = 7K) (b) Drawer Open (feedback 5K) (c) Drawer Open (feedback 3K)
Figure 7: RUNE performs better than PEBBLE in Drawer Open using different budgets of feed- back queries. The solid/dashed line and shaded regions represent the mean and standard deviation, respectively, across five runs.
330
W
1.0
MW
Success Satte 190
Success Rate
Succos
40
6.00
0.75
1,00
1/50 3.75 2:00
0.2
Eovmenmentt Steps (X1.0%
Enveronmmen Stapps (X110))
Enstronment Strpps 2x
(a) Sweep Into (feedback = 10K) (b)
Window Close (feedback (c) Door Unlock (feedhack 5K)
1 K)
Success Sate (50
40-
(5)
80-
(56)
60-
60-
Sheps (4:10%)
Steps (4110")
Steps
(d) Sweep Into (feedback = 5K) (c) Window Clos<(fecdbckk=500) (f) Door Unlock (fcedback=2200)
Figure 8: RUNE performs better than PEBBLE in variety of manipulation tasks using different budgets of feedback queries. The solid/dashed line and shaded regions represent the mean and standard deviation, respectively, across five runs.
16

---

Published as a conference paper at ICLR 2022
6.0
0.20
0.7
Episode Retum
510
REALE
CL
EPIC EPIC Distance
0.5
EPIC Distance
0.0
0.5
200
0.50
9.4
1.00
0.45
0.40
0.5
1.0
2.5
0.3
0.8
0.0
0.1
9.2
0.3
Environment Steps (x10F)
Steps (x10f)
Environment Steps (x10f)
(a) Pick Place (feedback 10K)
(b) Door Open (feedback = 5K) (c) Drawer Open (feedback 10K)
Figure 9: (a) RUNE achieves better sample-efficiency on manipulation tasks from World benchmark. (b/c) EPIC distance between the ensemble of learned reward functions and true reward in task environments. The solid/dashed line and shaded regions represent the mean and standard deviation, respectively, across five runs.
Reward (s.. (x)
Reward (s,, ax)
-1-
-2-
Reward
-.0
-23
Learned Reward: RUNE
-10
Learndd Reward: RUNE
Learned Reward: RUNE
102
200
30a
400
500
120
220
450
500
120
200
382
400
500
Timestep
Timestep
Timestep
(a) Door Close (feedback = IK)
(b) Door Open (feedback = SK)
(c) Drawer Open (feedback 10K)
Figure 10: Time series of normalized learned reward (blue) and the ground truth reward (red) using rollouts from policy optimized by RUNE.
17


Reviews:
[{"review_id": "TmNBKAkcqut", "reviewer": "Reviewer_v2Tv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and clear to understand. The method can be considered sound, as it is based on established building-blocks. Some minor issues should be improved, but have no substantial impact:\n- Ensemble disagreement (mentioned in abstract) and ensemble variance are not the same thing\n- Preference notation is introduced as (1,0),(0,1), but seq_0 > seq_1 is used\n- Indifference seems to be not handled, but is introduced. (Which is ok, but should be clarified)\n- The agent is not encouraged to visit more \"unseen\" states, but \"uncertain\" states (wrt. the true reward)\n\nAdditionally, the method has a hidden assumption, which is quite common, but should be made explicit: The preference feedback is assumed to be stationary and acyclic. Without these assumptions, the surrogate reward may not converge towards a low variance solution and the mean reward can not sufficiently reflect the (unknown, human) preference function. In the cyclic preference, case this not possible at all (with a scalar reward) and in the non-stationary case, the (unweighted) mean is not a good estimator. Both assumptions are often violated by real-world human preference feedback.\n\nThe evaluation is ok, but more domains should be added for allowing to derive more general conclusions. The ablation study is a good addition, with the exception of the \"Quality of learned reward functions\" section. As acknowledged by the authors, surrogate rewards may have a different scale than the underlying, true rewards (also when not using tanh). Furthermore, even rewards that differ not only in scale, can induce the same (optimal) policy. Therefore, the comparison should be done in terms of a different metric, like ranking error over states and/or sequences. Even rescaling the learned reward would help, because with the current plot, alignment is not clearly visible.\n\nHowever, surrogate-reward uncertainty-based exploration is not new and the according advantages have also been shown before (e.g. Akrour 2011 - for more, see \"Trajectory Generation\" in https://jmlr.org/papers/volume18/16-634/16-634.pdf). None of these methods have been applied to PEEBLE/DeepRL (as far as the reviewer is aware), but this still reduces the validity of the claims. Furthermore, the idea of using ensemble variance as intrinsic reward bonus is also already known from plain RL (as mentioned by the authors). Resultingly, the publication is applying known methods to a different framework (PEEBLE). The impact of this issue can maybe reduced by clearly stating the differences to other preference-based exploration strategies. \n\nHowever, even if these issues are resolved, the degree of novelty will likely stay limited. Therefore, the evaluation should be substantially improved (besides adding more domains, potentially adding some results from the appendix). Some suggestions:\n- Effect of different beta values\n- Comparison to \"comparable\" formulations, for intrinsic reward, based on preference uncertainty (e.g. ensemble disagreement, instead of variance)\n- Optimal/terminal policy in terms of preference queries (instead of only two query counts)", "summary_of_the_paper": "The authors introduce a method for integrating reward uncertainty into the exploration/exploitation tradeoff in a preference-based reinforcement learning setting. The method uses a surrogate reward ensemble variance as intrinsic reward bonus.", "summary_of_the_review": "The proposed method is of interest within the domain of preference-based reinforcement learning, as guiding exploration can reduce human operator workload. The novelty of the approach is limited and the claims have been shown before, but in a different framework. This limitation has to be overcome or leveled by a thorough evaluation, allowing generalized conclusions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "PCkwcwMLc2S", "reviewer": "Reviewer_VUcZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths: A simple solution to the problem of epistemic uncertainty driven exploration for RL from preferences. The experimental results demonstrate improved sample efficiency over other baselines. Also, the writing and the structure of the paper is clear.\n\nweaknesses: The paper suggests that capturing the epistemic uncertainty at the reward prediction level is better than capturing it at the level of the dynamics. Although I find this intuition correct, you could show this theoretically as well since the reward uncertainty incorporates the state transition uncertainty as well. I'd like to see some theoretical analysis that would support the experiments. Also, I see this paper as a direct application of the epistemic uncertainty driven exploration in RL from preferences that make its novelty small.\n\nC1: \"We take advantage of an ensemble of reward models ... which is not available in other traditional RL settings\" - What is the challenge here? To have a reward model or an ensemble of reward models? I don't think that any of those is challenging. The disagreement method doesn't need to happen over the forward dynamics model but can also happen over the rewards. In fact, the intuition here is that you model the epistemic uncertainty of the predictor of interest. If you want to cover more states you model the epistemic uncertainty over the forward states, if you want to cover more downstream task-related states you model the epistemic uncertainty over the reward model.\n\nC2: \"Quality of learned reward functions\" - The plot doesn't seem to add anything valuable. The scales are different (you explain why) but that could also have been the result of learning a wrong reward function. What you want perhaps to show instead is that for sampled $s_t, a_t$ pairs the predicted reward and the true reward is correlated.", "summary_of_the_paper": "With this work, the authors propose a bayesian active learning approach to the problem of reinforcement learning from preferences. To do this, they model the epistemic uncertainty of the reward function to intrinsically motivate the RL agents to explore. The solution demonstrates improved sample efficiency.", "summary_of_the_review": "I find the paper clearly written and a direct application of epistemic uncertainty driven exploration in RL from preferences. Although simple, its novelty is marginal. Also, it lacks theoretical justification (e.g. bayesian RL intuitions, why uncertainty over the rewards might work better than uncertainty over dynamics, aleatoric vs epistemic uncertainty, etc.). Finally, the experiments, although they support the argument, there are experiments that don't contribute to the discussion (C2)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P4jMh9WsgWy", "reviewer": "Reviewer_WzMo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Exploration strategies for preference-based RL is underexplored and this paper takes a step towards practical RL methods that can learn with humans-in-the-loop. Although the idea is not groundbreaking given existing work, RUNE is simple, scalable, and easy to implement. Adding an uncertainty-based intrinsic reward to guide preference-based RL is intuitive and at its core, I think the underlying idea can improve existing methods. \n\nHowever, I do have concerns about this particular instantiation of the approach. First, the method can be better explained and it's unclear how RUNE is trained:\n\n- Is each model in the ensemble trained using the same set of queries and feedback? Are the models trained independently given their respective data? \n- In general, there could be multiple reward functions that give rise to the same preference feedback (e.g., a translation of the reward function). In this case, each model in the ensemble might learn correct but different reward functions, yet the standard deviation of the state rewards would be non-zero? Is this still a meaningful estimate of uncertainty?\n- In the above case, is the average reward indicative of the actual extrinsic reward? \n\nSecond, the main claims are not well-validated by the experiments. While I appreciate the authors used rather complex robot domains, it is difficult to understand if RUNE actually increases exploration:\n\n- Iit is unclear if the better performance is due to extra exploration or simply the use of a better reward estimator, i.e., the ensemble. I encourage the authors to quantitatively compare the amount of exploration when RUNE is used with PEBBLE. If the current domains are too complex, a simple gridworld-type domain (or small-scale continuous domain) can be used. \n- If my interpretation is correct, Fig 5b seems to indicate that less exploration is performed with RUNE. We would expect that disagreement to be higher at the middle portions as the method explores uncertain states?  \n- For the experiments, how many runs are performed for each method? Is there noise in the simulation human feedback? Does the simulated human prefer one trajectory above all others or is there some ambiguity? These details are important to ascertain the robustness of the method and results. Also, please use a different color for the methods in Fig 4; I am unable to compare the standard deviations. \n- The analysis on the quality of learned rewards can be improved. Normalizing the rewards to a standard range, e.g., [0,1], can aid comparisons in Fig 5.c. Again, using a gridworld here may help to better illustrate that a reasonable reward function is learnt. At present, the figure shows the rewards from a single trajectory rather than the state space. \n\n\n", "summary_of_the_paper": "This work proposes an ensemble-based intrinsic reward to improve exploration in preference-based RL. The key idea is to incorporate uncertainty in teacher preferences as an intrinsic reward. An ensemble of reward functions is used to capture this uncertainty. The paper discusses experiments on three robot tasks with ablation studies to investigate the method's performance. ", "summary_of_the_review": "Exploration for preference-based RL is an interesting problem, but the current paper has multiple issues that should be addressed. I hope the authors can provide some clarification my questions above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G7aB2dcqr4f", "reviewer": "Reviewer_x6bk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of efficient exploration is very important for reinforcement learning, especially for cases where human feedback is used. The authors propose an interesting approach of looking at reward model ensemble. At a high level, this idea makes sense, but looking at the evaluation, the effect of the algorithm on the results is very minor. In many of the plots, it is not easy to see a big difference between different methods. In addition, the evalutation is very limited. The authors use one problem (manipulation) and 3 tasks from that problem. \n\nThe paper is well written, but same citations are repeated over and over. This is both unnecessary and distrupts reading flow.", "summary_of_the_paper": "The paper proposes an exploration method for \"preference based Reinforcement Learning\" methods, where human feedback is incorporated to the training regime. The authors use an ensemle of learned reward models and add an intrinsic reward based on disagreement (or uncertainity). The authors test the idea on robotic manipulation tasks from meta-world. The agent learns purely based on the feedback from a teacher that provides preference of trajectory one over another. The tasks presented are \"door close\", \"door open\" and \"drawer open\".\nThe authors combine their exploration strategy (RUNE) with the preference based learning method PEBBLE and compare it with PEBBLE with some other exploration strategies. The results show that the proposed method provide some improvement over others. The authors also compare with PEBBLE by using 700 feedback instead of 1000. The results show minor improvement.", "summary_of_the_review": "The paper tackles an interesting problem for RL, the proposed method is incremental but partially novel. On the other hand, the evaluation is limited and the results are not significant.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper. The scores were initially below the bar. The novelty of the work is limited and there are strong claims in the paper that should be revised. The authors can also do a better job in positioning their work with respect to the existing results. However, the authors managed to address several questions/concerns of the reviewers and convince them to raise their scores. I would strongly recommend the authors to address the rest of the reviewers' comments, especially those related to strong claims and connection to related work, and further improve their work in preparing its final draft.
Decision: Accept (Poster)