Few-Shot Preference Learning for Human-in-the-Loop RL
Authors: Donald Joseph Hejna III
Dorsa Sadigh
Venue: CoRL 2022 Poster (2022)
Keywords: preference learning
interactive learning
multi-task learning
human-in-the-loop

Abstract: While reinforcement learning (RL) has become a more popular approach for robotics, designing sufficiently informative reward functions for complex tasks has proven to be extremely difficult due their inability to capture human intent and policy exploitation. Preference based RL algorithms seek to overcome these challenges by directly learning reward functions from human feedback. Unfortunately, prior work either requires an unreasonable number of queries implausible for any human to answer or overly restricts the class of reward functions to guarantee the elicitation of the most informative queries, resulting in models that are insufficiently expressive for realistic robotics tasks. Contrary to most works that focus on query selection to \emph{minimize} the amount of data required for learning reward functions, we take an opposite approach: \emph{expanding} the pool of available data by viewing human-in-the-loop RL through the more flexible lens of multi-task learning. Motivated by the success of meta-learning, we pre-train preference models on prior task data and quickly adapt them for new tasks using only a handful of queries. Empirically, we reduce the amount of online feedback needed to train manipulation policies in Meta-World by 20$\times$, and demonstrate the effectiveness of our method on a real Franka Panda Robot. Moreover, this reduction in query-complexity allows us to train robot policies from actual human users. Videos of our results can be found at \url{https://sites.google.com/view/few-shot-preference-rl/home}.

Few-Shot Preference Learning for Human-in-the-Loop RL
Joey Hejna
Dorsa Sadigh Stanford University dorsaëcs sst.ssaaasoo edu
Stanford University jhe www.:ssstaaoo.....o
While reinforcement learning (RL) has become a more popular ap- proach for robotics, designing sufficiently informative reward functions for com- plex tasks has proven be extremely difficullt due their inability to capture hu- man intent and policy exploitation. Preference based RL algorithms seek to overcome these challenges by directly learning reward functions from human feedback. Unfortunately, prior work either requires an unreasonable number queries implausible for any human to answer overly restricts the class of re- ward functions to guarantee the elicitation of the most informative queries, re- sulting in models that are insufficiently expressive for realistic robotics tasks. Contrary to most works that focus on query selection to minimize the amount of data required for learning reward functions, take opposite approach: expanding the pool of available data by viewing human-in-the-loop RL through the more flexible lens of multi-task learning. Motivated by the success of meta- learning, we pre-train preference models on prior task data and quickly adapt them for new tasks using only handful of queries. Empirically, we reduce the amount of online feedback needed to train manipulation policies in Meta- World by 20x, and demonstrate the effectiveness of our method on real Franka Panda Robot. Moreover, this reduction in query-complexity allows to train robot policies from actual human users. Videos our results can be found https://sitent google........ooooooooooo.o..... wwwggle..om/vaae
Abstract:
Keywords: Preference Learning, Interactive Learning, Multi-task Learning
Introduction
The success of deep reinforcement learning (RL) methods in game-playing and simulated domains [1] has inspired recent work applying RL-based techniques to real-world robot control to middling success. Integral to the success of deep RL methods the reward function, which describes the desired behavior of the learning agent. While training robots via trial and error holds great promise, designing suitable reward functions remains challenging. For example, consider teaching a robot open door. The simplest reward function would be sparse - providing the robot with positive reward only when the door has been opened. However, such sparse signals offer little learning signal, hampering exploration and enlarging sampling complexity. Conversely in designing dense reward function, practitioners are tasked with summarizing multiple objectives like door angle or proximity the handle into single scalar. Such reward functions have proven be difficult to design [2] and can even cause agents to learn unintended behaviors. Hand-designed dense reward functions often do not directly parallel the goal-conditions humans want them to capture, causing RL agents to exploit them and potentially leading to hazardous policies that do not align with human intent [3]. All of these problems are exacerbated more realistic, multi-task scenarios with large state and action spaces [4] where we might wish to teach agents how to complete variety of tasks in their environment. A robot that can only open doors provides little utility the real world. Given the effort required to design single reward function, constructing reward functions for an entire family of tasks impractical.
Recent works attempt to circumvent the basic challenges of reward design by learning reward functions directly from human preferences. This paradigm has numerous advantages: learned
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

reward functions are dense [5, 6], easily aligned with human intent [7], and can be adapted [8]. While demonstrations are often difficult to provide due to expensive data collection [9] and large domain gaps [10, 11], human preferences can often be elicited solely through simple pairwise comparisons. However, given the large continuous state and action settings of robotics problems, learning a high-performance reward function from only handful of noisy user generated binary labels seems hopeless [12]. Consequently, methods from active learning maximize feedback efficiency by attempting to ask the most informative queries with simplistic or linear reward models [13, 14]. The constraints these methods place on the reward function class make them unable scale to complex domains that necessitate expressive reward models [15]. Moreover, such methods are not significantly more data efficient than random sampling in practice [16, 17]. On the other hand, recent works using general function approximators still require thousands to tens-of- of artificially labeled queries to learn sufficiently accurate reward functions [18, 19, 15]. This is far too onerous for real human labelers to provide, even in the single task setting. In order to train effective reward functions from actual humans, we need need a paradigm shift. Instead of optimizing for the most informative query, take an orthogonal perspective that maximizes the amount of overall data by leveraging pre-training on realistic multi-task settings, and fine-tuning on a small and manageable amount of human queries online.
In the multi-task setting, significantly more data is available from previously known tasks which can be used to accelerate reward function learning. In fact, the shared structure of many real-world tasks has already been shown to accelerate policy learning [20]. The same structure can be exploited to learn complex reward functions for new tasks with only a handful of queries. This is largely because most tasks have rewards that are non-trivial compositions of other tasks. For example, data collected on opening windows and drawers could help learn reward function for door-opening with fewer human queries. Our key insight is to use multi-task data in order to meta-learn reward functions for preference based RL. Pre-training reward functions on large dataset enables them to quickly adapt to new preferences with only a handful of queries.
Our core contributions are as follows. First we introduce method for efficiently training RL policies from human-feedee feedback using a meta-learned reward function. Second, we demonstrate its effective- ness across number of standard robotics benchmarks, reducing query usage by factor of 20 on robotic benchmarks in comparison to previous state-of-the-ar methods. This increase in efficiency allows to learn manipulation policies from real human feedback unlike prior work. Finally, we demonstrate the effectiveness of our method in the real-world using a Franka Panda robot.
2 Related Work
Our work builds on top of a number of prior works spanning RL, preference-leariing, and meta- learning. Here we review the areas most relevant to our method.
Reward Learning. As hand-designed reward functions are difficult to tune, casily mis-specified [3, 21], and challenging to implement in the real world [2, 22]. many recent works have lever- aged human-collected data in order to learn reward functions. A large body of work focuses on using inverse RL, where a reward function is learned from approximately expert human collected demonstrations [23, 24, 25, 26]. However, demonstration collection is often expensive [27, 9, 28, 10, 29] and collected demonstrations are sometimes not even aligned with true human preferences [30, 31, 32]. Alternative strategies for learning reward functions utilize physical corrections [33], natural language instructions [34], human-provided scalar scores [35, 36) or partial [37] complete [38, 39] rankings While physical corrections and language may be easier for the user, it is generally unclear how they translate to reward updates. Stronger signals are provided by scalar scores or multiple rankings, but they are harder for users to provide [40]. We thus use pairwise comparisons as they are the simplest and generally refer to this approach as preference learning. Many recent works have studied active preference-basee learning from human feedback, however such approaches often make restrictive assumptions of the reward function, like linearity in predefined features [13, 41, 42, 14, 43]. These assumptions make such methods too inexpressive to scale to modern robot learning with complex objectives [14]. While recent methods combining preference learning with deep RL make no assumptions on the structure of the reward function, they are far too feedback inefficient to be effectively used by humans [15, 18, 44, 45, 46). Other works that use preferences with deep imitation learning [47] still require demonstrations. Most related to our work, PEBBLE [18] combines the SAC off-policy RI. algorithm [48] with an ensemble
2

---

Pre-training
AdlipttAt
Prior Tasks
Segments o, d2
Replay Buffer
Open
Vildow Open
Blutioon Press
(s,a,s")
s')
(s,(),
Tip
y
TI (als)
Drawer Opan
Segments 02
New Task
Figure 1: An overview of our method. Pre-training (left): In the pre-training phase we generate trajectory segment comparisons using data from a family of previously learned tasks and use them to train a reward model. Online-Adaptation (Right): After pre-training the reward model, adapt it to new data from human feedback use to train a policy for a new task a closed loop manner.
of learned reward functions for sampling informative comparisons. Unfortunately, PEBBLE still requires an impractical number of queries to learn just a single task (25k for drawer opening). Distinct from prior work, we consider the more realistic multi-task setting that enables us to tap into a large amount of diverse data for for pre-training to increase query-efficiency.
Meta Learning. Meta-learning, methods [49, 50] address the few-shot learning problem, where pre- dictions on new tasks are made with a limited amount of data. Inspired by their success in supervised learning problems, adopt the MAML algorithm [49] for learning new reward functions based on a limited number of human queries. Though supervised meta-learning has been previously used to infer reward classifiers [8] or in learning from heterogeneous demonstrators [51] to our knowledge it has not been applied to reward learning from preferences. Instead of adapting the reward function to new tasks, other related work in meta-RL directly adapts the policy network after few exploratory episodes [52, 53, 54, 55J. As the RL problem is much more difficult than supervised reward learning, policy adaptation approaches are likely to be less query efficient.
3 Few-Shot Preference Learning for RL
In this section formally describe the problem of meta-learning for preference based RL, then detail how our algorithm leverages multi-task pre- pee-training for online few-shot adaptation.
Problem Setup. In standard RL, an agent maximizes its cumulative expected reward in Markov decision process (MDP). Unlike standard RI, we assume the reward function r(s, a) to be unknown and instead must be estimated from human feedback. Distinct from prior works in preference based RL, we focus on the multi-task regime and thus additionally assume the existence of a distribution of tasks p(T). Each task corresponds to unique MDP where the state space S, action space A, and discount factor y are held constant, but the unknown ground-truth reward function r(s, a) and sometimes transition function P, vary. Thus, write that T; (Pt, p(T).
Within this setting, we define the few-shot preference haasred RL problem. Given access to a dataset of N previous tasks, {T: J.V the agents goal is to learn a policy Tnew (a|s) for new task Tnew p(T) from human feedback with as few user queries as possible. We make no explicit assumption on the form of prior data for each of the N prior tasks, only that it contains sufficient information to learn an estimate the reward r,. After the pre- pre-training phase, depicted in the left half of Figure 1, learn policies from online human feedback (right half of Figure l). This setting is significant departure from past work in preference-basee
RL, as we do not assume that new tasks are learned in isolation. More realistically, there are multiple tasks that have been completed within the same state and action space. Next we explain the major components of our approach.
(a)w(a)s) Preference Learning. In order to learn the policy Tnow (als) 3) for a new task from human preferences, we choose to learn the new tasks' reward function Tnew (8,a), a). While alternative approaches might Tnew using human feedback, such meta-RI. style approaches
seek to directly adapt the policy often entail the difficult optimization challenges known to plague policy gradients and dynamic pro- gramming [56]. Instead, we directly model the reward using supervised learning techniques. We

---

denote (s,(s, to be a learned estimate of an unknown ground-trute reward function r(s, a), pa- rameterized by 1.. As in Wilson al. [57] we consider preferences over partial trajectory segments (st, Ct, 81+1; 3 & 11) of k states and actions, as they provide more infor- mation than single states [57, 15]. We then define a preference predictor over segments using the Brarley-Terry model of paired comparisons [58]:
11,
exp Po a!) Pl > 02] exp E, (s), at) + exp E, (s at)
In the above, 0i d2 indicates the event that segment 1 is preferred to segment 2, as shown in Figure 1. For a given dataset D comprised of labeled queries (a 02 y) where y = {1. 2} corresponds to whether di or 02 is preferred, we optimize the following objective to learn Pt..
Cprer(ts, D) -oo ()) (1) log(Plai > o2)) + v(2) log(1 Ploi > o2]]
(I)
In practice, this is just the standard binary cross-entropy objective where logits are determined by the sum of the learned reward function over &o timesteps. Intuitively, this objective seeks to maximize the logits, and consequently predicted reward values, of the preferred segment in comparison to the unpreferred one.
Pre-training for Preference Learning. To es- timate the reward function of new task Tnew in as few queries as possible, we want to pre-train reward function iee that can quickly adapt to new tasks with only a handful of compar-
Algorithm Few-Shot Preference-based RL
Require: Teacher freq K, Queries per session M I: 3b t arg min Σ, C (v a v L(b, D,), D2)
2: for = 1,2, 3, do
isons (01, 22 y). Tapping into offline data can help exploit shared task structure and poten- tial accelerate learning on new tasks. We pro- pose extending the meta-learning framework to preference learning across different tasks. Our approach is agnostic to the choice of meta- learning algorithm, but we choose Model Ag- nostic Meta-Learning (MAML) [49] for its sim-
3:
if 1%t == 0 then
4:
for m 1..,...M MM do
5:
(01 02) ~ Disagreement
6:
y k user preference
7:
D + Dnew U (0), 02, v)
8:
end for
9:
v/ +- 1/ Re-initialize reward model
for each gradient step do
plicity. Concretely, MAML searches for param-
11:
v & a v Epeer(ti Duew)
eters 1/ that attain high performance on a new
12:
end for end if
task after only a few gradient steps by train- 13:
ing set of previous tasks. In setting,
14:
Collect 8t+1 by taking as ~ a (st)
data for previous tasks can come from offline 15:
Store transition B < BU (sc. (sc.01, 01, S1+1)
16:
Sample batch {(se, ae, $141)] ~ B
datasets, simulated policies, actual humans.
In conjunction with our preference loss from Equation (1), we use the following pre-training update:
17:
Assign rewards r + rwr (st, at)) Optimize via SAC with ((Se, Gr 8441, To (8t, a, //
18:
19: end for
N
2b - - OV Σ
(V) D.).
(2)
Here and 3 are the inner and outer learning rates respectively. Each dataset D; is comprised of known queries for each of the N tasks T; ~ p(T). When we start training for a new task, we can quickly adapt the reward function using the new queries v/ + so - a (p Dnow). ). As si is explicitly optimized for performance on Lpeet after only a handful of updates, we significantly reduce query complexity.
Training i.s using Equation (2) however, requires access to query datasets D; for each task. While pre-training can be accomplished through several objectives, like reward regression, we use preference-based pre-training for consistency and its generality. Pairwise comparison data can be extracted from wide variety of sources. If reward values are present in offline data, artificial la- bels y for trajectory segments Ti 02 can easily be generated via the comparison Σ. r(s) at) > Σ. r(s,, af) as common practice in prior works [15, 18]. If reward values for previous tasks are unknown but policies are, reward values can be recovered via inverse-RI, or comparisons can be derived from direct behavior comparison. For example, when generating queries for task i, behav- iors from (als) would be preferred to behaviors generated from TTA (als). The left half of Figure

---

Train Tasks
Reach
Sweep
Dooe Open
Insert
Window Open
Drawer Close
Basketball
Pick Place
Button Top
Test Tasks
Window Close,
200
Door Unlock, 500
1.25
1.2
1.2
1.00
1.0
S000665 Rato
0.8
0.75
0.4
0.4
0:25
0.2
0.2
0.00
0.0
0.0
-0.2
0.2
100000
200000
300000 400000 500000
100000
200000
200000 400000 500000
0.0
0.2
0.4
0.6
0.8
L.0
Button Press, 500
Drawer Open, 1000
Sweep Into, 2500
1.25
1:25
1.00
Su0085s Rata
0.75
0.50
www
0.25
0.25
0.25
0.00
-0.25
Environment Steps
1e6
Environment Steps
1e6
Environment Steps
1e6
SAC
Pow-Shot.cr (Ours)
PEBBLE
Inil
Figure 2: Results MetaWorld tasks. The title of each subplot indicates the task and number of artificial feedback queries used training. Results each method are shown across five seeds.
shows the process of extracting query data from offline data for pre-training, which corresponds to line in Algorithm 1. In our experiments, we use the artificial reward labeling scheme described first for consistency with prior work [15].
Few-Shot Preference-
RL. Our pre- pre-trained preference function can then be used for few-shot
preference based RL during online adaptation phase, depicted in the right half of Figure 1. We modify the standard Soft-Actor Critic RL algorithm [48] to relabel transitions using our learned reward function before performing standard actor-critic update (Algorithm lines 17-18). Every K steps, we ask a user to answer queries and provide feedback labels as shown in lines 5-7 of Algorithm 1. Informative queries selected using the disagreement of ensemble of reward functions over the preference predictors. Specifically, comparisons that maximize std( P(a. > Ø2)) selected each time feedback is collected [59]. After new feedback collected, re-initialize the reward model f. to its pre-trained weights. Subsequently. we re-adapt it using the updated dataset Daow for the new task as shown in Algorithm lines 9-11.
To our knowledge, we are the first to leverage multi-task data for preference-based RL. The shift to the multi-task setting necessitates critical algorithmic changes in comparison with prior work. First, we pre-train the reward function from prior data instead using other warm-start methods like unsupervised exploration used in PEBBLE. Second, we crucially reset the reward model for adaptation. Our setting provides a novel framework that leverages pre-training on range of tasks for data-efficient adaptation any structural assumptions on the reward function.
tasks enabling human users to provide this data without making
Experiments
In this section we seek to answer following questions: First, does few-shot preference learning improve the query efficiency of preference-based RL. Second, is our method efficient enough to learn robot policies from real human feedback? Finally, can few-shot preference learning be used in the real world? Dataset, architecture, and hyperparameter details are available in the Appendix.
5

---

4.1 How query-efficient is few-shot preference-based RL?
To test the query-efficiency of few-shot preference based RL for realistic robotic tasks, we adopt the Sotad benchmark from Yu et al. [20]. Agent tasks include household activities like opening doors or closing windows, and standard manipulation problems like block pushing. Some learning they sequential:
Meta-World tasks are particularly difficult for samman-nn-the-oon feedback on the second part of the task, like where an agent should move a block, can only be provided once the agent learns the first part of the task, like how to grasp a block. Additionally, different objects introducing different manipulation dynamics across tasks. To evaluate the wwwperformance of our approach, we use the artificial queries induced by the task ground truth reward function. Previous works in preference based RL have required up to fifty-thousand artificial queries in order to solve some of the Meta-world tasks [18]. Our approach generally achieves the same performance using 20x fewer queries. Our reward models are pre-trained using only 10 prior tasks and evaluate query-efficiency on six previously unseen tasks. We compare our method, which we refer to as Few-Shot, to three basclines:
1. SAC: The Soft-Actor Critic RL algorithm trained from ground truth rewards. This repre- sents "oracle" performance.
2. PEBBLE: The PEBBLE algorithm from Lee et al. [18], which does not use any prior data.
3. Init:
This baseline demonstrates the importance of our adaptation procedure during train- ing. Instead of re-adapting the reward model each time new feedback is collected, initialize the reward model with the pretrained weights, and then perform standard updates with the Adam optimizer [60] in PEBBLE.
For each environment, we reduce the total feedback budget by factor of 20 in comparison to the maximum value used in PEBBLE. Full results are shown in Figure 2. Overall, we find that despite the 20x reduction in feedback our method is able to solve almost all of the tasks with a near 100% success rate. In the Appendix, we directly compare to Lee et al. [18] with using their amount of feedback. In all tasks, except Button Press, we achieve the same asymptotic performance as SAC with 20x less feedback than originally used for PEBBLE in Lee et al. [18] which is unable to learn meaningful policy under reduced feedback budget. In Appendix A we directly compare to PEB- BLE with the feedback schedules from Lee et al. [18]. While the Init baseline generally performs better than PEBBLE, it still falls short of our method, indicating that re-adaptation is important. Unlike in other pretraining and finetuning paradigms, preference learning is done online, causing the optimal reward function induced by the data to shift. Re-adapting weights each time feedback is collected ensures that we get the full benefits of MAML by considering all data points. Locomotion experiments and ablations on feedback and query selection are included in the Appendix.
4.2
Can few-shot preference learning be used with humans?
While no human could be sensibly be expected to provide thousands of pieces of feedback, around a hundred or less not too daunting task. Given the lower query-complexity of few-shot preference- based RL, we use it to learn complex robot manipulation policies from real-human feedback for the first time. In the process of doing so, we encountered a few challenges. First, humans often have a difficult time answering queries asked by preference based RL algorithms. Queries sampled by maximizing disagreement across an ensemble of reward functions often look identical to humans. Such queries at the margin may be maximally informative, but are more difficult to answer (See Figure 5). For example, it is unlikely that humans can accurately compare two behavior segments that only have slight variations in the robot joint positions. While this is not explicitly examined in prior work that largely uses artificially generated queries, it is important when considering the abilitics of humans and our desired to adapt reward functions with a handful of data points, making everything more sensitive errors.
To address this, we add the ability for human users to "skip" difficult queries instead of providing noisy answers and increased the number of uniform queries used to reduce the likelihood that difficult queries were presented. Second, despite these mitigations, humans still make mistakes in labeling resulting in query labels that are possibly inconsistent. We thus allowed policies to train for longer periods of time between feedback sessions in hopes of collecting more data on the current policy's 's belief over the reward.

---

Point, Mass,
Reacber,
Window
Close, 100
1:25
1.0
-25
1.00
-200
Suocess Rate
Reward
-800
-75
0.4
0.50
SAC
-000
0.25
Pew-Shar
PEBBLE
-125
50k 100k
150k
200k 2503
SOL 100%
150k 200k 250k
100k
2008
300k
100%
2o0k
300k
Environment Steps
Environment Steps
Environment Steps
Environment Steps
PEBBLE
Correct
Incorrect
Slep
Figure 3: Results on training DM Control and Meta-Worke tasks from real human feedback. The vertical dashed black line indicates the point at which feedback was stopped. The horizontal bars at the bottom show the proportion of times users provided feedback that "correctly" agreed with the tasks ground-truth reward function, "incorrectly" disagree with the ground-trume reward function, or skipped the comparison.
After making the aforementioned changes, we examined the performance of few-shot preference- based RL on two of the MetaWorld environments and two additional environments based on the DM Control benchmark [61]. J. We take the point mass and reacher environments from DM Control [61] and change the reward function to be the negative L.2 distance are pre-trained on random data and evaluated on unknown goal positions. The MetaWorld environ- ments are described in Section 4.1. Our full results are shown in Figure 3. As the ground-truth reward value for DM control correspond to the cumulative distance to the goal, the higher reward values of our method indicate that it can better communicate the human's 's objective with fewer queries. While PEBBLE was completely unable to solve Window-Close from human feedback, made non-trivial progress on Door Close. This is likely because Door Close can be trivially solved by slamming the robot arm into the door instead of first grasping the handle and then closing the door is encourage by our deward-functio ronments, users have an easier time answering queries from our method in comparison to PEBBLE. This likely because the reward function prior guides agents towards interacting with objects, leading to more distinguishable behaviors. Results with more users on Reacher are in Appendix A.
unknown goal. Reward models
prior. Moreover, we find that in the World envi-
Reach (.55, .35)
Block Push (.35, 33)
Goal Position
(.45, -3)
(0.35, -0.3)
Few-Shot
.061 + .041
.056 + .009
.188 = .175
.056 + .035
PEBBLE
.105 + .05
.129 + .067
.280 = .065
.173 + .097
1G
Table I: Results for the real-robot tasks. Performance measured in
meters to the desired goal position, lower is better. The targets for reach were 0.125 and 0.25, respectively. Results are averaged across multiple initial environment configurations. Best method is bolded.
Figure 4: Push rollouts
4.3 Can Few-Shot preference-based RL be used in the real world?
Finally, we investigate the use of few-shot preference-baseee RL in real world settings using a Franka Panda Robot. We design two basic tasks: reaching and block pushing where the robot moves its arm or the block, respectively, to an unknown goal location communicated only via the learned reward function. We pre-train reward models with artificial queries and learn policies in simulation. We then transfer the learned policies to the real world and test on unseen goal locations. Table I contains our results. Performance is measured in meters to the true goal. Again, few-shot preference learning consistently outperforms PEBBLE despite the large sim-to-real gap. One additional benefit of our approach in real-world settings that it potentially requires less instrumentation, as measurements previously needed to functionally compute reward are not required when using human feedback.
7

---

Fasy
Pasy
1
9
X
Difficult
Difficult
62
Figure 5: Examples of queries asked on the Meta World Door-Close and DM Control Reacher envi- ronments. each figure the top segment, ai had higher cumulative reward.
Discussion
Limitations and Future Work. While few-shot preference learning has several benefits, it has its limitations. Here we list the most salient ones, and possible means of overcoming them:
Query Complexity. Despite our gains in query-efficiency, the most complex tasks still require more feedback than we would like. Future work could examine how to expand the of ppe-training data.
Pre-truining Methods We investigate pre-training with artificial query data due to its generality, though method could be used in combination with other pre-training objectives, like direct ward regression, to further boost performance.
Pre-training Data. While meta-learning methods have proven to be somewhat robust changing dynamics in the real world [62], the efficacy of reward adaptation under larger distribution shifts induced by sub-optimal users, new tasks, sim-to-real transfer remains in question. For example, a new task significantly out of distribution, we would expect training a reward function from scratch to perform better than adapting. Furthermore, pre- pre-training can occasionally over-regularize the learned reward model, as exhibited in the Door Close experiment in Section 4.2.
Query Difficulty. Many queries asked by preference learning algorithms are too difficult for humans answer, as shown in Figure 5. fact, we find Section 4.2 that active query schemes often result in queries that too difficult for users to answer. Future work should explicitly consider how easy it is for a human to answer a query and not just its theoretical information content.
User Inconsistency. Unlike reward oracles, humans will inconsistently label queries. This challenge only exacerbated when attempting to crowd source data from many users with differing styles. Future work can investigate additionally modeling human users.
Conclusion. We shift the paradigm of human-in-the-loop RL from the single-task to the multi- task setting, unlocking additional data sources that can be used to boost the query-efficiency of preference-based RI. Algorithms. We believe our work's change in perspective be crucial step- ping stone towards training robots with human feedback. Our novel few-shot preference- bassed RL method is able to effectively minimize the number of human queries required train com- plex manipulation policies as demonstrated by our 20X improvement on standard benchmarks and effectiveness at real-human training.

---

Acknowledgments
This research was supported by NSF (1849952, 1941722, 2218760) ONR, and Ford. JH sup- ported by the Department of Defense (DoD) through the National Defense Science Engineering Graduate (NDSEG) Fellowship Program.
References
[1] V. Mnih, K. Kavukcuoglu, Silver, A. Graves, Antonoglou, D. Wierstra, and Riedmiller. Playing atari with deep reinforcement learning. arXiv preprini arXiv: 13/556002 2013.
[2] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine. The in- gredients of real world robotic reinforcement learning, In International Conference on Learn- ing Representations, 2020.
[3] D. Hadfield-Menel) Milli, P. Abbeel, Russell, and A. Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
[4] A. Pan, K. Bhatia, and Steinhardt. The effects of reward misspecification: Mapping and mit- igating misaligned models. In International Conference on Learning Representations, 2022. URL / /owprevieeeeeeeeeeee wwtprriwwweeeeeeeeeeeeeeeeeeeeee wwweeeeeermeaeeeeeeeeeeeeeeeeeee -o-ormmmid--ywwwi177777
[5] T. Brys, A. Harutyunyan, H. B. Suay, S. Chernova, M. E. Taylor, and A. Nowé. Reinforcement learning from demonstration through shaping. In Twenty-fourth international joins conference on artificial intelligence, 2015.
[6] Y. Wu, M. Mozifian, and F. Shkurti. Shaping rewards for reinforcement learning with imperfect demonstrations using generative models. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6628 6634. IEEE, 2021.
[7] H. Jeon, S. Milli, and A. Dragan. Reward-rational (implicit) choice: unifying formal- ism for reward learning, Advances in Neural Information Processing Systems, 33:4415-444 44266644444 2020.
[8] A. Xie, A. Singh, S. Levine, and C. Finn. Few-shot goal inference for visuomotor learning and planning, In Conference on Robot Learning, pages 40-52, PMLR, 2018.
[9] B. Akgun, M. Cakmak, Jiang, and Thomaz. Keyframe-basse based learning from demon- stration. International Journal of Social Robotics, 4(4):343-335 3355,,555,51 2012.
[10] D. P. Losey, K. Srinivasan, A. Mandlekar, A. Garg, and D. Sadigh. Controlling assistive robots with learned latent actions. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 378-384. IEEE, 2020.
[11] L. Smith, N. Dhawan, M. Zhang, Abbeel, and Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. RSS, 2020.
[12] Wright and Y. Ma. High-DDmensional Data Analysis with Low-Dimensional Models: Prin- ciples, Computation, and Applications. Cambridge University Press, 2022.
[13] D. Sadigh, A. D. Dragan, S. S. Sastry, and S. A. Seshia. Active preference- learning of reward functions. In Proceedings of Robotics: Science and Systems (RSS), July 2017. doi: 10.15607/RSS.20 201..X11.053..
[14] E. Biyik, N. Huynh, M. J. Kochenderfer, and D. Sadigh. Active preference-based gaussian process regression for reward learning. In Proceedings of Robotics: Science and Systems (RSS), July 2020.
[15] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.

---

[16] S. Karamcheti, R. Krishna. L. Fei-Fei, and C. D. Manning, Mind your outliers! investigat- ing the negative impact of outliers on active learning for visual question answering. CoRR, 00122107702331, 2021. URI. https: peetps://arxtee /awxiv arggabs/2100 2ttp//2007.00231.1.1
[17] D. Lowell, Lipton, and C. Wallace. Practical obstacles to deploying active learning- arXiv preprini arXiv: (00.::/807..40..2000 2018.
[18] K. Lee, L. Smith, and Abbeel. Pebble: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In International Conference on Machine Learning, pages 6152-6163, PMLR, 2021.
[19] K. Lee, L. Smith, A. Dragan, and P. Abbeel. B-pref: Benchmarking preference-based inforcement learning. In Fhirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL httthhtttt///eeeeeeeeetee id=ps95-mkil
net/forum?
[20] T. Yu, D. Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019, URL pettps://arxtar wwwpeiiii.o.///booo httpg//a////// /9100. 10897.
[21] A. Turner, N. Ratzlaff, and P. Tadepalli. Avoiding side effects in complex environments. Ad- vances in Neural Information Processing Systems, 33:21406-21415, 2020.
[22] P. Kormushev, S. Calinon, and D. G. Caldwell. Robot motor skill coordination with em-based reinforcement learning. 2010 JEEE/RSJ international conference on intelligent robots and systems, pages
IEEE, 2010.
[23] Abbeel and A. Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Pro- ceedings of the twenty-first international conference on Machine learning, page 1, 2004.
[24] D. Ramachandran and E. Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2586-25665 2591, 2007.
[25] B. D. Ziebart, A. L. Maas, A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[26] D. S. Brown, W. Goo, and S. Nickum. Better-than--dmmonsttttoo imitation learning via automatically-raneed demonstrations. In Conference robot learning, pages 330-359. PMLR, 2020.
[27] R. P. Khurshid and K. J. Kuchenbecker. Data-driven motion mappings improve transparency in teleoperation. Presence, 24(2):132-15,,22 2015.
[28] A. D. Dragan and S. S. Srinivasa. Formalizing assistive releoperation. MIT Press, July, 2012.
[29] D. P. Losey, H. J. Jeon, M. Li, K. Srinivasan, A. Mandlekar, A. Garg, J. Bohg, and D. Sadigh. Learning latent actions to control assistive robots. Autonomous robots, 46(1):115-145 2022.
[30] C. Basu, Q. Yang, D. Hungerman, M. Sinahal, and A. D. Draqan. Do you want your au- tonomous car drive like you? In 2017 12th ACM/IEEE International Conference Human- Robot Interaction (HRI, pages 417-425. IEEE, 2017.
[31] M. Kwon, Biyik, A. Talati, Bhasin, D. Losey, and Sadigh. When humans aren't op- timal: Robots that collaborate with risk-aware humans. In 2020 15th ACM/IEEE International Conference on Human Robott Interaction (HRI), pages 43-52. IEEE, 2020.
[32] L. Chen, R. Paleja, and M. Gombolay. Learning from suboptimal demonstration via self- supervised reward regression. arXiv preprint arXiv:2010. 1723, 2020.
[33] M. Li, A. Canberk, D. P. Losey, and D. Sadigh. Learning human objectives from sequences of physical corrections. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 2877-2883, IEEE, 2021.
10

---

[34] J. D. Co-Reyes, A. Gupta, S. Sanjeev, N. Altieri, J. Andreas, J. DeNero, P. Abbeel, and Levine. Guiding policies with language via meta-learning. In International Conference on Learning Reprexentations, 2019.
[35] W. B. Knox and P. Stone. Interactively shaping agents via human reinforcement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9-16, 2009.
[36] W. B. Knox and P. Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning, pages 292-297. IEEF, 2008.
[37] V. Myers, F. Biyik, N. Anari, and D. Sadigh. Learning multimodal rewards from rankings. In Conference Robot Learning, pages 342-352, PMLR, 2022.
[38] D. Brown, W. Goo, P. Nagarajan, and S. Nickum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. In International conference on machine learning, pages 783-792. PMLR, 2019.
[39] E. Biyik, D. A. Lazar, D. Sadigh, and R. Pedarsani. The green choice: Learning and influencing human decisions on shared roads. In 20/9 IEEE 58th conference on decision and control (CDC), pages 347-354. IEEE, 2019.
[40] G. A. Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):81, 1956.
[41] X. Wang, K. Lee, K. Hakhamaneshi, Abbeel, and Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1259-1268. PMLR, 08-11 Nov 2022. URL https: /ppoceedingg. mlr. www.epreeeeeeeee 21114//aagg22 html.
[42] M. Cakmak, S. S. Srinivasa, M. K. Lee, J. Forlizzi, and S. Kiesler. Human preferences for robot-human hand-over configurations. In 2011 IEEE/RSJ International Conference on Intel- ligent Robots and Systems, pages 1986-1993, 2011. doi: 00110/000.00011....0
[43] R. Lepird, M. Owen, and M. J. Kochenderfer. Bayesian preference elicitation for multi- objective engineering design optimization. Journal of Aerospace Information Systems, 12(10): 634-645, 2015.
[44] J. Park, Y. Seo, J. Shin, H. Lee, P. Abbeel, and K. Lee. SURF: Semi-supervised reward learning with data augmentation for feedback-efficien preference-basee reinforcement learning. In International Conference Learning Representutions, 2022. URL httpps://ppenreriier. net/forumiid: 2d=-Tff22L222000
[45] X. Liang, K. Shu, K. Lee, and P. Abbeel. Reward uncertainty for exploration in preference- based reinforcement learning. International Conference on Learning Representations, 2022.
URL https:://pperreviei wttt://orummm//--0/VD----
[46] G. Warnell, N. Waytowich, V. Lawhern, and Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[47] D. S. Brown and S. Niekum. Deep bayesian reward learning from preferences. arXiv preprint arxiv:1911..44722.201 2019.
[48] T. Haarnjja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[49] C. Finn, P. Abbeel, and Levine. agnel-ag meta-learning for fast adaptation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.

---

[50] A. Nichol and J. Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.02999 2(3):4, 2018.
[51] M. L. Schrum, E. Hedlund- Botti, and M. C. Gombolay. Personalized meta-learning for domain agnostic learning from demonstration. In Proceedings of the 2022 ACM/IEEE International Conference on Human- Human-Robot Interaction, pages 1179-1181, 2022.
[52] X. Wang, Z. Kurth-Nelson, H. Soyer, J. Z. Leibo, D. Tirumala, R. Munos, C. Blundell, D. Kumaran, and M. M. Botvinick. Learning to reinforcement learn. In CogSci, 2017. URL https: http:///mmmmmmmeemememm
org/ http://ogcc//////////o/ pwtppe///5//////aaaapppppppps ii///x.ddxxxe/// html.
[53] R. Agarwal, C. Liang, D. Schuurmans, and M. Norouzi. Learning to generalize from sparse and underspecified rewards. In International Conference on Machine Learning, pages 130- 140. PMLR, 2019.
[54] Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. 212: Fast reinforce- ment learning via slow reinforcement learning. arXiv preprint arXiv: 16111.02222 1.02779, 2016.
[55] K. Rakelly, A. Zhou, Finn, S. Levine, and D. Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In International conference ma- chine learning, pages 5331-5340. PMLR, 2019.
[56] W. B. Powell. Perspectives of approximate dynamic programming. Annals of Operations Research, 241(1):31--356, 2016.
[57] A. Wilson, A. Fern, and P. Tadepalli. A bayesian approach for policy learning from tra- jectory preference queries. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Wein- berger. editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012. URL https://ppet sttps://proceedingg inw.rrcceddnnn.eeeiipp.s......pp nwwrips wttppppeee/0/2/ffllel.... 16c222a11988000888111777777 aac.:9/8...ccabbbcc7Paaa.r.... pdf.
[58] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[59] C. Daniel, O. Kroemer, M. Viering, J. Metz, and J. Peters. Active reward learning with novel acquisition function. Autonomous Robots, (9(3):3899400) 2015.
[60] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. URL http://arxiv org/ abs//412... 1411.6980.
[61] S. Tunyasuvunakool, A. Muldal, Y. Doron. S. Liu, S. Bohez, Merel, Erez, T. Lil- licrap, N. Heess, and Y. Tassa. dm control: Software and tasks for continuous con- trol. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi.https://oii.rr//10.0006 simpa.2020. 100022. URL. http:////wa.ccccocooooccccooccaa com//cciennc///tticll///poa $2665963820300099..
[62] I. Clavera, A. Nagabandi, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In Inter- national Conference on Learning Representations, 2019. URL https: http::::///ppeevieerr. net/ forum?id-llyztsoCSYY
12


Reviews:
[{"review_id": "npx1dv9V2VE", "reviewer": "Reviewer_FhLm", "summary": "This paper introduces a few-shot preference learning framework via meta learning. A task in this setting is represented as solving for the optimal policy of an MDP with known transition but unknown reward function. \n\nTo learn the reward function, the model assumes access to a collection of preference indications over pair of state-reward trajectories. Each indication is 0/1 signaling whether a segment of state-reward trajectories is preferable to another. The reward function can then be learned via optimizing Eq. (1) which is the cross-entropy objective corresponding to the Bradley-Terry model of paired comparison.  \n\nTo combat the scarcity of data in new task, the paper assumes access to preference data of prior (similar) tasks. A direct application of MAML on the above cross-entropy objective function thus enables learning a base parameterization for the reward function, which can be fine-tuned to new tasks with little preference data. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe paper is generally well-written. The technical exposition is easy to follow.\nClear pseudocode of the proposed algorithm is also included which is nice.\nExperiments with both simulation & hardware are presented, which has a strong robotic focus\n\nWeakness:\n\nThe paper appears to be a direct application of MAML on an existing preference learning model but it is not quite clear what are the non-trivialities in doing so?\n\nFrom a learning perspective, MAML is a model-agnostic recipe that can be applied to any loss functions that are optimizable via gradient descent/ascent so suppose that preference data is available for a range of relevant tasks, I tend to think that if there is nothing else, then the application here is straight-forward. \n\nThe extensive experiment of course requires significant effort but it does not seem to add much insights to the literature as the empirical questions it addressed are pretty much the same set of questions that were raised and reasonably addressed in MAML (as a general principle to pre-training) and its follow-up variants with various applications (robotic applications included).\n\nI do notice an interesting point (4.2) about few-shot learning with human feedback. This might be a new aspect that has not been explored much but I do not see how this is connected to the technical formulation. In fact, the narrative in 4.2 is mostly about the potential noisiness of human feedback and engineering feature to allow human to skip on hard queries. \n\nIt might be more interesting to develop learning policies that seek human feedback in an optimal way, e.g. how to structure queries that are less ambiguous to human & given limited interaction budget, what are the most informative queries; and whether such proactive query strategies would reduce the sample complexity & etc.         \n\n--\n\nThese are my current understanding of this paper (from a learning perspective) but please do correct me if I miss anything important. I will be happy to discuss.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper seems to be a direct application of MAML on an existing preference learning model. From what was presented in the main text, I unfortunately do not see much new insights (as MAML is a generic model-agnostic recipe for pre-training). \n\nThe application with real robotic scenario is nice & interesting but on its own, it is only a re-confirmation to the broad applicability of MAML with no new insight. With this, I believe this paper does not quite meet the novelty bar for publication.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I think the main issue with this paper is that it has not highlighted the non-trivialities that require new learning capabilities to be innovated to enable the application of MAML. It would be good if the authors can focus on this while preparing the rebuttal."}}, {"review_id": "g0aYecatBma", "reviewer": "Reviewer_m37o", "summary": "This paper studies preference-based RL in a setting where the agent is also given to an offline dataset of preference/reward data from other tasks. The main idea is to use the offline dataset to meta-learn the preference model, which is then adapted on the target task. The motivation is that such meta-learning should significantly decrease the number of human queries required to solve the target task. Empirically, the proposed method outperforms 2 baselines (vanilla finetuning, no access to offline data) on 6/6 tasks. Additional experiments show that the method can work from real human feedback, and that the learned policy can be adapted to work on a physical robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* The paper is generally well written.\n* To the best of my knowledge, the proposed method is novel, and seems to address an important limitation with preference-based learning.\n* The results show strong gains relative to the prior method (PEBBLE).\n\nWeaknesses\n* My main concern is with the central premise of the paper: that preference-based RL is an effective way to learn tasks. While there is a large and growing body of work that studies this problem setting, I am unaware of convincing prior work that shows that it is easier/cheaper for humans to provide preferences than other sorts of supervision. For example, in the DM Control tasks described in L251, specifying a goal (or, equivalently, reward function) requires just 2 numbers. I would expect that specifying the goal location wouldn't take more than a few seconds, of similar complexity to specifying a preference comparison. Similarly, doing back-of-the-envelope calculations for the results in Fig 2: The tasks can all be described by a hand + object goal state (3 + 3 dimensions, sometimes less). If we want 1/16 resolution, we need 4 bits / coordinate, for a total of 24 bits of supervision. In contrast, the proposed method uses at least 10k / 20 = 500 bits of supervision (21x).\n* I found the presentation of the results in Fig 2 a bit confusing because the \"20x\" improvement isn't visualized anywhere. If possible, it'd be great to make a plot where one axis directly shows the 20x improvement (e.g., plot reward vs number of queries)\n* The real-world results are a bit lackluster because, if I understand correctly, they aren't actually obtained by querying human users with real robot images but rather are obtained through sim2real. While this isn't an issue for most robotics papers, it does seems like a missed opportunity for a paper studying how robots should learn from humans in the real world.\n* A number of minor writing comments (e.g., claims without citations attached). See below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this paper is strong in many respects: it presents a novel method, is well written, and shows compelling empirical results. The paper does a great job delivering on the stated title (\"few-shot preference learning for human-in-the-loop RL\"). My main concern with the paper is that I'm not convinced that preference based learning is an effective method for soliciting human feedback. If the task can be described as reaching a simple goal state, it seems easier to ask the human to provide that goal state than to provide O(100s-1000s) preferences. While this paper doesn't purport to show that preferences are more efficient to provide than reward functions or goal states, I think that studying this question is critical for demonstrating the *relevance* of work; without this, it seems like this paper is a great contribution to an irrelevant problem. Concretely, there are a number of ways the paper might study the effectiveness of different sorts of supervision. One easy method would be to quantify how many seconds it takes a human to provide a goal state versus a preference, assuming that a reasonable UI is used for both. If the paper can show that preference-based learning is more effective than alternatives (e.g., manually-specifying the reward function or goal state), I would advocate for accepting the paper.\n\n------------------------\n**Update after rebuttal** -- The discussion period clarified and resolved most of my concerns about the paper. The additional experiments seem to provide good evidence for the claim that the proposed method can tackle tasks that are hard to define in terms of rewards or success states. I now vote for accepting the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Larger issues:\n* Many of the claims in the introduction require evidence or citations (see below)\n* L160 - L164 -- I found this a bit confusing because it seems like providing a meta-learning dataset of preferences would be very, very expensive. Also, this seems a bit disingenuous, since none of the experiments actually involve meta-learning on real human preference data.\n* L195 \"paradigm shift\" -- This seems to be overclaiming a bit.\n* L249 -- L262 -- Why is a different environment used for these environments. This seems a bit suspicious, especially since the new DMC environments appear easier.\n* L264 -- L268 -- Do humans ever actually provide preference feedback on real robot experience? The \"sim2real\" note seems to imply otherwise. This seems important to clarify.\n\n\nWriting comments\n* L12 \"Contrary to ...\" -- I found this explanation very helpful!\n* \"20X\" --> \"$20\\times$\"\n* \"Deep Reinforcement Learning\" --> \"deep reinforcement learning\"\n* \"Markov Decision Process\" --> \"Markov decision process\"\n* \"inordinate\", \"galvanized\", \"unduly\" -- awkward word choice\n* L28 -- I found the discussion about door opening with sparse rewards a bit disingenuous, as many prior works can learn such tasks with sparse rewards.\n* L34 \"aim to capture\" -- Cite.\n* L37 \"their environment\" -- Cite.\n* L41 \"learned reward functions are ...\" -- Cite. (This claim seems surprising to me)\n* L46 \"seems hopeless\" -- Cite, or provide evidence.\n* L48 \"... or linear reward models\" -- Cite.\n* L49 \"necessitate expressive reward models.\" -- Cite.\n* L53 \"is far to\" --> \"is far too\"\n* L83 broken reference\n* L86 \"they are harder for users to provide\" -- Cite.\n* Fig 1 \"it\" -- Ambiguous pronoun reference\n* L90 \"These assumptions ... \" -- Cite.\n* L107 \"classification\" -- Is the implicit assumption here that reward learning is equivalent to classification?\n* L119 -- What is $\\mathcal{P}$?\n* L123 \"only that it ...\" -- At this point, I was confused about what the prior data actually contained. Was it rewards, preferences, demos? (This is answered a few paragraphs later)\n* L127 \"multiple tasks\" -- I was confused by this, as Fig 1 only seems to show adaptation to a single task.\n* L129 -- The placement of $\\pi_{new}$ makes it seem like $\\pi$ is a task, not a policy.\n* L133 \"[50]\" -- Does this 2016 paper really discuss meta-RL?\n* L133 \"model reward\" --> \"model the reward\"\n* L135 -- Why use trajectory segments instead of individual transitions?\n* L177 \"prior works\" --> \"prior work\" (or, provide more citations)\n* L210 \"difficult ... as they are sequential\" -- This seems a bit disingenuous, as most of these tasks can be solved in 10 - 20 steps.\n* L261 \"prior guides the agents\" -- It'd be great to provide evidence for this claim.\n* L445 -- L452 -- These citations have the wrong indent."}}, {"review_id": "IfKlPP_hNCA", "reviewer": "Reviewer_pAzu", "summary": "The overall idea of the paper is:\n1. Use an existing dataset to meta learn a reward function that can quickly adapt to a few comparison tuples $(\\sigma_1, \\sigma_2, y)$. The reward function is used in the preference prediction, and the underlying meta learning method is MAML.\n2. Given a new task, as well as a few comparison tuples, adapt the reward function, then perform RL. Specifically, the comparison tuples are provided every $K$ iterations, where the most \"uncertain\" segments are selected as queries for human.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths***:\n1. To my knowledge, the overall idea is novel and straightforward: meta-learn the preference is not only reasonable, but potentially important for applying preference learning to real-world problem.\n2. The \"active learning\" part also makes sense to me.\n3. The overall presentation is clear and easy to follow.\n\n***Weaknesses***:\n1. It might be good to see an ablation over $K$ or essentially how the model performs with different shots. \n2. It will also be good to see the difference of picking the $(\\sigma_1, \\sigma_2)$ that maximizes $std(P(\\sigma_1 > \\sigma_2))$ versus for instance randomly selecting two $\\sigma$s. Basically how important it is to have this particular selecting scheme.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think overall adding the meta-learning part to preference learning is a novel and important step. This is the main reason I recommend acceptance of this paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Please clarify the points I mentioned in the weakness section or provide further empirical evidence.\nIn general, I think adding more ablations on important design choices of the algorithm can much improve the paper. Minor point: Fig.2 will look better if its resolution is higher."}}, {"review_id": "GPkPwaXzwulW", "reviewer": "Reviewer_mqZu", "summary": "This paper addresses the problem of learning end-to-end reward functions from preferences. Rather than rely on hand-defined feature representations, the proposed method learns a reward function directly from high-dimensional input. To avoid requiring excessive training data, the proposed method leverages meta-learning to learn over multiple tasks and quickly adapt to a new one using a tractable number of preference queries.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The approach is novel and described clearly. \n+ The results back up the intended claims of the paper: the proposed method results in significantly (20x) fewer queries than current baselines.\n+ The proposed method is impactful in making meta-learning approaches more tractable for interactive robot systems by making use of a well-studied interaction type (i.e., preferences).\n\nWeaknesses:\n- There is very little information provided about the user study that aims to validate whether this system is practical. For example, how many queries were posed to participants? How many participants were there? What was their background and experience with robots? How much time did the interactions take? What subjective measures were collected about how \"practical\" the interaction was? How were preferences presented to users? I understand that there may not be room for all of this information in the main paper, but it should be at least present in the appendix.\n\n- Figures 2 and 3 present compelling results about the system's ability to converge during training. However, these graphs would be more interesting with a different x-axis. Rather than report performance w.r.t. # of update steps (with a fixed number of preference queries), it would be more interesting to see performance w.r.t. # of preference queries (with a fixed number of update steps). This would provide more granular information about how many queries are really needed by the system, and how much \"faster\" it learns compared to baseline (where faster is measured w.r.t. user engagement, rather than convergence speed).\n\n- The network architecture also appears to be missing from the main paper and the appendix. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, this is an interesting paper that presents techniques that will be useful to other researchers in interactive robot learning. The paper would be made even more impactful by adjusting the graphs to use a different x-axis (highlighting the relationship between # of queries and performance) and including necessary details on the approach and user study.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "I would love to see alternative graphs for Figures 2 and 3 that demonstrate the relationship between # of queries and performance, if you have them available. If this is not possible, any other discussion about this relationship would be useful.\n\nPlease do list the requested details about the technical approach and user study design.\n\nAdditional clarification:\n\"For each environment, we reduce the total feedback budget by a factor of 20 in comparison to the maximum value used in PEBBLE, but keep all other hyper-parameters the same.\"\nDoes this mean that the numbers in the Figure 2 graph titles are the number of queries used by the proposed method, but not PEBBLE?\n"}}]

Author rebuttal:

Decision: Accept (poster)