Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback
Authors: Max Balsells I Pamies
Marcel Torne Villasevil
Zihan Wang
Samedh Desai
Pulkit Agrawal
Abhishek Gupta
Venue: CoRL 2023 Poster (2023)
Keywords: reset-free reinforcement learning
learning from human feedback

Abstract: Ideally, we would place a robot in a real-world environment and leave it there improving on its own by gathering more experience autonomously. However, algorithms for autonomous robotic learning have been challenging to realize in the real world. While this has often been attributed to the challenge of sample complexity, even sample-efficient techniques are hampered by two major challenges - the difficulty of providing well ``shaped" rewards, and the difficulty of continual reset-free training. In this work, we describe a system for real-world reinforcement learning that enables agents to show continual improvement by training directly in the real world without requiring painstaking effort to hand-design reward functions or reset mechanisms. Our system leverages occasional non-expert human-in-the-loop feedback from remote users to learn informative distance functions to guide exploration while leveraging a simple self-supervised learning algorithm for goal-directed policy learning. We show that in the absence of resets, it is particularly important to account for the current ``reachability" of the exploration policy when deciding which regions of the space to explore. Based on this insight, we instantiate a practical learning system - GEAR, which enables robots to simply be placed in real-world environments and left to train autonomously without interruption. 
The system streams robot experience to a web interface only requiring occasional asynchronous feedback from remote, crowdsourced, non-expert humans in the form of binary comparative feedback. 
We evaluate this system on a suite of robotic tasks in simulation and demonstrate its effectiveness at learning behaviors both in simulation and the real world. 
Project website https://guided-exploration-autonomous-rl.github.io/GEAR/.

Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback
Max B
Marcel * Zihan Wang Samedh Desai!
Pulkit Agrawal Abhishek Gupta'
University of Washington 'aassacuusetts Institute of Technology [balsells, avinwang, samedh, abhgupta) washington edu (marcelto, pulkitp @mit. edu
Abstract: Ideally, we would place robot in a real-world environment and leave it there improving on its own by gathering more experience autonomously. How- ever, algorithms for autonomous robotic learning have been challenging to realize in the real world. While this has often been attributed to the challenge sample complexity, even sample-efficient techniques are hampered by two major chal- lenges the difficulty of providing well "shaped" rewards, and the difficulty of continual reset-free training. In this work, describe system for real-world reinforcement learning that enables agents to show continual improvement by training directly in the real world without requiring painstaking effort to hand- design reward functions or reset mechanisms. Our system leverages occasional non-expert intta------e--ooop feedback from remote users to learn informative distance functions guide exploration while leveraging simple self-supervised learning algorithm for goal-directed policy learning. We show that in the ab- sence of resets, it is particularly important account for the current "reachability" of the exploration policy when deciding which regions of the space to explore. Based on this insight, we instantiate a practical learning system GEAR, which enables robots to simply be placed in real-world environments and left to train au- tonomously without interruption. The system streams robot experience to a web interface only requiring occasional asynchronous feedback from remote, crowd- sourced, non-expert humans in the form of binary comparative feedback. We eval- uate this system tiveness at learning behaviors both in simulation and the real world. Project web- site https // guided- wewddeeepplororooooo -ot--aatotoousss rl. github........... io/GEAR/.
suite of robotic tasks in simulation and demonstrate its effec-
Keywords: Autonomous Learning, Reward Specification, Reset-Free Learning, Crowdsourced Human Feedback
Introduction
Robotic reinforcement learning (RI.) is a useful tool for continual improvement, particularly in un- structured real-world domains like homes or offices. The promise of autonomous RL methods for robotics is tremendous simply place robotic learning agent in a new environment, and see continual improvement behavior with an increasing amount of collected experience. Ideally, this would happen without significant environment-specife instrumentation, such resets, algorithm design choices (e.g. shaping reward functions). However, the practical challenges involved in en- abling real-world autonomous RL are non-trivial to tackle. While those challenges have often been chalked down to just sample efficiency [1. 2], we argue that the requirement for constant human effort during learning the main hindrance in autonomous real-world RL. Given the episodic nature
Denotes equal contribution
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

of most RL algorithms, human effort is required to provide constant resets to the system [3, 4, 5], and to carefully hand-design reward functions [6. 7, 8] to succeed. The requirement for constant human intervention to finetune the reward signal and provide resets [3], hinders real-world RL.
The field of autonomous RL [3, 5] studies this problem of enabling uninterrupted real-world training of learning systems. A majority of these techniques aimed to infer reward functions from demonstrations or goal specifications [9, 10, 11, 12], while enabling reset-free learning by training an agent to reset itself [13, 9, 5]. However, these techniques can be challenging to scale to tasks with non-trivial exploration [14].
Real World Autonomous Exploration
Guided Exploration Through Goal Seloction
Occasional Crowdsourced Comparative Feedback
Soloc Statio Ukoos to Targen Geal
Fite fead able Statee
Ho Morioodde acc
Goal Selector
Density Model
Goal
State
State
Figure 1: Problem setting in GEAR. The robot explores the world autonomously and reset-free only using cheap, occasional binary feedback from con-expert users to guide exploration. This allows for massive scaling of data experience and solving much more challenging tasks.
To enable autonomous RL techniques to scale to complex tasks with non-trivial exploration, we note that directly reaching final target goal through autonomous exploration can be difficult. How- ever, achieving a promising intermediate sub-goal can be relatively simple. This process can then be repeated until the desired target goal is accomplished, making the overall exploration process tractable, long as sub-goak are appropriately chosen. The important question becomes "Whar are promising sub-goals reach and how can we learn how reach them within autonomous RL?"
In this work, we note that a promising sub-goal is one that satisfies two criteria: (1) it is closer to the desired final goal than the current state of the system, and (2) it is reachable from the current state under the current policy. While criterion (I) challenging to estimate without actually having the task solved beforehand, in this work, we show that asynchronously provided non-expert human feedback in the form of binary comparisons can be used cheaply estimate state-goal proximity. This estimate of proximity can be paired with density model for measuring reachability under the current policy, thereby selecting promising intermediate sub-goals that satisfy both criteria (1) and (2). Our proposed learning system Guided Exploration for Autonomous Reinforcement learning (GEAR) leverages occasionally provided comparative human feedback [15, 16) for directing ex- ploration towards promising intermediate sub-goals, and self-supervised policy learning techniques [17] for learning goal-directed behavior from this exploration data. This is a step towards an au- tonomous RL system that can leverage both self-collected experience and cheap human guidance to scale to variety of complex real-world tasks with minimal effort.
2 Related Work
Autonomous Reinforcement Learning in Real World. RL has been used to learn skills in the real world through interaction with real-world environments [18, 19, 20, 21, 22, 23]. A common limita- tion is the requirement for episodic resets, necessitating frequent human interventions. "Autonomous reinforcement learning" aims to obviate these challenges by building learning systems that can learn with minimal interventions [4, 24, 5]. large class of autonomous RL methods, involve learning a forward policy accomplish task and backward policy to reset the system [24, 25, 4, 26, 27, 5]. A different class of methods [13, 28, 29, 30] views the reset-free RL problem
multi-task RL problem. These techniques typically require human provided reward [5, 24, 4, 13, 28] or rely

---

on simple techniques like goal-classifirss to provide rewards [11, 26]. These reward mechanisms fail to solve domains with challenging elements of exploration. GEAR able to learn autonomously with no manual resets, while using cheap, asynchronous human feedback to guide exploration more effectively.
To alleviate the challenges of reward function specification, build on the framework of learning from binary human comparisons [15, 31, 32, 16, 33, 34, 35, 36]. These techniques leverage binary comparative feedback from human supervisors to learn reward functions that can then be used directly with RL. methods, often model-free [37]. While these tech- niques have often been effective in learning for language models [38. 39], they have seen relatively few applications in fully real-world autonomous RL at scale. The primary challenge is that these methods are too sample inefficient for real-world use, require much human feedback [15, 35). In this work, build on recently introduced technique [40] that combines self-supervised policy learning via hindsight supervised learning [17] with learning from human feedback to allow for ro- bust learning that is resilient to infrequent and incorrect human feedback. While [40] was largely evaluated in simulation or in simple episodic tasks, we leverage insights from [40] to build RI. systems that do not require resets or careful environment setup.
Learning from Human Feedback:
Some work in robotics that rely on human feedback, scale it up by means of crowdsourcing (41,42) We show that GEAR can also work from crowdsoureed feedback by using Amazon Mechanical Turk as a crowdsourcing platform, collecting annotations in the form of binary comparisons. Note that other types of human feedback have also been leveraged in previous work, such as through physical contact [43, 44, 45], eye gaze [46], emergency stop [47]. Our method is agnostic to the kind of feedback as long as we can translate it into a sort of distance function to guide subgoal selection.
While it hasn't it been tackled in this paper due to the scarce amount of feedback needed in the tested tasks, research done in learning when to ask for human feedback [48, 49, 50, 51, 52, 53] could be leveraged in GEAR to increase efficiency in the amount of feedback requested. Similarly, previous work on shared autonomy and how to improve the understanding of the human/robot intentions [54, 55, 56] could also be applied to GEAR to allow for better use of human feedback.
foallconditioned reinforcement learning: In this work, we build on the framework of goal- conditioned policy learning to learn robot behaviors. Goal-conditioned RL [17, 57, 58, 59, 60] studies the sub-class of MDPss where tasks involve reaching particular "goal" states. The key insight behind goal-conditioned RL methods is that different tasks can be characterized directly by states in the state space that the agent is tasked with reaching. Based on this insight, a variety of self- supervised coal-conditioned RL techniques have been proposed [58, 61, 62, 63, 59, 17, 64, 65, 66] rellbbelii to learn policies. These techniques typically
that aim to leverage the idea of rely on policy generalization show that learning on actually achieved goals can help reach the desired ones. As opposed to these techniques, GEAR uses sulf-supervised policy learning [17, 61] for acquiring goal-directed behavior while relying on human feedback [67] to direct exploration.
3 Preliminaries
Problem Setting. In this work, we focus on the autonomous reinforcement learning problem [4]. We model the agent's environment a Markov decision process (MDP), symbolized by the tuple (S, A, T, R, Po. 7), with standard notation [68]. The reward function is r E R. where r S x A + R² is unknown to us, as it is challenging to specify. As we discuss in Section 4, an approximate reward f(s) must be inferred from human feedback in the process of training. As in several autonomous RL problem settings, of particular interest is the initial state distribution pu(s), which provided for evaluation, but the system is run reset-free without the ability to reset the system to initial states ~ po(s) during training [4]. The aim to learn policy . : S A. that maximizes the expected sum of rewards Ex,po Er "r((s,, (ss),,))) starting from the initial state distribution (s) and executing learned policy
3

---

Coal-Conditioned Reinforcement Learning, Since reward functions are challenging to define in the most general case, goal coal-conditioned policy learning techniques consider a simplified class of MDPs where rewards are restricted to the problem of reaching particular goals. The goal-reaching problem can be characterized as A, T, R. po: 7, 9, p(g)), with goal space & and a target goal distribution p(g) in addition to the standard MDP setup. In the episodic goal-conditioned RL setting, each episode involves sampling goal from the goal distribution g p(g), and attempting reach it. The policy, π,, and reward, are conditioned on the selected goal g. Goal-conditioned RL problems leverage special form of the reward function as r(s, a, = (s g), 1 when goal is reached, and 0 otherwise. As in standard RL, at evaluation time an agent is tasked with maximizing the discounted reward based on the goal, E gg)p(g))== Ei= r Qe, a..g). g)]. Note that, unlike most
work in coal-conditioned RI., we are the autonomous goal-conditioned RL setting, where we do not have access resets during training.
While goal-conditioned RL makes the reward r(sr, dr, g) particularly easy to specify, in continuous spaces r(st, .t. g) = 1 (s; = g) is zero with high probability. Recent techniques have circumvented this by leveraging the idea of hindsighs relabeling [62, 58, 17]. The key idea is to note that while when commanding a goal the reward will likely by 0, it can be set to had the states that are actually reached, been commanded as goals, thereby providing self-supervised learning signal [62, 58, 17]. This idea of using self-superviiion for policy learning has spanned both techniques for RL [58. 59] and iterated supervised learning [61, 17]. The resulting objective for supervised policy learning can be expressed as arg max, (r- E log π (se ((()) Self-supervised policy learning algorithms [17] alternate between sampling trajectories by commanding the desired goals under the current policy (-19)) and target goal distribution g ~ p(g), where denotes stop-gr nop-grraient. Policies can then be learned based on the goals reached in hindsight g (rr), where G is any function for hindsight relabeling for instance, choosing the last state of the goal.
4 GEAR: A System for Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback
Our proposed system Guided Exploration for Autonomous Reinforcement learning (GEAR), is able to leverage self-supervised policy learning scheme with occasional human feedback to learn behaviors without requiring any resets or hand-specified rewards. While typical autonomous RL methods aim to reset themselves autonomously, this comes at the cost of challenging reward design problem. The reward function has to be able to account for all eventualities that the system may find itself in, rather than behaviors from single initial state. In this work, we instantiate practical algorithm GEAR, that takes the perspective that cheap, asynchronous feedback in the form of binary comparisons provided remotely by humans can guide exploration in autonomous RL.
4.1 Reset-Free Learning via Goal-Conditioned Policy Learning
The problem of autonomous RL involves learning a policy to perform a task evaluated from a des- ignated initial state distribution Po. but without access to episodic resets during training. Typical paradigms for this problem [4, 26, 5] have alternated between training a "forward" policy to solve the task, and a "reverse" policy to reset the environment. The challenge with applying this in the real world boils down the difficulty of reward specification.
To circumvent the challenge of reward specification, we can model the problem as goal- conditioned one and leverage self-supervised learning methods [17]. This involves learning a single goal-conde conl-conditioned policy x {als,, g) that can perform both forward and reverse tasks. While trying to accomplish the forward goal, the policy takes g ~ p(g) as its goal, and once the goal is accom- plished, the reverse process takes in g ~ (s(s) as its goal, bringing the agent back its initial state. The policy It [a]ss,0 g) can be learned by self-supervised goal-conditioned policy learning methods [17, 61]. In these approaches, states that were reached in some trajectory during training rela-

---

Forward Exploration
Coal Target Goal
Fend
Kibet
Dupbroson
sandom seploration
by the reachablee set
Riwach the and flpp tsiget
reschab e sot
Statc-Goal Proximity
Gondor
Density Model
Backward Exploration
Goal = Initiall State
Depiction of autonomous exploration with GEAR the policy alternates between trying to go to goal state and getting back the initial state. In doing the agent commanded an intermediate aobggoal that both proximal the goal, reachable under the current policy. When this is absent, the policy performs random exploration. The resulting policy learns to go back and forth, while efficiently exploring the space.
Figure 2:
beled, in hindsight, as goal states from which those trajectories serve examples of how to reach them. Then, they use supervised learning to learn policies from this data (Section 3).
problem with this kind of self-supe policy learning algorithm that relies purely on policy generalization for exploration; the policy commanded to reach g p(g) or g po(s), even if has not actually seen valid paths to reach g. This may result in very poor trajectories since the pair (s, g) can be out-of-distributi This problem is especially exacerbated in autonomous RL settings.
4.2 Guided Exploration and Policy Learning via Asynchronous Human Feedback
Rather than always commanding the initial state or the target goal as described above, human feed- back can help select meaningful intermediate sub-goals to command the policy to interesting states from which to perform exploration. In autonomous RL, meaningful intermediate sub-goals Jsub are those that make progress towards reaching the currently desired goal. Given desired goal g. the intermediate sub-goal Isab should be (I) close g in terms of dynamical distance [66], and (2) reachable from the current state under the policy π. Without knowledge of the optimal value func- tion V*, it is non-trivial to estimate both conditions (1) and (2). In GEAR, rely binary feedback provided by human users to estimate state-goal proximity (condition (1)) and on density estimation for computing state-goal reachability (condition (2)).
Proximity Estimation Using Human Feedback To estimate state goal proximity, we draw inspi- ration from work [40, 15, 16] in learning from human preferences. Specifically, build directly on recently proposed technique that uses human preferences to guide exploration in the episodic setting [40]. While [40] also guides exploration using human preferences to estimate state goal proximities, it has a strict episodic requirement making it unsuitable for autonomous RL.
Techniques based on comparative feedback aim to learn reward functions from binary comparisons provided by non-expert human supervisors. In this framework, human users can be asked to label which state 8; 8j is closer particular desired goal g. These preferences to infer (unnormalized) state-goal proximity function d g) by optimizing dø with respect to the following objective (derived from the Bradley-Tere Terry economic model [67, 15, 33]): Lrank (0)
then be used
-E Liss log exp (xp -d (2)
exp (xp=dg(9t.9)
+ (1 Liss) exp (s. de
exp (9(()
This suggests that if state s, is preferred over sy in terms of its proximity to the goal g. the unnormalized distance should satisfy do(st, g) < ds(s).
5

---

During exploration, we can choose a sub-goal by sampling batch of visited states and score them according to d g). ) As outlined in [40], the chosen sub-goal doesn't It need to be the one that mini- mizes the estimated distance to the goal, but rather can be sampled softly from a softmax distribution exp dd(s.g)
p(ysubl8 (..)) Evec exp do g))
to deal with imperfections in human comparisons.
Reachability Estimation Using Likelihood Based Density Estimation While the aforemen- tioned proximity measure, suggests which intermediate sub-goal Jub is closest to particular target goal g. this does not provide a measure of "reachability". whether Juub is actually reachable by the current policy n, from the current environment state s. This not a problem in an episodic setting, but in the autonomous reset-free learning case, many states that have been visited in the data buffer may not be reachable from the current environment state using the current policy π
In our self-supervised policy learning scheme (Section 4., reachability corresponds directly to marginal density- seeing (s, Isbb) pairs in the dataset is likely to indicate that Jsub is reachable from a particular state s. This is a simple consequence of the fact that policies are learned via hindsight relabeling with supervised learning [17], and that supervised learning oracle would ensure reachability for states with enough density in the training set. This suggests that the set of reachable intermediate sub-goals 9lub can be computed by estimating and then thresholding the marginal likelihood of various (s, Buhh) in the training data. To do so, standard maximum likelihood generative modeling technique can be used to learn a density Pv ( (5t: Isub) [69, 70, 71, 72, 73].
The learned density model (st (ab)) can be used to select reuchable goals with a simple procedure given batch of sampled candidate sub-goals from the states visited thus far, we first filter reachable then be
c. The set of reachable candidates used for sampling a proximal goal proportional to the state-goal distance de estimated from human feedback as described above. Note that when there are no viable reachable candidates, the policy can perform random exploration. In our experimental evaluation in simulation, we estimate this density with a neural autoregressive density model [69, 70, 71] or a discretized, tabular density model.
candidates by thresholding density ( (ubb)
4.3 System Overview
The overall system in GEAR learns policies in the real world without needing resets or reward func- tions and minimal non-expert crowdsourced human feedback. GEAR alternates between trying to explore and learn a policy (o(als,g) g) to reach the target goal distribution g ~ p(g) in the forward process, and reach the initial state g ~ po(s) in the reverse one. In each of these processes, the agent selects intermediate sub-goals Jlub for exploration based on the current state and the desired goal. The sub-goal selection mechanism first samples set of visited states from the replay buffer Deandidates . D. Then, it uses the density model Py to estimate the reachability of these states from the current one, filtering out the unreachable states. Amongst these, D Dillered candidates inter- mediate sub-goals are sampled 9sub proportional to their estimated proximity to the desired goal, exp -dpp-ddd
according to the numan-trained model p(gsubl s. Ed'eplient exp 10--ddd (e) If no states
reachable, the agent performs random exploration. This exploration process repeats until the target goal g ~ p(g)(or the start state g ~ Po(s)) is reached, then the goal is flipped and the learning con- tinues. Occasionally, the agent updates its density model ps (*t, (ut, by likelihood-basse training and accepts occasional, asynchronous feedback from crowdsoureed human supervisors to update the state-goal -oal proximity estimate dig (gra. g). To warm-start training, we can add a set of teleoperated data (potentially suboptimal) to the replay buffer, and pretrain our policy π x{a|s, g) via supervised learning on hindsight relabeled trajectories as described in Section 3 and [17, 61, 40J. Detailed pseudocode and algorithm equations can be found in Appendix A.
5 Experimental Evaluation
To evaluate proposed learning system GEAR, aim to answer the following questions: (1) Is GEAR able to leam behaviors autonomously in simulation environments without requiring resets or

---

careful reward specification? (2) Is CEAR able to scale to learning robotic behaviors directly in the real world using asynchronous crowdsourced human feedback and autonomous RI.2
5.1 Evaluation Domains
We evaluate our proposed learning system on four domains in simulation and two in the real-world. The benchmarks are depicted in Fig and consist of the following environments: Pusher: simple manipulation task in which has to move to certain goal location. Kitchen: another manipulation task in which the robot has to open and close microwave. Four Rooms: another navigation task in which an agent has to move through some rooms with tight doors. Details of these environments can be found in Appendix
object within table. A Navigation task in which Turtlebot
Pusher [real]
Navigation (real]
Pusher
Navigation
Kitchen
Four Rooms
Figure 3: Evaluation Domains for GEAR. We consider mixture of navigation and manipulation tasks both in simulation and the real world for autonomous lcarning.
Baselines and Comparisons
To test the effectiveness of GEAR, we compare with several baselines on autonomous reinforcement learning and learning from human feedback. Details of the baselines presented in Appendix B
5.3
Does GEAR learn behaviors autonomously in simulation?
Four Rooms nevigation
LoCoBot navigation
Black Pusher
Kitchen manipulation
30.0
0.
300
30c
104
03
13
100c
1908
94X
9600
1400
0.2M
0.4M
0.6M
0.BM
1M
dtage
of of ofateze
-Orsch
-HUGE
GDS
VVICE
-ituman
Figure 4: Success rate of autonomous training in simulation of GEAR as compared to baselines. We find that both autoregressive and tabular variants of GEAR are able to successfully accomplish all tasks, more efficiently than alternative reset- free, goal conditioned, and human-in-the th-thh-loop techniques
As seen in Fig 4, when evaluation success rates are averaged over seeds, GEAR able suc- cessfully learn across all environments in simulation. In particular, previous work in reset-free RL like VICE FBRL. In the of FBRL, this is consequence of GEAR not relying belling methods are more sample efficient than PPO-based methods [40]. This, together with the more accurate reward signal that GEAR gets from comparative human feedback, explains why GEAR outperforms VICE.
see that GEAR outperforms
carefully tailored reward function as well the fact that hindsight rela-
We also that GEAR is significantly more performant than HUGE, due the fact that GEAR counts for policy reachability when commanding subgoals. By ignoring this, the goal selection mechanism in HUGE often selects infeasible goals that get the agent trapped. We explore this topic further in E.1 where we show that GEAR reaches a higher percentage of commanded goals than HuGE. Additionally, by relying on subgoals to guide exploration, GEAR also outperforms GCSL.
Moreover, GEAR beats previous work that rely on human feedback, such as the Human Preferences method, by guiding exploration via subgoal selection and also for the robustness to non-tailon con-tailored reward signal (feedback) that subgoal selection together with hindsight relabeling brings [40].

---

Notice that in Fig 4, we conduct experiments using both autoregressive neural density models [69, 70] and tabular densities for measuring policy reachability, except in the kitchen environment, in which the high dimensionality inhibits using tabular densities. While the comparisons Fig 4 are done from the Lagrangian system state, we will explore scaling this to visual inputs in future work.
In order to obtain fair comparison between all of the baselines, we leveraged synthetic oracle instead of a real human. In Section 5.4, we show CEAR succeeds in learning optimal policies when using human feedback in the real world. Furthermore, in Appendix D we show that CEAR can learn successful policies no matter where the feedback is coming from: synthetic oracles, non-expert annotators on Amazon Mechanical Turk, and expert annotators.
In Appendix E, provide further analysis of the hyperparameters of GEAR, well show the effect of the amount and frequency of human feedback needed to learn optimal policies.
5.4 Does GEAR learn behaviors autonomously in the real world from crowdsourced human feedback?
Next, we trained GEAR in the real world, learning from crowd- sourced human feedback from arbitrary, non-expert users on the web. We set up two different tasks, first, navigation one with the TurtleBot, and second, a manipulation task with the franka arm consisting of pushing a bowl (Fig 3). The first task involves leav- ing the robot in living room and allowing it to explore how navigate the room autonomously other through obstacles, with human feedback crowdsoureed from Amazon Mechanical Turk. The robot is provided with around 10 trajectory demonstrations of teleoperated seed data and then left to improve autonomously. Human supervisors provide occasional feedback: 453 comparative labels provided asynchronously over 8 hours, from 40 different annotators. For the second task, the robot again provided with 10 trajectories and 200 labels from 22 an- notators the course of one hour. We observe in Fig 5 that policies successfully learn solve each task from minimal human supervision and reset-free setting in the real world. A depiction of the interface can be found in Appendix G, and more details on the demographics of crowdsourced supervision can be found in Appendix D.
Luttectioo Nategaatonn (seal)
Nurttrr 90003
go from one location to an-
Pusher (maa))
Sutgen
Figure 5: Evaluation in the
real-world for Franka pusher and TurtleBot Navigation showing continuous improvement
Conclusion and Limitations
In this work, we present framework for autonomous reinforcement learning in the real world using cheap, asynchronous human feedback. We show how a self-supervised policy learning algorithm can be efficiently guided using human feedback in the form of binary comparisons select interme- diate subgoals that direct exploration and density models to account for reachability. The resulting learning algorithm can be deployed on a variety of tasks in simulation and the real world, learning from self-collected data with occasional human feedback.
This work has several limitations: (1) Safety Guarantees: Real-world exploration with RL can be unsafe, leading to potentially catastrophic scenarios during exploration, (2) Limitation to Binary Comparisons: Binary comparisons provided by humans are cheap, but provide impoverished feedback since it only provides single bit of information per comparison. (3) Requirement for Pre- training Demonstrations: For practical learning in the real world, the efficiency of GEAR is enhanced by using teleoperated pretraining data. This can be expensive to collect (4) Density Model as a Proxy for Reachability: We use density models as proxies for reachability, but this is only a valid metric in small set of quasistatic systems. More general notions of reachability can be incorporated. (5) Learning from low dimensional state: the current instantiation of GEAR learns from low dimensional state estimates through visual state-estimation system, which needs considerable tuning.
Limitations:
8

---

Acknowledgments
We thank all of the participants in our human studies who gave us some of their time provide labels. We thank the members of the Improbable Al Lab and the WEIRD Lab for their helpful feedback and insightful discussions.
The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper. This research was supported by NSF Robust Intelligence Grant 2212310, the MIT-IBM Watson AI Lab, and the Sony Research Award.
Author Contributions
Max Balsells led the project and the novel contributions together with Marcel and Zihan. Max and Marcel led the writing. Max led the navigation experiments in simulation and real world and the Kitchen simulated experiments. Max and Marcel led work the baselines. Max led the work the ablations.
Marcel Torne led the project and novel contributions together with Max and Zihan. Marcel and Max led the writing of the manuscript. Marcel led the pusher simulation and real-world experiments. Marcel led the human crowdsoureing experiments. Max and Marcel led work on the baselines. Marcel was in charge of the figures in the paper.
Zihan Wang led the project and novel contributions together with Max and Marcel. Zihan helped in the writing of the paper and conducting ablations.
Samedh Desai helped to run the navigation experiments in the real world.
Pulkit Agrawal provided some of the valuable resources needed to execute this project.
Abhishek Gupta provided guidance throughout the development the project, assisted with the novel contributions the paper, and helped in the writing of the paper.
References
[1] L. M. Smith, I. Kostrikov, and S. Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning. CoRR, abs/2208.07860, 2022. 1085555.45 2208.07860. URL atttps://dot org/10. /00 48550/arXiv 2208. 07860.
[2] Wu, A. Escontrela, D. Hafner, P. Abbeel, and K. Goldberg. Daydreamer: World models for physical robot learning. In K. Liu, D. Kulic, and Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 2226-2240. PMLR, 2022. URL https: //proceedings. mlr press tress/v2055 v200 w200/wuu233 an23c.hhmll
[3] H. Zhu, Yu. A. Gupta, Shah, K. Hartikainen, A. Singh, Kumar, and Levine. The ingredients of real world robotic reinforcement learning. In 8th International Conference Learning Representations, ICLR 2020, Addis Ababa. Ethiopia, April 26-30, 2020. OpenRe- view.net, 2020. URL https: wtwppppwwww..eeeeettorroormoeeee
[4] A. Sharma, K. Xu, N. Sardana, A. Gupta, K. Hausman, S. Levine, and C. Finn. Autonomous inforcement learning: Formalism and benchmarking. arXiv preprint arXiv:2112.0960) 2021.
[5] B. Eysenbach, Gu, Ibarz, and S. Levine. Leave no trace: Learning reset for safe and autonomous reinforcement learning. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceed- ingy. OpenReview.net, 2018. URL https::/opeerrviiie wwwpee/wwwneettttmmiiemee wt
net.
[6] W. Knox, A. Allievi, H. Banzhaf, Schmitt, and Stone. Reward (mis)design for au- tonomous driving. Artif. Intell,, 316:103829, 2023. doi: 1o011000066))arrmn.22222. 103822, URL attps://doi: org/10..o 1016/j artint. 2022.n..022............ 103829.

---

[7] A. Handa, A. Allshire, V. Makoviychuk, A. Petrenko, R. Singh, J. Liu, D. Makovi- ichuk, K. Wyk, A. Zhurkevich, B. Sundaralingam, Y. Narang, J. Lafleche, D. Fox, and G. State. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. CoRR, abs/2210.13700 2022. doi: 10010444550/axi122000..000 URL https://doi.ors
10. 28..4855//aa /ariiv/r/v..... 2210. 13702.
[8] Agrawal. The task specification problem. In Faust, D. Hsu, and G. Neumann, ed- itors, Conference on Robot Learning. 8-11 November 2021, London, UK, volume 164 of Proceedings of Machine Leurning Research, pages 1745-1751. PMLR, 2021. URL https: //proceedings.. mam.edrinngmm........m.mmm.mm. w.t.press/ trressvv64// agraaaaaaaaaaaaaaaaa
[9] A. Sharma, R. Ahmad, and C. Finn. A state-distribution matching approach to non-episodic reinforcement learning. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvári, G. Niu, and Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 19645-19657. PMLR, 2022. URL https:///prett ittps://proceedinng. mlr press/ 7162/sharma224 html..
[10] A. Sharma, A. M. Ahmed, R. Ahmad, and C. Finn. Self-improving robots: End-to- end autonomous visuomotor reinforcement learning. CoRR, abs/2303.01488. 2023. doi: 10. 00448550/rXXi.2203314453 URL https://doi. wrtp./...././/////..i... http://aaa.vvv........ 23.. 01488.
[1I] K. Xu, Z. Hu, R. Doshi, A. Rovinsky, V. Kumar, A. Gupta, and S. Levine. Dexterous manipu- lation from images: Autonomous real-world RL via substep guidance. CoRR, abs/2212.09900, 2022. doi: 10.445500/aXxii2222..9999 09902.
URL attpss///oo. org/10. 48511...85/88888 http://rXivv..211
[12] Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine. Variational inverse control with events: A general framework for data-driven reward definition. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Informa- tion Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 8547-8536, 2018. URL https://proceeddnng .eeggis.nurss... /c/pappr//008//aa 00919967003811000220000aaaa0od00
Awttsrttttttttttaeeeeaeeeeee html..
[13] A. Gupta, J. Yu, Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, Devlin, and S. Levine. Reset- free reinforcement learning via multi-task learning: Learning dexterous manipulation behav- iors without human intervention. In IEEE International Conference Robotics and Automa- tion, ICRA 2021, Xi' Xn, China, May 30 Juue 5, 2021, pages 6664 6671. IEEE, 2021. doi: URL http: //oiii org/10. 1109/ 1119/00AA///0666661 2021.
9561384.
[14] K. Xu, S. Verma, C. Finn, and S. Levine. Continual learning of control primitives Skill discovery via reset-games. In Larochelle, M. Ranzato, Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 12, 2020, virtual, 2020. URL https: htttt:::/ppooceedngg.eeeee abbtaea htmll.
wwtppppee/02///aaaaeeeeee hash/
[15] P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep re- inforcement learning from human preferences. gio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors. Ad- vances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
Guyon, U. von Luxburg, S. Ben-
2017. URL http::/t//p//poodditiiiii nwwrins cctpaper/2111 /0/0/////hh/////// (B2200aaaa000e111100000000000042 Awstteettt--tttttttteeaaaa html.
10

---

[16] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welin- der, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. NeurIPS, 2022. URL http::hhpppppeeeeeee httppppperreeiipeeeeeeppeeeeeeee http::/22222eeleeccoooofffrr2s50 html.
Conference...........
[17] D. Ghosh, Gupta, J. Fu, A. Reddy, C. Devin, B. Eysenbach, and S. Levine. Learning to reach goals without reinforcement learning. 2019.
[18] Lange, M. Riedmiller, and Voigtländer. Autonomous reinforcement learning on raw visual input data real world application. In The 2012 international joint conference on neural networks (ICCNN) pages 1-8. IEEE, 2012.
[19] D. Kalashnikov, A. Irpan, Pastor, J. Ibarz, A. Herzog, Jang. D. Quillen, Holly, M. Kalakrishnan, V. Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[20] H. Zhu, A. Gupta, Rajeswaran, Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference Robotics and Automation (ICRA), pages 3651-3657. IEEE, 2019.
[21] M. Bloesch, J. Humplik, V. Patraucean, R. Hafner, T. Haarnoja, A. Byravan, N. Y. Siegel, S. Tunyasuvunakool, F. Casarini, N. Batchelor, et al. Towards real robot learning in the wild: A case study in bipedal locomotion. In Conference on Robot Learning, pages 1502-1511. PMLR, 2022.
[22] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4):1307-1319, 2020.
[23] H. Bharadhwaj. Z. Wang, Y. Bengio, and L. Paull. A data-efficient framework for training and sim-to-real transfer of navigation policies. In 2019 International Conference on Robotics and Automation (ICRA), pages 782-788. IEEE, 2019.
[24] W. Han, S. Levine, and P. Abbeel. Learning compound multi-step controllers under unknown dynamics. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6435-6442. IEEE, 2015.
[25] C. Finn and S. Levine. Deep visual foresight for planning robot motion. In 2017 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 2786-2793. IEEE, 2017.
[26] A. Sharma, A. M. Ahmed, R. Ahmad, and C. Finn. Self-improving robots: End-to-end au- tonomous visuomotor reinforcement learning. arXiv preprint arXiv:2303.01488 2023.
[27] A. Sharma, A. Gupta, S. Levine, K. Hausman, and C. Finn. Autonomous reinforcement learn- ing via subgoal curricula. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, vir- tual, pages 18474 18486, 2021. URL https: httpp::/ neurips cc/paper/2021/ html.
[28] A. Gupta, C. Lynch, B. Kinman, Peake, Levine, and K. Hausman. Demonstration- bootstrapped autonomous practicing via multi-task reinforcement learning. CoRR, abs/2203.15755 2022. doi: do::10445500axxx.222..5555. URL https://doi.org:
A8550/arXiv arXiv.2203.1. 15755.
2203.

---

[29] K. Xu, S. Verma, C. Finn, and S. Levine. Continual learning of control primitives: Skill discovery via reset-games. Advances in Neural Information Processing Systems, 33:4999- 5010, 2020.
[30] Z. Zhang and L. Weihs. When learning is out of reach, reset: Generalization in autonomous visuomotor reinforcement learning. CoRR, 12002303.17600, 2023. doi: 46r:100488500//xx..2333.3 17600. URL httpp:://oi.orgg org/ 2at///00.855//////. arXiv 2303 10300.1176000 17600.
[31] E. Biyik. Learning preferences for interactive autonomy. CoRR, abs/2210.10899 2022. doi: 10. 20555//r2Xiv22200108858 URL https://doi. org/10 10/ 48550/arXiv 2210 10899.
[32] D. Sadigh, A. D. Dragan, S. Sastry, and S. A. Seshia. Active preference-basee learning of reward functions. In N. M. Amato, S. S. Srinivasa, N. Ayanian, and S. Kuindersma, editors, Robotics: Science and Systems XIII, Massachusetts Institute of Technology, Cam- bridge, Massachusetts, USA, July 12-16, 2017, 2017. doi:10. 100.11667///....1...... http///wwwwoooi.ccpppeeeeeeeeeee 1
URL
html.
[33] D. S. Brown and S. Niekum. Deep bayesian reward learning from preferences. CoRR, 1bs/1992.04442, 2019. URL http:://arrd...rrra////9122022 04422
[34] K. Lee, L. M. Smith, and P. Abbeel. PEBBLE: feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 6152-6163. PMLR, 2021. URL. http: pttp:://proceedingg. mlr -pwsss//139/1ee// html.
[35] K. Lee, L. M. Smith, A. D. Dragan, and P. Abbeel. B B-pref: Benchmarking preference-based reinforcement learning. In J. Vanschoren and S. Yeung, editors, Pro- ceedings of the Neural Information Processing Systems Track on Datasets and Bench- marks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL http::///tttttttteeebbboaaaatspe htt.aa/ca..aaa.eeo6666666666666-
www.sis.e.rip...r. wtt..nns.....apeeee/eeee/ep htc/paper/202///aae .html.
[36] Hejna and D. Sadigh. Inverse preference learning: Preference-based rl without a reward function. arXiv preprint arXiv:2305.. 15363, 2023.
[37] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/ abs/1707.00347, 2017. URL httppp://axvvvr.r/aaaaaaa
06347. [38] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback, 2022. URL https://arxiv. wgg/abs/220302/555 13, 2022.
[39] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
[40] M. Torne, M. Balsells, Z. Wang, S. Desai, T. Chen, P. Agrawal, and A. Gupta. Breaderumbs to the goal: Goal-conditioned exploration from human-in-the-loop feedback. arXiv preprint 13X.:23077.11499 2023.
[41] A. Jain, D. Das, J. K. Gupta, and A. Saxena. Planit: A crowdsourcing approach for learning to plan paths from large scale preference feedback. In 2015 IEEE International Conference Robotics and Automation (ICRA), pages 877-884, IEEE, 2015.
[42] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, Gao, Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
12

---

[43] A. Bobu, A. Bajesy, J. F. Fisac, and A. D. Dragan. Learning under misspecified objective spaces. In Conference on Robot Learning, pages 796-805. PMLR, 2018.
[44] A. Bobu, M. Wiggert, C. Tomlin, and D. Dragan. Feature expansive reward learning: Rethinking human input. In Proceedings of the 2021 ACM/IEEE International Conference Robot Interaction, pages 216-224, 2021.
[45] D. Losey, A. Bajesy, M. K. O'Malley, and A. D. Dragan. Physical interaction as communi- cation: Learning robot objectives online from human corrections. The International Journal of Robotics Research, 41(1):20-44, 2022.
[46] A. Saran, E. S. Short, A. Thomaz, and S. Nickum. Understanding teacher gaze patterns for robot learning. CoRR, abs/1907.07202, 2019. URL http:///axxxii.orr httppp///aaxi..oorgr/bb///////// 07202.
[47] S. Ainsworth, M. Barnes, and S. Srinivasa. Mo'states mo' problems: Emergency stop mecha- nisms from observation. Advances in Neural Information Processing Systems, 32, 2019.
[48] Tellex, R. Knepper, A. Li, D. Rus, and N. Roy. Asking for help using inverse semantics. 2014.
[49] C. Basu, E. Biyık, Z. He, M. Singhal, and D. Sadigh. Active learning of reward dynamics from hierarchical queries. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 120-127. IEEE, 2019.
[50] E. Biyik and D. Sadigh. Batch active peference-bbsed learning of reward functions. In Con- ference robot learning, pages $29.528. PMLR, 2018
[51] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, Xia, Varley, et al. Robots that ask for help: Uncertainty alignment for large language model planners. arXiv preprint arXiv:2307.011 00922992........ 2023.
[52] A. Peng, A. Netanyahu, M. K. Ho, T. Shu, A. Bobu, J. Shah, and P. Agrawal. Diagnosis, feedback, adaptation: A numan-in-the-loww framework for test-time policy adaptation. 2023.
[53] A. Nanavati, C. Mavrogiannis, Weatherwax, L. Takayama, M. Cakmak, and S. S. Srini- vasa. Modeling human helpfulness with individual and contextual factors for robot planning. In Robotics:
Science and Systems, 2021.
[54] C. Pérez-D' Arpino and A. Shah. C-learn: Learning geometric constraints from demonstra- tions for multi-step manipulation in shared autonomy. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4058-4065. IEEE, 2017.
[55] M. Zurek, A. Bobu, D. S. Brown, and A. D. Dragan. Situational confidence assistance for lifelong shared autonomy. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2783-2789. IEEE, 2021.
[56] S. Booth, S. Sharma, S. Chung, J. Shah, and E. L. Glassman. Revisiting human-robot teaching and learning through the lens of human concept learning. In 2022 17th ACM/IEFE Interna- tional Conference on Human-Robot Interaction (HRI), pages 147-156. IEEE, 2022.
[57] M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In L. D. Raedt, editor, Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5502-551 15 ijcai.org, 2022. 20o:10029633/icaa.2222227 2022/770.
URL https: /ttls:///ooooooo org/org////o wttg 24963/1jcaa
[58] M. Andrychowicz, F. Wolski, A. Ray, Schneider, R. Fong, Welinder, McGrew, J. To- bin, O. Pieter Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
13

---

[59] A. Levy, G. Konidaris, R. Platt, and K. Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv: 171..00494 2017.
[60] M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. To- bin, M. Chociej. P. Welinder, V. Kumar, and W. Zaremba. Multi-goal reinforeement learning: Challenging robotics environments and request for research. CoRR, abs/1802.09464, 2018. 09464.
URL http: http//aaxvvv..g/bbbba//ee
[61] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, latent plans from play. Conference on robot learning, pages 1113-1132. PMLR, 2020.
S. Levine, and P. Sermanet. Learning
[62] L. P. Kaelbling. Learning to achieve goals. In IJCAI, volume 2, pages 1094-8. 8. Citeseer, 1993.
[63] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time, arXiv preprint arXiv:2210.06400 2022.
[64] T. Davchev, O. Sushkov, J.-B. Regli, S. Schaal, Y. Aytar, M. Wulfmeier, and J. Scholz. Wish you were here: Hindsight goal selection for long- long-horizon dexterous manipulation. arXiv preprint arX0v:2112.00559, 2021.
[65] A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 9209-9220, 2018. URL https: http::::s//pocceeddeng wtw.rreeeeinn.n...eree..eeeeeeee nwwrips....... cc/ html.
[66] K. Hartikainen, X. Geng, T. Haarnoja, and S. Levine. Dynamical distance learning for unsu- pervised and semi-supervised skill discovery. arXiv preprint arXiv: (rXiv:1907.08222 2019.
[67] R. A. Bradley and M. E. Terry Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 3963/4):324-345, 1952.
[68] R. Sutton and A. G. Barto. Reinforcement learning: An introduction. 2018.
[69] B. Uria, M. Côté, K. Gregor, I. Murray, and H. Larochelle. Neural autoregressive distribution estimation. CoRR, abs/1605, 600006605.0226, 2016. URL http: org/abs/ 02226.
[70] A. van den Oord, N. Kalchbrenner, L. Espeholt, K. Kavukcuoglu, O. Vinyals, and A. Graves. Conditional image generation with pixelcnn decoders. In D. D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances Neural Information Processing Sys- tems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4790-4798, 2016. URL https: http://///oc//eeee//eeeee/eee seeeiiinnnnneuurpeeeeeee.e neurips. cc cc/ cc/paper http:///00//aee/01111111aaaaaaaa
attpaa//dttteesaaaaaatt..mm. Abstract. html.
[71] A. van den Oord, N. Kalchbrenner, and K. Kavukcuoglu. Pixel recurrent neural networks. In M. Balcan and K. Q Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1747-1756 JMLR.org, 2016. URL htwpeeeiinnn..r..sssss.eee
http://proceedingg mlr. S....prres///4/oorr166 html.
[72] D. Kingma and M. Welling. Auto-encoding variational bayes. In Y. Bengio and Y. LeCun, editors, 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. http:ttp://rrxis org/ abs/1312. 6114.
[73] L. Dinh, Sohl-Dickstein, and Bengio. Density estimation using real NVP. In 5th Inter- national Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https://openrevien. net htt.forumiid--Hlppppae
14

---

[74] J. Fu. A. Singh, D. Ghosh. L. Yang, and S. Levine. Variational inverse control with events: general framework for data-driven reward definition. CoRR, abs/1805. 1686, 2018. URL 11686.
[75] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine. Leave trace: Learning reset for safe and autonomous reinforcement learning. arXiv preprint
2017.
[76] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization, 2017.
15

---

Finally, we provide some more insight into our work. In particular, it consists of:
Appendix A Algorithm, shows the pseudocode of method.
Appendix B Baselines, explains the different baselines that we compared GEAR to in more detail.
Appendix Benchmarks, explains the different comparison environments in more detail.
Appendix D Human Experiments, gives further details on the human experiments, as data collection procedure and annotators demographics.
Appendix E Ablations, We study the effect of changing key parts of our algorithm or certain hyperparameters.
Appendix F Hyperparameters, depicts the hyperparameters used in the different runs.
Appendix Interface, show the web interface that we used for collecting human feedback.
A Algorithm
We reproduce of the learning objectives here for posterity. The following is the objective for training the goal selector with human-provided comparative feedback:
exp (s(...))
Crans (8) -B(s 1ics log exp (sp 9) + exp do(s),g)
(1)
exp dol (5
1(<))
exp do(s + exp do (s).
The density model (sr, Jaub) can be trained on dataset D = {(s), ) of relabeled (st. Isub) tuples via the following objective:
max (e (log Pos (St, Sruh))
Different choices of family for pol (St, Isuh) yield different variants. We leverage tabular density models and autoregressive. Policies trained via hindsight seff-supervision optimize the following objective:
arg max (e
Σ log ग (a. |s, S(T))
(4)
To sample goals from the learned proximity metric, sample Jsub p( ss g), where
exp dd. g)
P(arub (s, E EvED exp (s'
(5)
Below, show algorithm in pseudo-code format.
16

---

Algorithm GEAR
1: Input: Human H. goal g. starting position
2: Initialize policy n, density model de, proximity model fa. data buffer D, proximity model buffer
g
3: while True do
4:
p p(g)
5:
DD < PolicyExplorationns 9, G.8.7 D)
6:
D DU Dz
7:
π f TrainPoliey(r:: (1, D) (hindsight relabeling [17], Eq 4)
8:
G GU CollectFoodback(). H) 4.2)
9: fo f "rrainGoaSSeletorr fe. () (Eq via the Bradley-Tery model [67]) Train de, 9) (Eq 3, [69, 70, 71))
11: end while
Algorithm 2 PolicyExploration
Input: policy æ, goal selector fo. goal g. data buffer
2: D, 0
3: s 80 50
4: i= 1,2, do
5: every k timesteps:
6: S -OOtainneachaallstttttssssd/
(Sec 4.2, [69, 70))
7: gb ~ SampleClossstttate(fff 9, (Sec 4.2, 5)
8: while NOT stopped do
9: Take action a ~ π (a) 88 9b)
end while
11: Execute Frandom for H timesteps
Add D, without redundant states
13: end for
return D,
B Baselines
We compare GEAR against relevant baselines in autonomous reinforcement learning and learning from human preferences, which are presented next:
(1) GCSL [17]: this baseline involves doing autonomous learning with purely self- supervised policy learning [17], alternating between commanding the start and the goal during exploration
(2) HuGE [40]: this baseline compares with the HUGE algorithm [40], which leverages comparative human feedback asynchronously, but without accounting for policy reachabil- ity during training
(3) Classifier Based Rewards (VICE) [12, 9]: this bascline [74, 26] performs forward- backward autonomous RL going between a start position and goal position with the wards provided by classifier that is trained with goal states as positives and on-policy
states as negatives, as in [26)
(4) Learning from Human Preferences [15]: this baseline adapts the learning from hu- man preferences paradigm [15] to the autonomous RL setting by commanding goals both forward and backward.
(5) Forward Backward RL (FBRL) [24, 75]: this baseline uses dense reward functions learn a goal-conditioned policy reach the goal and go back the starting set. To evaluate all methods, we follow the protocol in [4], where training proceeds reset-free but intermediate checkpoints are loaded in for evaluation from the initial state distribution.
17

---

C Evaluation Environments
We briefly discussed the evaluation environments we used to compare our method to previous work. In this section, we will go through the details of each of them.
Pointmass navigation:
This is a holonomic navigation task in an environment with four rooms, where the objective is to move between the two farthest rooms. This is modification of a benchmark proposed in [17].
In this benchmark, the observation space consists of the position of the agent, that is, (x. E RR2 while the action space is discrete of cardinality 9. In particular, there are 8 actions corresponding to moving a fixed amount relative to the current position, the di- rections are the ones parallel to the axis and their diagonals. Additionally, there is an action that encodes no movement.
The number of timesteps given solve this task is 50. Finally, for human proxy, use the distance the commanded goal, taking into account the walls, i.., consider to shortest distance according to the restrictions of the environment.
LoCoBot navigation:
This benchmark is similar to the Four Rooms one since we are also dealing with 2D nav- igation. The main difference is that we are working with simulated robot in Mujoco, in particular LoCoBot, in a real-life-like environment, in which there is a kitchen and living room, thus presenting some obstacles for the robot such as tables or couch. Ad- ditionally, the robot works with differential driving, as LoCoBot or Turtlebot would do. The environment tries resemble the one we do in the real world with TurtleBot, so that results obtained in simulation are, a certain extent, informative about how our robot would perform with the different algorithms in the real world.
In this environment, the goals the robot should learn how to reach are the lower right and the upper left corners. In this environment, the state space is the absolute position of the robot, together with its angle (a, y, 0) E R As are working with differential driving, the action space is discrete encoding 4 actions: rotate clockwise, rotate counterclockwise, move forward, and movement.
The LoCoBot should reach the given goal within 40 timesteps. As before, for the human proxy, we just use the distance to the goal, accounting for obstacles.
Block Pusher:
This is a robotic manipulation problem, where a Sawyer robotic arm pushes an obstacle to given location. This benchmark also a modification of one of the benchmarks proposed by [17]
In this environment the state space consists of the position of the puck and the position of the arm (I1, y1, 22, 32) E IR4, The actions space is the same as in the Pointmass navigation benchmark (i.e. discrete with 9 possible actions).
The should push the object to the desired location in at most 75 timesteps. As for the human proxy, the reward function use is the following:
max maz(disttoeee-puck fiiistaanee-puk..fineer 0.05) + distance .oaal
Kitchen:
This environment is modification of one of the benchmarks in [4]. It consists of a Franka robot arm with DoF doing manipulation in kitchen. The objective is to learn how to open and close the microwave.
The observation space consists of the position of the end-effector of the robot, together with the angle of the microwave joint, that is (x, y, z, 0) E IR4 The action space is discrete with cardinality 7, representing moving the end-effector forwards backwards into any of the three axes, as well as action encoding no movement. Note that, despite the fact that the
18

---

observation space S R4, as in the Pusher, the actual range that the values can take is larger than in other benchmarks, and in order to be able to manipulate correctly the microwave, more precision is needed. This why we couldn't run GEAR with the oracle densities. The number of timesteps that the Franka has to either open close the microwave is 100. Finally, when using human proxy we use the following reward signal:
distance(arm, goal arm position) + |goal joint current joint), if success distance(arm, microwave handle) bonus, otherwise
(6)
Where by joint we mean the angle of the joint the microwave, success means that the distance between the current state and the goal state is already below a certain threshold, and the bonus can any fixed number greater than said threshold.
TurtleBot navigation in the Real World:
This benchmark is similar the LoCoBot navigation one, the major difference between the two is that this one takes place in the real world instead of a simulation.
The goal is to learn how to navigate between two opposite corners in a home-looking environment, with lot of obstacles. The action and the observation space are the same in the LoCoBot navigation environment. That is, the action space is discrete with possible actions (move clockwise, counterclockwis, forward, and don't 't move), while the state space consists of the absolute position of the TurtleBot and its angle (1r, y, 0) E RR3. In order to get this state, we have top-down camera and the TurtleBot has blue and red semispheres, whose position can be detected by the camera, thus obtaining the position of the TurtleBot, and its angle (by computing the direction of the vector between the blue and red semispheres of the LoCoBot). Finally, we do collision avoidance by leveraging the depth sensor of the top-down camera.
The TurtleBot should reach any goal in 25 timesteps. For the human proxy, we just use the Euclidean distance to the goal.
Real World Pusher with Franka Panda: This benchmark is relatively similar to the pusher environment simulation, except it is with a Franka Emika panda robot in the real world. The goal is to learn how push object on the plane between two different corners of an arena. The challenge here is that the pusher is cylindrical object and planar pushing in this case needs careful feedback control, otherwise, it is quite challenging. The action space is dimensional denoting motion in each direction, diagonals, and a no-op. The state space consists of the position of the robot end effector and the object of interest. In order to get this state we use a calibrated camera and an OpenCV color filter, although this could be replaced with more sophisticated estimmsstiationn system. The system is provided with very occasional intervention when the object is stuck in corners, roughly nudge every 30 minutes. Success during evaluation is measured by resetting the object to corner of the arena and commanding it goal to reach the other corner.
D Human Experiments
We ran GEAR from real human feedback on the Four Rooms navigation environment. We compare the performance of GEAR varying the source of human feedback, coming from crowdsourced pool of non-expert and expert annotators, and a single expert and non-expert annotator. These experiments were done through the IRB approval of the Massachusetts Institute of Technology.
Qualifications for annotators: We requested certain qualifications from the annotators, some default defined by AMT: Masters qualification. This qualification indicates that the annotator reliable since has above a threshold of accepted responses. We also defined our own requirements for ac- cepting the responses:
We required certain performance the control task, they had to do better equal to providing incorrect label and "I don't 't know" label.
19

---

We required the response to be complete, the annotator should have responded to all ques- tions.
Payment: We paid 0.30S for set of 12 questions which take around 2 minutes to answer, which would be equivalent an hourly pay of 15S/hour.
D.1 Human Experiments in Simulation
We ran GEAR from real human feedback on the Four Rooms navigation environment. These experi- ments were run in the span of 4 hours.
Human experiment on Four Rooms
Evaluation success
0.8
0.6
Synthetic
0.4
Non-expert (327 labels)
Non-expert Crowdsourced (445 labels)
0.2
Expert (341 labels)
Expert Crowdsourced (752)
100k
200k
300k
400k
500k
600k
700k
Number steps
Figure 6: Comparison of GEAR trained from different types of real human feedback. We observe GEAR can be trained with non-expert human feedback without degrading performance.
D.1.1 Non-expert Crowdsourced feedback
The experiment for this data collected from annotators Amazon Mechanical Turk (AMT). There total of 78 users who participated. Out of which 12 were discarded because of not meeting the requirements (see Qualifications section above), meaning we collected 445 labels from a total of 66 annotators. The users were presented with the interface shown in Appendix G. They each got set of 12 questions, on a control task and 8 on the target task. The answers on the control task, where we know the ground truth labels, were used to make sure the annotator understood what the question was and discard those annotators who did not understand it. As for the demographies, the annotators could optionally respond demographies survey. The results presented in Figure 7.
Male
Female
16
48.6%
51.4%
8
(a) Sex demographics
(b) Age demographics
Crowdsource experiment for non-expert annotators sex demographics. right: Crowd- sourced experiment for non-expert annotators age demographics
Figure 7: left:
20

---

D.1.2 Expert Crowdsourced Feedback
The experiment for this data collected with the interface presented in Appendix G. We recruited the expert annotators through mailing list in our institution. The annotators were not related to the project, however, they are mostly experts in the fields of computer science and robotics. We collected 752 labels from 29 annotators. We asked the annotators to respond to a demographics survey and the results are presented in Figure 8.
6.9%
3
Mare
34.5%
2
Famall
Genderg q
55.2%
Peeler not to
25
48
(a) Sex demographics
(b) Age demographics
Figure 8: left: Crowdsourced experiment for expert annotators' sex demographics the simulated navigation environment. right: Crowdsourced experiment for expert annotators' age demographics the simulated navigation environment.
D.1.3 Non-expert feedback
We collected the data from a single annotator, who is an acquaintance of the authors. They are not knowledgeable about either the project or the fields of robotics and computer science. We collected 327 labels using the same interface Appendix G.
D.1.4 Expert feedback
We collected data from a single annotator who is knowledgeable in the field of robotics and computer science and very familiar with the project and the underlying algorithm. We collected 341 labels using the interface in Appendix G.
D.2 Human Experiments in the real-world
Below we present the demographic statistics of the annotators that provided feedback on the real- world experiments. We note that we left it optional for the annotators to fill in the demographics form, which is the reason why there fewer data points for the demographics than actual annota- tors who helped in the experiment.
D.2.1 Real-world navigation
For the experiment of the pusher in the real world, collected 453 labels from 40 annotators Amazon Mechanical Turk.
D.2.2
Real-world pusher
For the experiment of the pusher in the real world, collected 200 labels from 22 annotators Amazon Mechanical Turk.
21

---

Made
Femalo
66%
33.33%
(a) Sex demographics
(b) Age demographics
Figure 9: left: Crowdsource experiment on the real LoCoBot Navigation task for non-expert anno- tator's s sex demographics. right: Crowdsourced experiment on the real LoCoBot Navigation task for non-expert annotator's age demographics
Male
16
Femalo
40.9%
59.1%
34
35
44
(a) Sex demographics
Age demographics
Figure 10: left: Crowdsource experiment on the real pusher for non-expert annotators sex demo-
graphics. right: Crowdsourced experiment on the real pusher for non-expert annotators age demo- graphics
E
Ablations
E.1
Analysis of GEAR vs HuGE
In this section, we explore how accounting for reachability makes a crucial difference to the quality of the commanded subgoals. In particular, in Fig 11 we see the percentage commanded subgoals that are reached in HuGE and in GEAR. Notice that GEAR commands goals every timesteps, while HuGE does so once per episode only, meaning that in HuGE the agent has more time reach the goal. Despite this, than in HuGE, thus, showing how, by accounting for reachability, we manage to command states that are more likely to be reached by the agent.
clearly how subgoals in GEAR are clearly reached more consistently
E.2 Ablations on the amount of feedback needed
In this section, we study how the frequency at which we provide feedback affects the total amount of feedback or steps we will need to learn a successful policy. In Figure 12 we see that by decreasing the frequency at which we give feedback, we can get successful policy using fewer queries, that is, less feedback overall. However, when working with low frequencies, we see that the algorithm takes longer to start succeeding. On the other hand, by increasing the frequency at which we provide feedback, that the algorithm starts succeeding in terms of timesteps, but we end up needing more annotations overall. So we trade-off between the time takes to succeed and the amount of feedback that we will require.
22

---

Pusher
0.25
subgoals reached
0.2
0.15
0.1
0.05
Oracle densities (Ours)
HuGE
0.5M
1M
1.5M
2M
Number of steps
Figure 11: Number of commanded subgoals reached by GEAR (Ours) and HuGE throughout train- ing in the pusher environment. We observe Ours reaches much higher number of commanded subgoals throughout training, which means that the goals commanded in HuGE are unreachable, because of the reset-free setting, and clearly shows the effect and necessity of introducing reachable sets.
Pusher
Pusher
Svaluation success
Evaluation success
0.6
0.6
0.4
frog
freq 5
freq 50
0
100k
200k
300k
400k
500k
20k
40k
60k
80K
100k
120k
140k
Number of steps
labels
The frequency represents the number episodes we weit before giving feeobssck agein
Comparison the timesteps needed to succeed depending on the frequency in which we provide feedback. The frequency corresponds to the number of episodes, wait before giving feedback again. We see that by lowering the amount feedback, GEAR takes longer to start reaching the goal, however, it still succeeds. right: Comparison of the labels needed succeed depending on the frequency in which we provide feedback. In general, by lowering the frequency of feedback, can manage solve tasks with increasingly fewer labels, but seen Figure if the frequency is lowered too much, it will have negative impact on the required number of timesteps to succeed. Hence, we see tradeoff between the number of labels given and the number of timesteps to achieve the goal
Figure 12: left:
E.3 Analysis of hyperparameters
To better understand the details of which design decisions affect the performance GEAR, we con- duct ablation studies understand the impact of various design decisions on learning progress. Specifically, aim to understand the impact of (1) the threshold for the likelihood at which state is considered "reachable", (2) the frequency at which new intermediate subgoals are sampled during exploration, (3) the algorithm removes redundant exploration steps during exploration, we ablate how important this step is in performance, and (4) we ablate how much pre-training data required for learning and how this affects learning progress. We find that 1) choosing the right threshold for the success of our algorithm is critical. The best reachable threshold used for the point- make
mass navigation task is Using larger threshold (10 20) smaller one (I) would the algorithm work better. 2) the ablation for sampling frequency shows that the right sampling frequeney would help boost the performance. We tried sampling frequency with 1, 5, 10, and 20.
23

---

Success Rate
0.8
timesteps
timesteps
100k
200k
300k
400k
500k
600k
100k
200k
300k
400k
500k
600k
20
(a) Reachable Threshold
(b) Sampling Frequency
Success Rate
Success Rate
0.8
0.8
0.6
0.6
timesteps
timesteps
100k
200k
300k
400K
500k
600K
100k
200k
300k
400k
500k
50
100
200
300
with removed
sithout removed
(c) Buffer Initalization
(d) Redundant Steps
Figure 13: Ablations of GEAR: We studied the ablations of GEAR with four different setups. In a), we modify the reachable threshold with parameter of 1, 5, 10, 20. In b), we verify the effects of sampling frequency. In c), we use a different number of offline data initialize the buffer. The offline data are collected by driving the agent randomly explore the environment. In d), we evaluate performance of our algorithm by removing and not removing the redundant exploration steps at each end of the trajectories.
The results show that and 10 have a similar performance. Too small (sampling frequency = 1) or too large (sampling frequency = 20) do not work well. the sampling frequency is too small, the agent might not able reach the subgoal and too frequent subgoal selection would make the performance drop. If the sampling frequency is large, there would be more redundant wandering steps which make the learning less efficient. 3) We found that removing the redundant steps would help training significantly. Without removing the redundant steps in the trajectory sampling, there would be stationary states when the agent is stuck in the environment which could lead to the drop of performance. 4) More random pre-trained data would help build up the reachable set and further improve the performance.
Hyperparameters
In this section, we state the primary hyperparameters used across the different experiments. All the values are shown Table
24

---

Parameter
Value
default (to those that apply)
Optimizer
Adam [76]
Learning rate
5.10-
Discount factor (7)
0.99
Reward model architecture
MLP(400, 600, 600, 300)
Use Fourier Features in reward model
True
Use Fourier Features in policy
True
Use Fourier Features in density model
True
Batch size for policy
100
Batch size for reward model
100
Epochs policy
100
Epochs goal selector
400
Train policy freq
10
Train goal selector freq
10
goal selector num samples
1000
Stop threshold
0.05
LoCoBot navigation
Stop threshold
0.25
TortleBot navigation in Real World
Stop threshold
0.1
policy updates per step
50
Orucle Densities
reachable threshold
5
VICE
reward model epochs
20
Human Preferences
reward model epochs
20
Autoregressive
reachable threshold
0.25
Epochs density model
30000
Train autoregressive model freq
300
Batch size for the density model
4096
Table I: Hyperparameters used when GEAR
25

---

G Web Interface for Providing Feedback
In Figure 14 we show an example interface for providing feedback for the TurtleBot navigation task, the same used for the pusher task.
Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback
Anorymous
Affiliation
Nowigation
Pleaee fist WATCH the video bace
Rease 1it 2am
prompled images
Goal State
DON'T
Figure 14: Visualization of the human supervision web interface to provide feedback asynchronously during
robot execution. Users are able to label which of the states is closer goal say they unable judge.
26


Reviews:
[{"review_id": "ujvnkFWV_qU", "reviewer": "Reviewer_yNZB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a Reinforcement Leaning (RL) method, which uses asynchronous human feedback in a binary form (positive or negative), which is a very interesting, and novel way which brings knowledge from the learning from preferences framework to RL. An special attention is given to the world-reset problem in RL of real robots, which here is tackled by learning a monolithic forward and inverse model, instead of one model for the former and another for the later. The paper is tested in simple virtual and real tasks of going towards a goal and pushing an object towards a goal. ", "summary_of_recommendation": "The approach is unlikely to be deployed in hardware in the future due to the still very large amount of trials and human feedback necessary to make a practical working solution. \n\nThe main limitations are the still high number of training steps and feedback required, and the limitation to scenarios which the world is reversible. But those were not mentioned.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "I would like to note that 1900 positive/negative feedback is not really practical, however, reinforcement learning is know for those large numbers. Could you share how this amount of feedback compare to other methods in RL?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is well written and clear.\n\nHowever, I the claims are stronger than the contributions. I do not agree that the approach solves the world-reset problem, since it is applicable only if the actions that the robot exert in the world are reversible. In simple navigation cases the robot can (often) go back, or the robot can push the object back to a previous position, but in cases that the world change irreversibly, e.g. pouring liquid in a glass, the proposed approach is not applicable. As such, I would not claim that the world-reset problem is addressed, but rather that the approach is only applicable in cases where the actions are reversible.\n\nThe presentation could be improved if algorithms and diagrams were provided, giving a better idea of how things work and what are the precise contributions of the paper.\n\nFigures 4 and 5 seem to be trimmed before the convergence and success rate convergence, which makes the results incomplete. Specially Fig. 4, where we can see the figures are trimmed in different points (without any explicit reason) on the x axis. Also there are many more tests, and the results in Figure 5 are presented just for a couple of tests.\n\nAnother good point against the paper is the lack of source code, datasets, and the empty website. If there are simulations we should be able to run some tests and evaluate hyper-parameters effects, which would greatly improve the understanding and the strength of the paper. \n\nThe Achilles heel of the proposed approach is that binary feedback is know for being best for refining models, applying this in the RL directly is not a good idea. As such, I would recommend to have a learned model in the tests, and see how much faster the model is refined with the binary feedback. Maybe that could be the \"niche\" of the approach. No modifications would need to be done to the paper for that as well, it is just a matter of testing.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "lna6XcvCkMd", "reviewer": "Reviewer_SMKn", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes Guided Exploration for Autonomous Reinforcement learning (GEAR). Existing approaches for autonomous RL learn forward and backward policies to solve the task and reset the environment respectively, posing the challenge of reward specification. To avoid this problem, GEAR uses self-supervised learning to learn a goal-conditioned policy. However, rather than alternatively sampling the target goal or the initial state, GEAR identifies promising sub-goals. Such goals are selected leveraging occasional asynchronous human feedback in the form of binary comparisons to estimate state-goal proximity and density estimation to compute state-goal reachability given the current policy and environment state.", "summary_of_recommendation": "The paper requires improvements on the weaknesses highlighted in the review. Also, contributions are not as clear as they should be.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1.  The authors should better explain the differences between their method and [37], which is a work under review, not available as a preprint.\n2.  The authors should include a more in-depth discussion of the obtained results.\n3.  The authors should provide a comparison of performances obtained by increasing or reducing the amount of human feedback.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Strengths\n\n-   Overall, the paper is well-structured, and it clearly presents the problems the authors want to address using each specific component of the proposed learning system.\n-   The authors present experiments on 3 different environments, dealing with holonomic and differential driving navigation and a manipulation problem. In the last two cases, they also present real-world experiments, using TurtleBot and Franka Panda manipulator.\n-   The authors present an ablation study in which they report performances obtained by changing several hyperparameters and by not considering the part of the algorithm in which redundant exploration steps are removed during exploration.\n-   The experimental results show an improvement in performance on all the tested environments with respect to the considered baselines in the simulated environments. The approach is able to\n\n### Weaknesses\n\n1.  As a minor suggestion, the language could be slightly improved in the introduction. Avoiding present perfect tenses, such as has looked and has aimed, when possible, and the modal verb “can” followed by the passive form would make the reading flow more smoothly.\n2.  There are some inconsistencies throughout the text. Concerning the proposed approach, the authors sometimes say that it is “reset-free” or that it “does not require resets”, while other times it is said that it requires “minimal manual resets”. This point should be clarified.\n3.  The authors should better explain the differences between their method and [37], which is a work under review, not available as a preprint. This should be better clarified to evaluate the novelty of this work. Indeed, in section 2, the authors say “In this work, we build on a recently introduced technique [37] that combines self-supervised policy learning via hindsight supervised learning [17] with learning from human feedback to allow for robust learning that is resilient to infrequent and incorrect human feedback. While [37] was largely evaluated in simulation or in simple episodic tasks, we leverage insights from [37] to build RL systems that do not require resets or careful environment setup.” The fact that the proposed method is reset-free and is tested in real-world scenarios seems to be the main difference. However, in section 5.2, [37] is used as a reference for HUGE **reset-free** baseline. In this same section, the authors state that HUGE “leverages comparative human feedback asynchronously, but without accounting for policy reachability during training”. So, is the latter the only difference between your proposed approach and [37]? If the answer is yes, the authors should pose more attention to this last aspect while presenting their approach.\n4.  The authors should include a more in-depth discussion of the obtained results. Why does the innovative feature of GEAR with respect to HUGE (density estimation?, see point 3) bring a much more significant performance improvement in LoCoBot and PointMass navigation than in the Block Pusher environment? Why does the autoregressive neural density model work badly in the Block Pusher environment, while it generally reaches higher performances than the discretized densities in the other two environments? Which of these two models did you use to obtain the results shown in Table 5 for real-world environments?\n5.  The authors should better clarify how the amount of human feedback is determined and how it influences performances. In the TurtleBot environment, 1900 labels are provided over 6 hours, which is approximately 317 labels per hour. This can still be considered a relatively high amount of human feedback. Could it be reduced without significantly impacting performance? Otherwise, this should probably be added in the limitations section.\n6.  The authors use actual human feedback, and not a synthetic oracle, only in the TurtleBot real-world environment. A comparison between the results obtained using the synthetic oracle in this scenario would have been interesting.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "hMIdQi2_W8", "reviewer": "Reviewer_epUg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper extends a prior method for learning using human feedback to guide exploration by adding a reachability model and validating it in robot learning without resets.", "summary_of_recommendation": "Overall, this work has interesting technical elements, but the claim that this would actually work with users is not well supported by the validation as currently presented.  Because the method is otherwise a valuable contribution to CoRL, I am rating this as a weak accept, but I would like to see the authors explain in the rebuttal how they will be more careful about their claims (especially if the human subjects validation was not conducted using appropriate experimental techniques).\n\nPost rebuttal comment: I would still rate the validation as one of the weaker elements of the paper, and I'm not at all sure how the authors will pull all the material together into a paper.  I keep my score as a weak accept.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See above; I would like to see the authors clarify the methods used in the human subjects validation and discuss how they will expand their related work to cover more related work in HRI.  They may wish to also explain how they will update the limitations to address the limitations of the validation.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- This paper tackles an important problem for real-world RL (especially real-world human-in-the-loop RL).\n- The authors show that their method is effective in several real and simulated tasks with simulated human feedback\n- The paper is mostly well-written, although lacking detail in some places, especially regarding the user testing\n\nWeaknesses:\n- The related work is very narrow; 1/3 of the cited works (19/56) are from the same group.  I encourage the authors to read and cite much more widely, especially on the human-robot interaction (HRI) side (e.g. Andrea Thomaz, Julie Shah, Sidd Srinavasa, and their students and postdocs).  Work in HRI may also help to provide better support for how well this method would work in the real world, since it includes testing with non-experts, crowd workers, and other realistic user populations and provides insights into how real feedback deviates from \"synthetic proxies\".\n- There is no information on the human evaluation (in appendix section B the implication is that a proxy is used \"And for the human proxy we just use euclidean distance to the goal).  Was the study approved by an ethics board (e.g., an IRB)? Who were the participant(s)? Did they include robotics experts or research team members? The results in 5.4 are impossible to interpret without this information -- for example, if the feedback was provided by research team member, it is possible that the team member used their knowledge of the approach to provide more useful feedback than a typical human teacher would provide.\n- Did the pusher environment use a proxy? If so, is not appropriate to include it in section 5.4 (or the title of 5.4 needs to be changed), since it implies that both tests in that section were conducted with human feedback.\n\nMinor points:\n\n- Binary feedback often means a \"good\" or \"bad\" (or thumbs up/down) label on a single state; the authors may want to refer to their feedback mechanism as \"binary preference\" or more consistently as \"comparative feedback\" to be clear", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "NXtthulJol", "reviewer": "Reviewer_oFcp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes an autonomous reinforcement learning method based on guided exploration to address the reset-free robot learning problem, by incorporating goal-conditioned policy learning and human feedback.", "summary_of_recommendation": "In this paper, a framework for autonomous reinforcement learning in the real world using cheap, asynchronous human feedback.  Some weaknesses such as the impact of the initial state distribution in a reset-free setting, the experimental baselines and so on should be addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "Please check the weaknesses in Review.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "strengths：\n1.The approach of using human feedback to select subgoal is an interesting work.\n2. The paper is well-written and its main content is relatively easy to understand.\n3. The experiments in the paper are comprehensive, involving various simulations and real-world robot experiments.\nweaknesses：\n1.In my opinion, the paper seems to focus more on goal-conditioned policy learning rather than reset-free robot learning. The methods and experiments primarily address the problem of subgoal selection, without adequately considering the impact of the initial state distribution in a reset-free setting.\n2.The experimental setup and tasks appear to be relatively simple. In reset-free robot learning, the main concern is the influence of changes in the environment due to the task execution on its subsequent learning process. The experiments in the paper do not seem to fully capture this problem.\n3.The baseline used for comparison lacks the reset-free methods.\n4. When using human feedback for subgoal selection, it is necessary to estimate the distance from two states to the goal. However, the human proxy provided in the appendix is defined as a reward function. It is unclear how this reward function is used to evaluate the distance between states and the goal.", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper initially got mixed reviews as the reviewers were not satisfied with some claims in the paper regarding "reset-free" RL as well as missing details for the human experiments. Moreover, the experiments were conceived to be rather simple even though a large range of environments/tasks have been used. The authors did a good job in their rebuttal and could convince all reviewers to change their score such that we have now an overall positive evaluation. I agree with the reviewers that the paper is now in a much better state given the new experiments. It will however be hard to add this new material into a consistent representation of the paper. I nevertheless would rather to see the paper accepted as the approach and the presented results are very interesting.
Decision: Accept (Poster)