Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback
Authors: Marcel Torne Villasevil
Max Balsells I Pamies
Zihan Wang
Samedh Desai
Tao Chen
Pulkit Agrawal
Abhishek Gupta
Venue: NeurIPS 2023 poster (2023)
Keywords: Learning from human preferences
self-supervised learning
exploration in reinforcement learning
TL;DR: Solving long-horizon robotic tasks with crowdsourced human feedback

Abstract: Exploration and reward specification are fundamental and intertwined challenges for reinforcement learning. Solving sequential decision making tasks with a non-trivial element of exploration requires either specifying carefully designed reward functions or relying on indiscriminate, novelty seeking exploration bonuses. Human supervisors can provide effective guidance in the loop to direct the exploration process, but prior methods to leverage this guidance require constant synchronous high-quality human feedback, which is expensive and impractical to obtain. In this work, we propose a technique - Human Guided Exploration (HUGE), that is able to leverage low-quality feedback from non-expert users, which is infrequent, asynchronous and noisy, to guide exploration for reinforcement learning, without requiring careful reward specification. The key idea is to separate the challenges of directed exploration and policy learning - human feedback is used to direct exploration, while self-supervised policy learning is used to independently learn unbiased behaviors from the collected data. We show that this procedure can leverage noisy, asynchronous human feedback to learn tasks with no hand-crafted reward design or exploration bonuses. We show that HUGE is able to learn a variety of challenging multi-stage robotic navigation and manipulation tasks in simulation using crowdsourced feedback from non-expert users. Moreover, this paradigm can be scaled to learning directly on real-world robots.

Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedback
Marcel Torne
Max Balsells² Zihan Wang Samedh Desai
Tao Chen Pulkit Agrawal Abhishek Gupta²
Massachusetts Institute of Technology Harvard University University of Washington (marcelto, taochen, tag)) edu
(balsells, aoalsells,vvinang samedh, abhgupta) washington. edu
Abstract
Exploration and reward specification are fundamental and intertwined challenges for reinforcement learning. Solving sequential decision-making tasks requiring expansive exploration requires either careful design of reward functions or the use of novelty-secking exploration bonuses. Human supervisors can provide effec- tive guidance in the loop to direct the exploration process, but prior methods to leverage this guidance require constant synchronous high-quality human feedback, which is expensive and impractical to obtain. In this work, present a technique called Human Guided Exploration (HuGE), which uses low-quality feedback from non-expert users that may be sporadic, asynchronous, and noisy. HuGE guides exploration for reinforcement learning not only in simulation but also in the real world, all without meticulous reward specification. The key concept involves bifur- cating human feedback and policy learning: human feedback steers exploration, while self-superve supervupere learning from the exploration data yields unbiased policies. This procedure can leverage noisy, asynchronous human feedback to learn poli- cies with hand-crafted reward design exploration bonuses. HuGE is able to learn a varicty of challenging multi-stage robotic navigation and manipulation tasks in simulation using crowdsourced feedback from non-expert users. More- over, this paradigm can be scaled to learning directly on real-world robots, using occasional, asynchronous feedback from human supervisors. Project website at https:: hat::://humaa.a eetededdexpllraaeee github. io/HuGE/
Introduction
Pick place
Drawing
Figure 1: HuGE leverages noisy and asynchronous feedback from multiple non-expert humans to train robot control policies directly the real world.
How should we teach agents a new task? A general method is to provide agents with reward function and employ reinforcement learning. This may appear to be straightforward plug-and-play
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

procedure, yet in reality, it usually means iterative human intervention in designing reward functions to overcome exploration obstacles and circumvent potential reward hacking. The cyclical process of a human meticulously crafting reward function, assessing the resulting behavior of the learned policy, and then refining the rewards to enhance performance can be a painstaking and inefficient process.
If an agent is able to effectively and autonomously explore its environment, it can learn new tasks using an easily designed sparse reward function [50] 20]. This would make the training process less reliant on humans designing detailed reward functions for "dense" supervision. Several strategies incentivize state coverage in exploration by expanding the frontier of visited states and explicitly rewarding discovering novel states. [46 [ 21 43]. When purely optimizing for state coverage, the agent overexplores parts of the state space that are not relevant the task at hand [33. This quickly
becomes a futile effort when exploring in environments with large state spaces, suggesting the need for a more directed exploration strategy.
A promising avenue for directed exploration, without requiring a prohibitive amount of reward design, is in developing techniques that leverage non-expert human feedback adjust the agent's training (55) 30 II 36). This work is a step towards the overarching objective of learning new skills in the real-world through meaningful human-robot interactions. A challenge with techniques that optimize policies over learned reward functions is that humans may provide noisy feedback, or may not even know the optimal solution for the task hand [18] rendering their feedback unreliable and biased, a problem which is exacerbated with a large number of non- non-expert supervisors. When this noisy and incorrect feedback used learn reward functions for RL, can lead to learning biased and incorrect policies. Requiring human supervisors to provide high-quality feedback synchronously and frequently during the training process becomes prohibitively expensive. If we want to reduce the burden on supervisors it is important to ensure that inaccuracies, biases in the reward function are not reflected directly in the learned policy.
We note that reward functions in reinforcement learning serve two distinct purposes. One, determining which goal or behavior is optimal, and two, determining how to explore the environment achieve this goal. In this work, we make the observation that these two independent purposes of reward functions can be decoupled. Human feedback can be used to guide exploration, while policy learning techniques like self-supervised reinforcement learning can learn unbiased optimal behaviors for reaching various goals from the data collected via exploration. This provides the best of both worlds: a way to guide exploration using non-expert human supervisors, while avoiding inheriting the bias from providing asynchronous, low-quality feedback.
Human Guided Exploration (HuGE) uses noisy, infrequent, and asynchronous binary human compar- ative feedback to guide exploration towards subsets of the state space closer goal, while learning optimal policies via self-supervision. Rather than having humans directly label which states to visit, HuGE solicits users for binary comparisons between visited states to determine which are closer the goal. These comparisons help train a goal selector which biases exploration towards chosen states that are more proximal to the objective. The process of exploration involves returning to these selected states and subsequently expanding the set of visited states from there. From this exploration data, we learn optimal policies for goal-reaching in self-supervised manner using ideas from "hindsight" relabelling [24 28), hence avoiding reliance on any explicit reward function.
Iterating between exploration guided by the goal selector and self-supervised policy learning, HuGE expands the frontier of visited states until the desired target goals can be reached reliably. Opposed to directly using human feedback and infrequent feedback, as it is simply used to guide exploration, not directly as an objective for policy learning. In the worst case, when human feedback incorrect or absent, the exploration is hampered but the policy learning scheme is still convergent to the optimal solution. This way. humans are able to essentially "drop breadcrumbs". incrementally guiding agents to reach distant goals, as shown in Fig D] This helps us solve more complex tasks than if solely relying on self- sef-ssuperviioon and able to overcome the optimization bias that is often encountered when optimizing over a learned and noisy reward function [16]. Overall, this work presents the following contributions:
reward [16] 45,, HuGE lets humans provide noisy, asynchronous,
Guided Exploration from Noisy and Minimal Human Feedback: We propose an algorithm for separating (noisy) human feedback from policy learning by utilizing comparative, binary human feedback to guide exploration while learning& a policy through self-supervision. This ensures that we can derive optimal goal- reaching policies even from imperfect human feedback.
2

---

Capability to learn from crowdsourced non-expert human feedback: We show HuGE can learn from noisy, asynchronous and infrequent human feedback. We validate this by collecting crowd- sourced pilot data from over hundred arbitrarily selected, non-expert annotators who interact with the robotic learning through a remote web interface.
Demonstration with real-world policy learning: We show that HuGE can learn policies on robotic learning system in the real world, demonstrating effective exploration via human feedback and compatibility with pretraining from non-expert trajectory demonstrations as seen in Figure ]
2
Related Work
Our work builds techniques for exploration methods, goal-conditioned, and human- in-the-loop RL, but presents a technique for learning from noisy, asynchronous and infrequent human feedback. We present the connections each of these sub-fields below:
Exploration in RL: While exploration is a widely studied subfield RL
this is
typically concerned with either balancing the coverage 10 37). This contrasts with our goal of performing targeted exploration, informed by human feedback. Our work is related to ideas from Go-Explore (2]], which explores by maintaining an archive of visited states, returning to states with a low-visitation count, and performing random exploration from there. However, this results in redundant overexploration shown in Figure 2 under the novelty-based exploration paradigm. In contrast, HuGE is able to leverage human feedback to only explore in promising directions as directed by human supervisors, significantly improving sample efficiency.
cetoraatiooneppiiiaatioono tradeoff 4 or maximizing state
For goal-rea problems, alternatives for exploration include self-supervision techniques such as hindsight relabeling 2 24 35). By learning goal-reaching behavior in hindsight for reached states, these methods obtain dense supervision, albeit only for the states that were actually reached. This has been shown to empirically aid with exploration in certain cases, via policy generalization across goals. However, as shown in Figure 3 these methods suffer from underexploration, since the policy generalization across states can be challenging to control. In contrast, HuGE does not rely on arbitrary policy generalization, instead revisiting already reached states at the frontier of visited states and performing further exploration from these states.
Goal-conditioned reinforcement learning: conl-coondiioned RL algorithms 34
35
2 33
28
37 are multi-task RL methods where various tasks are defined as reaching different goal states. A number of different approaches have been proposed for condicond RL -leanning with hindsight
relabeling 2
19 33 28 learning latent spaces for reward assignment
42
48 learning dynamical
distances 27 22 and goal conditioned imitation learning [25 35
24
40 While these algorithms
are able to solve tasks with a simple component of exploration, they can struggle with tasks with complex sequential nature, requiring complex machinery such as hicrarchical architectures [33] 39). In contrast, our work shows the ability to solve sequential goal-reaching problems with standard techniques, just using occasional comparative feedback from human supervisors to guide exploration.
Perhaps most closely related to our work is the paradigm introduced in DDL supervisor occasionally selects which states to explore from. This state is used to define the reward function, which when combined with supersupe Q-learning can aid with exploration for goal- conditioned RL problems. However, exploration is limited to exactly the state selected by the human, whereas in HuGE, learning the parametric goal selector allows the exploration frontier to continue expanding and decreases the overall amount of human feedback needed (Section 5 Figure
where a human
Reinforcement Learning from Human Feedback (RLHF): RLHF typically characterizes class of methods that learn reward models from binary human comparative feedback for use in RL. These techniques have seen use in better aligning language models, and guiding learning of embodied agents in simulation with expert feedback 7| In much of this work, humans are assumed to provide high-quality and frequent feedback learn rewards. In HuGE, we take complementary view and show that by considering human feedback in self-supervised policy learning setting, much of the burden can be placed on self-supervision, rather than on receiving high-quality human feedback. More broadly, human in the loop RL. is well explored concept, with various interfaces ranging from preferences binary right/wrong signals
[66
to scalar rewards (2), language based corrections (53), 12 and even sketching out rewards (II. While we restrict the study
3

---

in this work to binary comparative feedback, exploring how the underlying principles in HuGE (of decoupling human feedback and self supervised policy learning) can be extended to other forms of feedback is an exciting avenue for future work.
Hindsight Relabelling
Novelty Based Exploration
Directed Exploration
(G25L)
ESA, Eaperrr
10uun
Goos
Collanse
Stort
Gool
Start
Figure 2: Comparison of exploration types. hindsight relabelling: suffers from exploration collapse, novelty- based exploration suffers from overexploration, we propose directed exploration from human feedback Ours.
3 Problem Setup and Preliminaries
This work solves goal reaching tasks by using goal-conditioned policy learning methods. The goal reaching problem is characterized by the tuple (S, A, A, ((s)),T.pgg)) adopting the standard MDP notation with p(g) being the distribution over goal states g S that the agent is tasked with reaching. We assume sampling access to p(g). We aim to find a stationary S + A(A). where ((A) is the probability simplex over the action space. We will say that goal is achieved if the agent has reached the goal at the end of the episode (or is within e distance of the goal). The learning problem can be characterized as that of learning a goal conditioned policy that maximizes the likelihood of reaching the goal <- arg max, J(x) = 1gypp()) [Pr, (sT g)}
conditiodd policy (--|s, 9):
Coal-conditioned Policy Learning: Goal-conditioned RL methods approach this problem 1(s g), defining a sparse reward problem: π +- arg max, (s-n( D (rrg(s)). This problem can be difficult to solve with
using the reward function (s)
typical RL algorithms 26 52 23
because the training process largely devoid of learning signals.
3.1 Why is exploration in goal-conditioned reinforcement learning challenging?
Hindsight relabelling: To circumvent the challenge of sparse rewards in goal-conditioned rein- forcement learning, the structure of goal-reaching problems can be exploited using the hindsight relabeling technique 22 3. Hindsight relabeling leverages the insight that transitions that may be suboptimal for reaching "commanded" goals g, may be optimal in hindsight had the actually reached states sT been chosen as goals. This allows us relabel a transition tuple (s, a, s' g, (s)))) with a hindsight tuple (s, a, s',g', g (s)). where g' can be arbitrary goals chosen in hindsight. When g' chosen be states s actually visited along a trajectory. the reward function rs(s) provides dense reward signal for reaching different goals g = which can be used to supervise an off-policy or supervised learning algorithm
RL algorithm
40 47 24
The key to exploration in these methods is generalization even though the desired goal distribution p(g) is not accomplished, learning policies on the accomplished goal distribution p(g') influences exploration towards p(g) via generalization between g' and g. However, this form of exploration is unreliable across domains and goal representations. As shown Figure 2 and in Appendix E this often underexplores the environment without accomplishing the target goal.
Novelty-based based exploration: On the other hand, pure novelty-basee exploration 21
51
44 is also
ineffective, since performs task agnostic exploration of the entire state-space, thereby overexploring the environment (Fig 2 This becomes intractable with large state and action spaces.
Directed exploration (Ours): practical solution would explicitly encourage exploration but directed towards the goal. In the following sections, we will argue that providing noisy, asynchronous
and infrequent binary comparison feedhack by variety of human users can provide this directed exploration, without inheriting the noise and bias of the provided feedback.

---

HuGE: Guiding Exploration in Goal-Conditioned RL with Human Feedback
Goal Selector Training
Data Collection
Policy Learning
We want to reach cup
Hindstght Reiobelling
einory
haxt prae
Ue red Ststes
t (al s:
Train with BIE
20051 cesseesiee
1g(5,8)
Supenised Learning
Policy
Coal Selector
Figure 3: Overview of HuGE. We train a goal selector, from human feedback through state comparisons, to perform directed exploration in self- supervivee learning.
The key idea behind HuGE to leverage (noisy) human feedback for guiding exploration and data collection, but decouple this from the process of learning goal-reaching policies from the collected data. We show how this can be done in self-supervised way, allowing unbiased goal-reaching policies to be learned, while human feedback can still be useful in guiding exploration. We describe each component of our proposed algorithm below.
4.1 Decoupling Human Feedback from Policy Learning for Goal Reaching Problems
While human feedback has been shown to be an effective tool to guide exploration [66 [ policies obtained by maximizing reward functions learned from noisy and infrequent human feedback be very suboptimal, often getting trapped local optima. This places significant burden on human supervision, making challenging for non-expert, casual supervisors to guide robot learning. How can we enable policy learning from infrequent and noisy human feedback?
Our key insight is that human feedback and policy learning can be disentangled by decoupling the process exploration from the policy learning, while only leveraging the human feedback on the exploration process. In this new paradigm, where the generation of exploration data does not necessitate optimality, we can harness guiding signals originating from noisy human feedback, yet still effectively learning optimal goal-conditioned policies.
In particular, for human guided exploration we build on the idea of frontier expansion where exploration is performed by revisiting states at the frontier of visited states and then continuing exploration from these. This technique would maintain a frontier F that consists of the set of all visited states. We can then select previously visited state for the policy to revisit, which call breadcrumb state gb, by querying a "goal selector" on F, that learned from (noisy) human feedback and defined in detail in 4.2 In this way, human feedback can be used to "softly" guide exploration towards promising states go- Frontier expansion can then performed by exccuting random exploration from the reached breadcrumb state Gb. and adding the collected data to the replay buffer for self-supervised policy learning.
will
Given data collected by frontier expansion directed using human feedback, goal- reachinn policies can then be learned by leveraging sulf-supe policy learning techniques [24 (35] that are able to learn policies from collected data using "hindsight" relabeling and supervised learning. Importantly, this self-supervised policy learning procedure does not depend on the human feedback at all and purely self-supervised. The decoupling of exploration and policy learning implies that even if this exploration biased and imperfect, the self-supervised policy learns unbiased paths to reach all goals that were visited. In this way, cheap low-quality supervision from non-expert human feedback can
5

---

be used to ensure guided frontier expansion towards relevant subsets of the state space, while still being able learn unbiased goal-reaching policies. Below, we delve into each of these components guided frontier expansion and self-supervised policy learning and describe the overall algorithm in Algorithm|]
4.2 Guiding Exploration from Human Comparative Feedback by Learning Goal Selectors
To direct exploration more effectively, we hypothesize that receiving small amounts of occasional guidance from non-expert human users can beneficial, even if the feedback biased and noisy. To leverage this feedback, learn a parametric goal selection function directly from binary human comparison feedback. Learning a parametric goal selection function allows this feedback to be useful even when humans are providing feedback infrequently and asynchronously.
Learning State-Goal Distances from Binary Comparisons: We propose simple interface be- tween the human and the algorithm that simply relies on the human supervisor to provide binary comparisons of which of two states $1, S2 closer (given a human's judgment) to a particular goal g (as demonstrated in Fig and Appendix Fig B.4). These binary comparisons can be used to train an unnormalized estimate of distances fo(8, g) by leveraging the Bradley-Terry model of choice 16 g 7
exp fo(s1,9) 9)
max 11(11 > 82)g) log
exp fo(s) 9) + exp Sa($2,9)
(1)
exp fo(32.
(1 - (s) 82 lg) log
exp fo + exp f0(82.9)
This objective encourages states closer to particular goals g to have smaller [o(s, g). While this estimate may be imperfect and noisy, it can serve to bias exploration promising directions.
Using the Learned State-Goal Distances for Biasing Exploration: To guide exploration, we can select which breadcrumb state db to command and start exploring from during frontier expansion, by sampling states inversely proportional to their distance to the goal measured as, exp(- fo (s, g)), where is learned above. This encourages guiding exploration towards states that have lower estimated distance to the goal since these are more promising directions to explore, as indicated by exp -f(ggeg))
the human comparisons. 9b ~ p(gs g): p(gll)) = Ey'cp exp fol''' :g) (2) where 'D represents the set of reached states. Sampling through a soft distribution instead of choosing the state with the minimum estimated distance, overcome noise and errors in the learned distance function, albeit at the cost of slower (yet convergent) exploration. The choice of softmax temperature a determines how much the exploration algorithm trusts the goal selector estimate (large at), versus resorting back to uniform frontier expansion (small (x). The key to effectively using this state-goal distance function, is getting away from directly optimizing the reward function learned from human preferences, avoiding
the aforementioned problems regarding the optimization of the reward function learned from human preferences and instead relying on self-supervision for policy training, Section 4.3
4.3 Sule-Supe Policy Learning: Hindsight Relabeled Learning for Goal-Conditioned Policies
Given exploratory data collected by guided frontier expansion (Algorithm Section 4.1 we can leverage simple supersupe learning scheme building on [44
24 for goal-conditioned policy
learning. Given trajectories T {so: ag. $1, 82 ......87 ST, or V we construct a dataset of optimal tuples using: D.- = {(s ar, g = S h) t, > 0, + h < (3). This relabeled optimal dataset
can then be used for supervised learning: Jpolicy (T) = (,,.t,1mmmt-ll))) Σ log π (( Se., .) (4).
This process can be repeated, iterating between collecting data, relabeling it, and performing su- pervised learning. As policy learning continues improving. the learned policy can be deployed to solve an expanding set of goals, eventually encompassing the desired goal distribution. The overall pseudocode of HuGE is shown in Algorithm I We note that this relabeled supervised learning scheme has been studied in past works [1 47 24J, the unique contribution here being how low-quality human comparative feedback is used to guide exploration for self-supervised policy learning. While prior techniques such as 1 47 24 often struggle with exploration, the occasional human feedhack can lead to significantly more directed exploration behavior for HuGE as compared to prior work.

---

Boostrapping Learning from Trajectory Demonstrations: While the system thus far has de- scribed strategy that is applicable for learning policies from scratch, training HuGE from scratch can be prohibitively slow in the real world, and instead we may look to finetune already pre-trained "foundation" " models or existing goal-directed policies using HuGE, or to pretrain from human demonstrations as considered in huge plethora of prior work 49 32 41
Given that the self-supervised policy learning method in HuGE is fundamentally based on iterated supervised learning, a natural way to ensure the practicality of the method is by simply initializing the policy ı using supervised imitation learning on trajectory demonstrations (or from other sources if available). While this technique is effective policy pretraining, it fails to account for the goal selection model fo (s. A simple modification to the training process allows also for effective pre-trate training the sate--ooal distances model (s, 9) from demonstrations. Concretely, since demon- strations are typically monotonically decreasing terms of effective distance to the goal, a single trajectory can be reinterpreted as a set m(n++)
2 comparisons, where later states in the trajectory preferred earlier ones: {8t, < St, < 12V 8t, 7 These comparisons
then be used to pre-train the goal-selector, as discussed in Equation [ Our proposed technique not a replacement for learning from demonstrations, but rather serves complementary role. As show experimentally, this combination of HuGE and pre-training from non-expert demonstrations actually enables learning the real world under practical time constraints.
Algorithm 1 HuGE: Guided Exploration with Algorithm 2 PolicyExploortion
Human Feedback
Input: policy æ. goal selector fo. goal g, data
1: Input: Human H. goal distribution p(g)
buffer D
2: Initialize policy म. goal selector fa. data
gb ~ Sampleerreadcuunb(//
D)
buffer D, goal selector buffer
3: D, + [
3: while True do
4: for i= = do
4: Sample goal g ~~ p(g)
5:
5: D, PolieyFxxplorationnfs G. fo. D)
6: while NOT stopped (see G.I do
6: D < DU RelabeTTraecctrry
7: Take action TTa 9b
7:
TrainPolicy D) (4)
8: Execute Traudom for H timesteps
8: G + G U CollectFeedback (D, DDHH. H.
4.2
Add to D, without redundant states
9: fe TraiiGGolSSelector/ ()
10: return D,
5
Experimental Evaluation
In this work, we show that HuGE learns to successfully accomplish long-horizon tasks, and tasks with large combinatorial exploration spaces through little human supervision. To demonstrate these experimentally, we test on several goal domains in simulation, shown in in the MuJoCo (54) and PyBullet [17] simulators where compare against late-of-thhearr baselines. Furthermore, we show the benefits of our method by learning policies directly on a real-world LoCoBot robot. We additionally show the ability for HuGE be used with uncurated, large-scale crowdsourcing setup with non-expert human supervisors. With these experiments, we show 1) HuGE outperforms prior work solving long horizon goal-reaching tasks in simulation; 2) HuGE is suitable for collecting crowdsourced, noisy and asynchronous feedback from humans all over the world with different backgrounds and education levels; 3) HuGE is suited to learning in the real world with robotic platform in practical time-scales, directly from visual inputs.
Bandu
Block stacking
Kitchen
Pusher two walls
Maze
Four Rooms
Figure 4: Six simulation benchmarks where we test HuGE and compare against baselines. Bandu, Block Stacking, Kitchen, and Pusher, are long- long-horizon manipulation tasks; Four rooms and Maze are 2D navigation tasks, Appendix D

---

Four Roome
Pusher with walle
Maze
2380
Number of
Kitchen
Block Stacking
Bandu
5x
Number stepe
15%
2B
1004
123h
of
2304
2004
300.
Models
PEDBLE
Oracle
DDL
GO-Explae-GGSL
Figure 5: Success curves of HuGE on proposed benchmarks compared the baselines. HuGE outperforms the rest of baselines, some which cannot solve the environment while converging to the oracle accuracy. For those curves that are not visible, it means they never succeeded and hence are all at O (see (sce|E.10 I0 distance curves). The curves are the average of 4 runs, and the shaded region corresponds to the standard deviation.
5.1 Learning Goal-Conditioned Policies with Synthetic Human-in-the-Tne Feedback in Simulation
We consider goal-reaching problems in the six domains shown Fig These are domains with non-trivial exploration challenges the agents must assemble structure in a specific order without breaking it, navigate around walls, etc., and purely random exploration is unlikely to succeed. We evaluate HuGE compared to the baselines described in Appendix EJ these domains. These baselines were chosen compare HuGE with methods that perform different types of exploration, hindsight relabeling, and methods that use human preferences to learn reward function instead, to highlight the benefits of decoupling exploration from human preferences and policy learning. We report the number of final goals reached successfully in Fig 5 as learning progresses. Only in these and the analysis experiments, the human feedback synthetic (see Appendix D).
In Figure 5 and in Appendix with further detail. we show HuGE learning a goal selector model to perform goal selection (Ours) matches, convergence, the performance of using an Oracle always choosing the best goal to explore. Guiding exploration (Ours) is significantly better than techniques that perform indiscriminate exploration (Go-Explor++GCSS methods that purely rely on policy generalization and stochasticity to perform exploration and do not explicitly expand the frontier (Inverse Models) [ 47 24 which fail in complex exploration domains. coal-conditioned reinforcement learning methods with PPO [52) and sparse rewards do not experience enough reward signals to actually learn directed behavior. PPO with dense reward, we use the same reward as for generating the synthetic human labels and we observe, that in most cases the performance lower. The reason that we did not any reward engineering for either PPO HuGE, showing that Ours is more robust simpler underlying reward functions than PPO, we analyze these results further in Appendix E.22 Similarly, using human feedback to bias exploration Ours beats using it to learn reward function (Human Preferences) [6]. Finally,in Appendix we show that pretraining HuGE with noisy trajectories (BC + Ours) gives a significant decrease the number of timesteps needed to succeed and beats plain Behavior Cloning (BC) and also compared to other baselines with demonstrations.
37 Ours also beats
5.2 Learning Goal-Conditioned Policies with large-scale crowdsourced data collection
In this section, we show HuGE works on the kitchen environment from crowdsourced data collection of 109 non-experts annotators labeling asynchronously and from all over the world, as shown in Figure 6 We spanned across three continents, having annotators living in 13 different countries, with ages ranging from 18 65+ and a variety of academic backgrounds, and we collected 1600 labels, refer the reader to B. and B.2 for more details about the demographics. Each annotator

---

Kitchen [Human Experiment)
the Gook Grl Condil Exploradion Fowddwoth
N
(human demos)
30%
-Ora (smaddsouraa 5 demoee)
30%
-Ours (symiheiic cemos)
05
114
EM
PM
No data
slops
Figure 6: left: Crowdsourcing experiment learning for the kitchen, middle: human annotators spanned 3 continents and countries, right: screenshot of interface data collection.
could provide labels at any time during the day, recommendation which took on average less than 2 minutes of their time. We collected a total of 2678 labels for this crowdsourcing experiment beating Oury when labels are synthetically generated, This indicates that the crowdsourced data might provide more information than the simplistic reward function, designed to provide the synthetic feedback. We repeated the experiment collecting just 1670 labels from annotators and saw that the crowdsoureing experiment also yield better results, suggesting that wider variety of feedback does not damage performance. We refer the reader Appendix (B] for additional results of HuGE from real human feedback on other simulated benchmarks, details on the platform we developed for these experiments, and where we clarify that this study was approved by the Institutional Review Board.
provide 30 labels,
5.3 Learning Goal-Conditioned Policies in the real world
Pick & place in the real world
steps
2600 stens
5000 staps Training steps
7500
X000O
Number
130
Drawing in the real world
s
2000
StApS
Numberr chinae
Training steps
Figure Accomplished goals the 5 different evaluation episodes throughout training the real world.
HuGE's s qualities of being robust noisy feedback and requiring minimal and asynchronous human supervision together with its self-supervised policy learning nature and the capability to be pretrained from trajectory demonstrations makes it suitable for learning in the real world. As shown in Fig 7 HuGE can learn policies for pick and place and drawing in the real world with a LoCoBot (38). Some adaptations
made run HuGE on real hardware such changing the state space images, instead of point space. In Appendix A we provide more results on HuGE from image space in the real world and simulation. We also pretrained our policy with 5 trajectory demonstrations (Appendix C For both experiments, the robots learned directly on the real hardware. In the pick and place experiment, we collected around 130 labels across 20 hours of real-world robot training, whereas in the drawing one, we collected 150 labels across 6 hours.
5.4 Ablation Analysis
Lastly, we conducted number of quantitative analysis experiments to better appreciate and convey the benefits of HuGE. In Figure we show that learning parametric goal selector (Ours) is more sample efficient than directly using the human-selected goals (DDI. (77). More concretely, learning
parametric goal selector needs 50% less human annotations than DDL (27). The underlying

---

idea is that by deriving a goal selector from human feedback, we extract recurrent patterns and generalizations that can be redeployed in future rollouts. Conversely, without learning from this goal selector, we would overlook these patterns, leading to more frequent queries for human input.
One of the properties of HuGE is that it works from noisy human feedback. In Figure B we observe that despite adding large amounts of noise on the human annotations, HuGE still converges to the optimal solution. This supports the idea that when the human feedback noisier, exploration will be less efficient and will take longer to converge nevertheless, still converging to an optimal policy.
Furthermore, in Figure 8 (right), we provide visualization explaining why optimizing over the reward function (Human Preferences [16]) can provide non-optimal solutions. We observe reward functions learned from noisy human feedback are suboptimal, with multiple local maxima, and hence optimizing over them will potentially lead to the policy getting stuck in local optima. On the other hand, HuGE, which samples goals from this noisy learned reward function will still converge the optimal solution. The explanation is that the goal selector will steer exploration towards the three modes uniformly, making exploration suboptimal but still converging an optimal goal-conditioned policy reaching each one of the modes, the goal included.
In Appendix F. we explain further analysis as the trade-off between the amount of human querying and sample complexity of HuGE, as well as showing that HuGE is also robust to incomplete feedback and further details of its robustness to noisy underlying reward functions compared to PPO.
Increased feedback efficiency by leaining a goel selector
Timee to succnss for differrnt injectad noise
Robust to Multimoda ity
3000
VIDES
1500
Nose
100
100
Mathed +008
15 splppddd
100 sdladdes
500 apiicds
Outs
Freqqareyy
Figure 8: (left): Learning a goal selector (Ours) needs on average 50% fewer labels than not (DDL) (middle): As the noise in the feedback increases, so will the number of timesteps to succeed. however HuGE still finds solution. (right): HuGE is robust to noisy goal selectors since trajectories going to each mode will be sampled while we ran RL the policy would become biased and fail. See Appendix |F for more details.
Discussion
In this work, we build from the insight of decoupling human feedback from policy learning. We introduced HuGE, that guides exploration by leveraging small amounts of noisy and asynchronous human feedback, improving upon indiscriminate exploration or relying on hindsight generalization. Moreover, the policy training using self-supervised learning remains disentangled from the human feedback and makes it robust to asynchronous and noisy human feedback. We firmly believe this insight is key to learn policies from crowdsourced human feedback, where the noise in the labels due to non- con-expertise and variety of the annotators will be significant. As we show, when the annotations are noisy, optimizing over the learned noisy reward function using standard RI. techniques will fail. However, once we disentangle the human feedback from the learned policy, and only use this to softly guide exploration, we can still learn optimal policies despite the noise. We rigurously demonstrate that HuGE is able to solve difficult exploration problems for various control tasks learning in both the real world and in simulation. Moreover, we run crowdsourced data collection experiment with 109 human annotators living across three continents, with a wide variety of backgrounds, education level, ages, and we show HuGE succeeds in learning optimal policies. Finally, we provide ablations and analysis on the design decisions and better show the benefits of HuGE.
There are number of directions for future work that are very exciting building on HuGE. On an immediate note, adapting HuGE to work in reset-free environments so that we can scale up the applications on learning on the real robot is the most promising. It is with excitement that we would like to see this system scaled to fine-tune foundation models in robotics. behaviors not just from bianry comparative feedback, but from richer forms of communication such as language or physical corrections or vector valued feedback would be intriguing as well.
the future, aiming to learn
10

---

7 Acknowledgements
We thank all of the participants our human studies that gave of their time provide labels. We thank the members of the Improbable Al Lab and the WEIRD Lab for their helpful feedback and insightful discussions. We also thank the reviewers for their insightful comments and helping us make the paper stronger.
The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper. This research supported by MIT-IBM Watson AI Lab.
8 Contributions
Marcel Torne and Abhishek Gupta jointly conceived the project. Marcel set up the simulation and training code, designed and conducted experiments in simulation, designed and implemented the interface for collecting human feedback, led the human experiments. conducted the ablations, and made the figures. Marcel led the manuscript writing together with Abhishek. Max Balsells designed and conducted the experiments in the real-world, integrated the vision models in the code, helped running some of the simulation experiments and ablations and helped writing the manuscript. Zihan Wang provided feedback on the manuscript. Samedh Desai helped Max setting up the real-world experiments. Tao Chen was involved in the initial research discussions and provided feedback on the paper. Pulkit Agrawal was involved in research discussions, contributed some of the main ideas behind the project and provided feedback on the writing and positioning of the work. Abhishek Gupta conceived the project jointly with Mareel, led the manuscript writing together with Marcel, and provided the main overall advising.
References
[1] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and Levine. Learning to poke by poking: Experiential learning of intuitive physics, Advances in neural information processing systems, 29, 2016.
[2] M. Andrychowiez, D. Crow, A. Ray, Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In I. Guyon, U. von Luxburg. S. Bengio, H. M. Wallach, R. Fergus. S. V. N. Vishwanathan, and Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5048-5058, 2017.
[3] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, and W. Zaremba. Hindsight experience replay. Advances in Neurul Information Processing Systems, pages 5048-5058, 2017.
[4] M. G. Azar, I. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 263-272. PMLR, 2017.
[5] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count- based exploration and intrinsic motivation. In Advances Neural Information Processing Systems, pages 1471-1479, 2016
[6] A. Billard and G. Hayes. Learning to communicate through imitation in autonomous robots. In W. Gerstner, A. Germond, M. Hasler, and J. Nicoud, editors, Artificial Neural Networks ICANN 197, 7th International Conference, Lausanne, Switzerland, October 8-10, 1997, Proceedings, volume 1327 of Lecture Notes in Computer Science. pages 763-768. Springer, 1997.
[7] E. Biyik. Learning preferences for interactive autonomy. CoRR, abs/2210. 2022.
[8] E. Biyik and D. Sadigh. Batch active preference-based learning reward functions. In 2nd Annual Conference Robot Leurning, CoRL 2018, Zärich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages 519-528, PMLR, 2018.
11

---

[9] R. I. Brafman and M. Tennenholtz. R-MAX- A general polynomial time algorithm for near- optimal reinforcement learning. J. Mach. Learn. Res., 3:213-231, 2002.
[10] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint 18100 12894, 2018
[11] S. Cabi, S. G. Colmenarejo, A. Novikov, K. Konyushkova, S. E. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, O. Sushkov, D. Barker, J. Scholz, M. Denil, N. de Freitas, and Z. Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In M. Toussaint, A. Bicchi, and T. Hermans, editors, Robotics: Science and Systems XVI, Virtual Event / Corvalis, Oregon, USA, July 12-16, 2020, 2020.
[12] T. Cederborg, Grover, C. L. I. Jr., and A. L. Thomaz. Policy shaping with human teachers. In Q. Yang and M. J. Wooldridge, editors, Proceedings of the Twenty-Fourt International Joins Conference Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015, pages 3366-3372 3372. AAAT Press, 2015.
[13] E. Chen, Z.-W. Hong, Pajarinen, and P. Agrawal. Redeeming intrinsic rewards via constrained optimization. Advances in Neural Information Processing Systems, 35:4996-5008, 2022.
[14] T. Chen, S. Gupta, and A. Gupta. Learning exploration policies for navigation. arXiv preprint 1933v:1903.01959, 2019.
[15] T. Chen, S. Kornblith, M. Norouzi. and G. E. Hinton. A simple framework for contrastive learning of visual representations. CoRR, abs/2002.05709, 2020.
[16] P. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences, 2017.
[17] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. In Gitflub Repository, pages 5026-5033, 2016.
[18] Y. Cui, P. Koppol, H. Admoni, S. Niekum, R. Simmons, A. Steinfeld, and T. Fitzgerald. Un- derstanding the relationship between interactions and outcomes in human-in-the-loor machine learning. In Z. Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Monireal, Canada, 19-27 August 2021, pages 4382-4391. ijcai.org, 2021.
[19] T. Davchev, O. O. Sushkov, J. Regli, S. Schaal, Y. Aytar, M. Wulfmeier, and J. Scholz. Wish you were here: Hindsight goal selection for long-horizon dexterous manipulation. In The Tenth International Conference on Learning Representattons, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[20] R. Devidze, P. Kamalaruban, and A. Singla. Exploration-guider reward shaping for reinforce- ment learning under sparse rewards. Advances in Neural Information Processing Systems, 35:5829-5842. 2022.
[21] A. Ecoffet, Huizinga, J. Lehman, K. O. Stanley, and Clune. Go-explore: a new approach for hard-exploration problems. CoRR, bbss1991..1995, 2019.
[22] B. Eysenbach, R. Salakhutdinov, and S. Levine. Search on the replay buffer: Bridging planning and reinforcement learning. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bue, E. B. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 15231, 2019.
[23] S. Fujimoto, H. van Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. CoRR, abs/ abss/800.099. 110.1/900.997..000 2018.
[24] D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devin, Eysenbach, and Levine. Learning to reach goals without reinforcement learning. CoRR, abs/1912.0600 2019.
12

---

[25] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. L. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference on Robot Learning, CORI. 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1025-1037. PMLR, 2019.
[26] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, (tbs/1801.012990 2018.
[27] K. Hartikainen, X. Geng, T. Haarnoja, and S. Levine. Dynamical distance learning for unsuper- vised and cemi-supervised skill discovery. CoRR, abs/1900.08825, 2019.
[28] L.. P. Kaelbling. Learning achieve goals. Citeseer, 2013.
[29] W. B. Knox and P. Stone. TAMER: Training Agent Manually via Evaluative Reinforcement.
In IEEE 7th International Conference on Development and Learning, August 2008.
[30] K. Lee, L. Smith, and P. Abbeel. Pebble: Fredback-efficien interactive reinforcement learning via relabeling experience and unsupervised pre-traie training.n In International Conference on Machine Learning, 2021.
[31] K. Lee, L. M. Smith, and P. Abbeel. PEBBLE: feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 6152-6163. PMLR, 2021.
[32] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, 0bs/2005.011433 2020.
[33] A. Levy, G. D. Konidaris, R. P. Jr., and K. Saenko. Learning multi-level hierarchies with hindsight. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[34] M. Liu, M. Zhu, and W. Zhang. Goal-conditioned reinforcement learning: Problems and solutions. In L. D. Raedt, editor, Proceedings of the Thirty-Firs: International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 5502-5511 ijeai.org, 2022.
[35] C. Lynch, M. Khansari, T. Xiao, V. Kumar, Tompson, S. Levine, and Sermanet. Learning latent plans from play. In L. P. Kaelbling, D. Kragic, and K. Sugiura, editors, 3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1113-1132. PMLR, 2019.
[36] C. Lynch, A. Wahid, J. Tompson, T. Ding, J. Betker, R. Baruch, T. Armstrong, and P. Florence. Interactive language: Talking to robots in real time. CoRR, abs/2210.06407, 2022.
[37] R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak. Discovering and achieving goals via world models. In M. Ranzato, A. Beygelzimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages (4379-2439) 2021.
[38] A. Murali, Chen, K. Alwala, D. Gandhi, L. Pinto, Gupta, and A. Gupta. Py- robot: An open-source robotics framework for research and benchmarking. arXiv preprint a9Xiv:1906.0 08236. 2019.
[39] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning.
In Advances in Neural Information Processing Systems, pages 3303-3313, 2018
[40] A. Nair, D. Chen, Agrawal, Isola, P. Abbeel, J. Malik, and S. Levine. Combining self-supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE International Conference on Roboties and Automation, ICRA 20/7 Singapore, Singapore, May
29 - June 3, 2017, pages 2146-2153, IEEE, 2017.
13

---

[41] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine Combining self-supervised learning and imitation for aasion-based rope manipulation. In 2017 lEEE international conference robotics and automation (ICRA), pages 2146-2153. IEER, 2017.
[42] A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 2209-9220, 2018.
[43] I. Osband, C. Blundell, A. Pritzel, and B. V. Roy. Deep exploration via bootstrapped DQN. In D. Lee, M. Sugiyama, U. von Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4026-4034, 2016.
[44] P.-YY Oudeyer, Kaplan, and V. V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE Iransactions on evolutionary computation, 11 11(2):265-286, 2007.
[45] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback, 2022.
[46] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 16-17, 2017.
[47] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell. Zero-shot visual imitation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 2050-2053, 2018.
[48] V. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine. Skew-fit: State-covering self- supervised reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 7783-7792. PMLR, 2020.
[49] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:
2017.
[50] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Wiele, V. Mnih, N. Heess, and J. T. Springenberg. Learning by playing solving sparse reward tasks from scratch. In International conference machine learning, pages 4344-4353, PMLR, 2018.
[51] J. Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference simulation of adaptive behavior: From animals to animats, pages 222-227, 1991.
[52] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization 2017.
algorithms. CoRR,
[53] Sharma, B. Sundaralingam, Blukis, C. Paxton, Hermans, A. Torralba, Andreas, and
D. Fox. Correcting robot plans with natural language feedback. CoRR, abs/2204. 051866, 2022.
[54] E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[55] G. Warnell, N. R. Waytowich, V. Lawhern, and P. Stone. Deep TAMER: interactive agent shaping in high-dimensional state spaces. In AAAI, 2018.
[56] G. Yang, A. Ajay, and P. Agrawal. Overcoming the spectral bias of neural value approximation. In International Conference on Learning Representations, 2022.
[57] A. Zeng, P. Florence, J. Tompson, S. Welker, J. Chien, M. Attarian, T. Armstrong, I. Krasin, D. Duong, V. Sindhwani, and Lee. Transporter networks: Rearranging the visual world for robotic manipulation. Conference on Robot Learning (CoRL), 2020.
14

---

Next, we provide additional details of our work. More concretely:
Appendix A Real-Robot experiments: provides more details on the real-robot experiments and on HuGE working from images.
Appendix B Real-Human experiments: shows all the details of the real human experiments presenting results for both the crowdsoureing experiment as well as experiments on more benchmarks with fewer annotators.
Appendix HuGE from demos: show HuGE is compatible with warm starting from demonstrations.
Appendix D Simulatee Benchmarks: we provide further details on the simulated bench- marks used in this work.
we provide details about the baselines together with more detailed learning curves of the baselines on the simulated benchmarks.
Appendix Baselines:
Appendix Further Analysis and Ablations: we provide more insights on where the benefits of HuGE come from, well as provide some ablations the method.
Appendix Implementation Details: we provide further implementation details, hyperpa- rameters and resources used.
The code is available at http:///wwrooolllaaaaaaaaaaaaaaa
A Real-Robot Experiments
HuGE's qualities make suitable to learn policies directly the real world. However, we adapted the method with respect to the simulated experiments in Fig[5] The main change consisted in changing the state space to image space instead point space. Next, we show HuGE works from image space in two of the simulated environments, four rooms and block stacking.
A.1 HuGE from images
Adapting HuGE to work from image space was trivial process, the goal selector and policy networks were modified introducing first encoding network consisting of convolutional layers (with stride and kernel size 5) to map the input image to lower dimension space and then we passed it through an MLP to predict the score and action respectively.
Learning stopping criteria: When using the point state space, we could easily detect whether the policy stopped, indicating it reached the target goal, or that it got stuck, to then start random exploration from there. This could be done by computing the Euclidean distance and setting a small enough threshold. This is more difficult when working from image state space. What did was train an image classifier Φ(51, 82) that predicts whether the two images correspond to states close in space (i.e. the state corresponding image 82 be reached from the state corresponding to image within felose timesteps). We trained ø by using contrastive learning [15]. In particular, we sampled images from our replay buffer and assigned the corresponding label based on their distance in timesteps: ((s;, 55) = 1 if i jl < felose and 0 if j - jjl 2 thr- Based on the premise that, in most cases, images obtained far away in time, will probably correspond to states that take longer than felose timesteps to reach, if we were to act optimally.
A.2 Results in the real-world
For the real robot experiment, we used a LoCoBot with a WX-200 arm.
Pick and place in the real world: The state space consisted of RGB images of 64 64 pixels, and the action space was continuous with dimension 2, representing an absolute position in the space (z. y) from which to predict grasping point in even timesteps, or a dropping point in odd timesteps. For the experiment to be succeed reasonable amount of time, we pretrained the policy and the goal selector by using 5, sub-optimal demonstrations. The robot was trained for around 30h, during which, 130 labels were provided via the interface shown B.4 by one annotator. Finally, we used reset mechanism to pull the socks to the same corners, though, it had some stochasticity.
15

---

Four Rooms
Block staking
1
Numbor blocks
Distance goo
0.8
0.8
1.5
Orade
-uus
0.2M
Number of steps
Number steps
Figure A.I: Success rate for the four rooms (left) and block stacking (right) using images as input space for both the policy and goal selector.
Pick & place in the real world
stops
soops
Training steps
Drawing in the real world
ssapps
1000 stops
staps Training stops
staps
Number
2:00
A000
Figure A.2: Accomplished goals at the end of different evaluation episodes along training on the real world to pick and place, and draw the letter U in the real world.
Drawing in the real world: The state space consisted of RGB images of 64 64 pixels, and the action space was discrete, encoding a total of actions: no movement and moving across the two axis on the plane in polar coordinates (i.e. increasing r, decreasing and moving fixed amount clockwise or counterclockwise), to move the end effector with the brush. An episode consisted of 12 timesteps. For the experiment to ran in reasonable amount of time, we pretrained the policy and the goal selector by using 5, sub-optimal demonstrations. The robot was trained for around 6h, during which, 150 labels were provided via the interface shown in B.4 by one annotator. Finally, the reset was done by using the erase mechanism in this drawing boards and moving it with the arm by using a script. As a final note, this environment we had perform few exploration steps and slowly increase the frontier. This is because in this environment there is only one optimal solution (the actions taken must be exactly the optimal ones, due to the fact that all past actions within the episode will affect the current state of the board), in particular, any non-optimal action will leave trace, making that trajectory not that useful for the policy to learn from
Human Alignment evaluation for drawing in the real world: Designing a reward function for drawing a hard and tedious labor. HuGE does not need a reward function and we can fully leverage human feedback to learn this behavior as shown in A.2] Without reward function evaluation cannot be performed either. For this reason, we defined this "Human Alignment Score" that basically consists in querying humans and asking them for a score between 0 and 10 of how well the robot draw the target picture. In the case of the drawing experiments, we asked 2 annotators to label the performance of the robot drawing the letter U with score from 0 to 10. This score was only used for evaluation and is the metric used to plot the drawing plot A.2
16

---

B Real-Human experiments
Kitchen
Block Stacking
Reeto
Succrs 94000
-Ouns
-Oun ie temes)
-Ous systhaic dersa!
-Ous Gytbhkk
Number of staps
Number of staps
Four Rooms
Pusher (Human Expermment)
Complex Maze
Jaans 27020ber1
Nfmberrollan staps
Number (TSXps
Buiserr coscerofisaag ofistags
Figure B.3: Learning Progress with Human-in-the-Loopp Feedback for the kitchen, block stacking, four rooms navigation, pusher walls and complex maze.
In this section, we give more details on how we ran the human experiment. We designed a simple interface shown in Figure B.4] We can see the two states to be compared in blue and red and the goal we aim to achieve in green. Then the annotator has to decide which one of the two states is closer to the given goal and provide feedback by clicking either on the blue red button. In the case in which the annotator is undecided, they can click on the gray button that simply skips the current case. Finally, if the annotator does provide any feedback after 30 seconds of being presented with the scenario, we skip the current batch of labeling and continue with training the policy. With this, we when
take advantage of the properties of our method and continue training the policy labels are given.
In Figure B.3 we share again the results obtained with the human experiment on a larger scale. We ran both experiments using the same frequency of labeling and number labels per batch. particular, We labeled every 50 rollout trajectories and queried the annotators for 20 labels. These parameters were identified through empirical experiments.
Human Guided Exploran (HUGE)
Breadcrumbs to the Goal: Buuman-in-the-loo Feedback
Exploration from
Kitche
SCMT KNOW
NIGHT
Figure B.4: Screenshot of the interface from our proposed crowdsourcing platform. It shows comparison of two image states of the kitchen environment, and the user needs to click one of the three buttons below depending on their answer of which one is best: Left/I don't know/right
17

---

B.1 Details about the crowdsourcing experiment
Subject 109 subjects participated in this pilot crowdsoureing study. Subjects were recruited from the acquaintances of the collaborators. The average time to complete the study was about 2 minutes. The subjects participated voluntarily without financial remuneration. The participants age ranged from 18 to 65+ years old. Gender Male=58.795 Female=39.49 Non-binary=1.855 The participants were from 21 nationalities and participated from 13 distinct countries. More detailed information presented in tables B.1 and B.2) There is no reason to believe that subjects experienced any physical or mental risks the course of these studies.
Procedure This study was approved by the Institutional Review Board of the Massachusetts Institute of Technology protocol E-4967.
We provide the participants with the following detailed instructions:
Thank you very much for participating in this study. It should not take you more than a couple of minutes to complete. First of all click on the link sent you to get directed to the main page (B.4). you can either phone or your computer. Please, start by filling out the form for us to get an overview of the participants' demographic. The task consists of controlling robot to do different things in the kitchen: 1) open the slider on the right, 2) open the microwave on the right 3) open the hinge cabinet on the left. [We show video of successful trajectory]. To help us. we will present you two images and you need to tell us which one of the two images is closer to achieving the task. Click on the left/right button depending whether the left/right image is better. If some point you don't know which one is best please click the "I don't it know" button. [We present couple of examples demonstrating this]. We will show pairs images and after that, you will receive a message saying you completed the task. You can stop at any moment before that you want.
your
50%
No
Figure B.5: Heatmap on the country representation during our crowdsourcing experiment.
18

---

Metric
Percentage
Age group Current country of Residence
USA
41.3% (45)
Spain
30.3% (33)
India
8.3% (9)
Germany
6.4% (7)
Canada
2.8% (3)
France
2.8% (3)
Singapore
1.8% (2)
China
1.8% (2)
Andorra
0.9% (1)
Austria
0.9% (1)
Ireland
0.9% (1)
Switzerland
0.9% (1)
United Kingdom
0.9% (1)
Prefer say
0% (0)
Gender
Male
58.7% (64)
Female
39.4% (43)
Non-binary
1.8% (2)
Prefer not to answer
0% (0)
18-24
48.6% (53)
25-34
24.8% (27)
35-44
7.3% (8)
45-54
11.0% (12)
55-64
7.3% (8)
65+
0.9% (I)
Prefer answer
0% (0)
Education
Graduate or professional degree
39.4% (43)
College degree
33.9% (37)
High school or some college
20.2% (22)
Other
12.8% (14))
Prefer not to say
2.8% (3)
Table B.I: Demographics on the participants of the crowdsourced data collection experiment
19

---

Metric
Percentage
Nationality
Spain
26.6% (29)
USA
20.2% (22)
India
9.2% (10)
Germany
8.3% (9)
China
7.3% (8)
France
4.6% (5)
Mexico
3.7% (4)
Colombia
2.8% (3)
Switzerland
1.8% (2)
Hong Kong
1.8% (2)
Canada
1.8% (2)
Uruguay
0.9% (I)
Singapore
0.9% (1)
Russia
0.9%
Ireland
0.9% (L)
Lebanon
0.9% (I)
South Korea
0.9% (1)
Sweden
0.9% (1)
Andorra
0.9% (1)
Puerto rico
0.9% (1)
Israel
0.9% (1)
Prefer not say
0.9% (1)
Ethnicity
Hispanic, Latino or Spanish
38.5% (42)
Asian
28.4% (31)
White or Caucasian
24.5% (27)
Middle Eastern North African
3.7% (4)
South-east Asian
2.8% (3)
Black African American
0.9% (1)
Perfer not to say
0.9% (1)
Table B.2: Demographics on the participants of the crowdsoureed data collection experiment
20

---

C HuGE from demonstrations
As explained in Section 4.3 HuGE is suited easily incorporate demonstrations and therefore use these to warm start the method providing significant reduction in sample complexity over HuGE without demonstrations. Below we show comparison of HuGE against other baselines that use demonstrations.
Four Rooms
Pusher with wabts
Maze
Ax arimi
Namber of
Nummer otipp
Number ships
Kitchan
Block Stacking
Bandu
Number idace
Number of stapp
(0
Noress
Figure C.6: Comparison of HuGE warmstarted with demonstrations, We observe that significant speed up is provided compared to running HuGE without demonstrations. Furthermore, we observe improvement over simply learning from demonstrations (Behavior Cloning, BC) baselines. Sim- ilarly, HuGE with demonstrations continues to outperform the PPO and Human Preferences with demonstrations baselines.
21

---

D Simulated Benchmarks
Bandu
Contruamiton
Ren piece %i piaze picks pioco
Red and diue pleces elaced pioks groen
Planee goner placee fool piece ona
pleces geot contgurocion
Block Stacking
Strcking
Droca sorget
Stocks gesan
Kitchen
Opena
Opena
Pushers with walls
Strrrt Prothe Winile
Rearchw
Moits puck around
Moles puck around obstacie
Puck reaches
Figure D.7: Results of our method on four of the hardest benchmarks. From left to right, the timestep in the trajectory increases.
Block stacking
Kitchen
Pusher two walls
Maze
Four Rooms
Figure D.8: Results our method on four of the hardest benchmarks. From left right, the timestep in the trajectory increases.
22

---

In this section, we give more details on the benchmarks used to compare our method with the baselines. All of these benchmarks are variations of benchmarks presented in previous work. In general, we have made them harder to showcase the benefits of our method. More concretely, for each method, we will give an overview of the difficulties it has and we will present the reward function we designed to provide synthetic labels in our experiments.
1. Four rooms (small 2D Navigation): We consider goal-reaching problems in 2-D nav- igation problem in four rooms environment, shown in Fig D&8 The challenge in this environment is navigation through obstacles, which are unknown without exploration. The agent is initialized in the bottom right room and the goal is sampled from the top right room. The state observation of this environment is the absolute position of the agent in the world, i.c. a vector (zz, y), and the action space is discrete with 9 possible actions, encoding 8 directions of movement (parallel to the axis and diagonally), plus non-move action. To solve this benchmark the agent needs to traverse the two intermediate rooms to get to the target room, traversing total of four rooms.
The reward function in this case is the shaped distance between the state and the goal. This benchmark is a modification of the benchmarks proposed by [24).
2. Maze (large 2D Maze Navigation): We consider a second 2-D navigation problem in a maze environment. The additional challenge in this environment compared the previous one relies upon having a longer horizon (see Figure G.4). The agent is initialized in the green dot and has to reach the red dot. The state space is the absolute position of the agent in the maze, i.e. a vector (z, y), and the action space is the same as in the Four rooms one, i.e. discrete with dimension 9.
The reward function in this case is the shaped distance between the state and the goal.
3. Pusher two walls: This a robotic manipulation problem, where a Sawyer robotic arm pushes an obstacle in an environment with multiple obstacles. The puck and arm start in the configuration seen in Fig [.8] The task considered successful if the robotic manipulator brings the puck to the goal area, marked with red dot.
The state space of this environment consists of the position of the puck and the position of the arm. The action space is the control of the position of the robotic arm. It is also a 9-dimensional discrete action space where each one corresponds to a delta change in the position in 2D. This benchmark is a modification of the benchmarks proposed by (24). The reward function designed for this environment the following:
maz (distancecc_epcck
0.05) + distance_puck _gal
4. Sequential Kitchen Manipulation:
This benchmark is a harder robotic manipulation task where apart from being long horizon the agent needs to show three different skills solve the task. We operate a 7 DoF' Franka robot arm in a simulated kitchen, manipulating different cabinets, sliding doors, and other elements of the scene. The observation space consists of the position of the end effector and its rotation together with the joint states of the target objects. The action space consists in controlling the end effector position in 3D, we discretize it so the dimension is 27, and the control of the gripper and rotation of the arm. In our evaluation, we consider tasks where the goal to sequentially manipulate three elements in the kitchen environment- the sliding cabinet, the microwave and the hinge cabinet to target configurations. The reward function we use is the following:
distance(arm, hinge cabinet) - lhinge cabinet target joint- hinge cabinet current joint]. if slide cabinet and microwave opened distance(arm, microwave hinge) |microwave target joint- microwave current joint| bonus if slide cabinet opened distance(arm, slide cabinet hinge) (sldee cabinet target joint- slide cabinet current joint| 2bonus otherwise
(5)
5. Block Stacking: This domain is another long horizon robotic manipulation task, we operate a 6 DoF UR5 robot arm with a suction gripper as an end effector in a simulated tabletop configuration, stacking blocks. The observation space consists of the position of the end effector and the position of each block in 2D, and a bit indicating whether the hand is holding a block. This is continuous action space domain with dimension 2, where the agent will predict a grasp position it does not hold an object and a drop position if it holding an object. We consider the goal to be accomplished if the three blocs stacked in the correct
23

---

order (red, green, blue) on the correct fixed place on the table. Our code was inspired from the Ravens benchmark [57]. The reward function is the following:
distance(arm, blue block) distance(blue block, target goal). if red and green block at position distance(arm, green block) + distance(green block, target goal) + bonus, if red block at position distance(arm, red block) + distance(red block, target goal) 2bonus, otherwise
(6)
d(arm, blue) + d(blue, goal), if red and green at goal d(arm, green) + d(green, goal) + bonus, if red at goal d(arm, red) + d(red, goal) 2bonus, otherwise
6. Bandu: This domain is very similar to the block stacking. We operate a DoF URS robot arm with suction gripper as an end effector in simulated tabletop configuration. The observation space consists of the position of the end effector and the position of each block in 2D, and a bit indicating whether the hand holding block. This a continuous action space domain with dimension 2, where the agent will predict grasp position if it does not hold an object and drop position if holding an object. We consider the goal to be accomplished if the four blocs are stacked in the target configuration building the castle like structure seen in Figure D.8 Our code was inspired from the Ravens benchmark [57]. The reward function is the following:
distance(arm, yellow star) distance(yellow star, target yellow star), if all except star position distance(arm, green block) distance(blue green block, target green block) - bonus if red and blue blocks at position distance(arm, blue triangle) distance(blue triangle, target blue triangle) 2bonus if red cylinder at position
r
distance(arm, red cylinder) distance(red cylinder, target red cylinder) 3bonus otherwise
More details about how these benchmarks were run, such as the number of episodes we ran the benchmarks for, are presented in Section G
24

---

E Baselines
We compare HuGE relevant baselines from prior work.
Inverse Models: We compare with the iterative supervised learning algorithm for goal-
reaching introduced in 224, consisting of relaeigtt without additional exploration (GCSL).
2. Learning from Human Preferences:
We consider the technique introduced in [16]. which learns a goal-agnostic reward model using binary cross-entropy. This learned reward is then combined with an on-policy RL algorithm [22 learn the policy.
3. DDL: Dynamical Distance Learning 27 proposes method learn goal-conditioned reward function by regressing on the time distance between states achieved in the same trajectory. human synchronously provides preferences which state brings the agent closest to the goal, note that no goal selector is being learned. The policy is then trained to maximize the learned reward get this selected state.
4. Go-Explore/LEXA. We compared with version of goal-reaching with indiscriminate exploration. In particular, we perform frontier goal selection by identifying goals with the lowest densities. The policy returns to these states and perform random exploration from there. This is equivalent to performing indiscriminate exploration.
5. Proximal Policy Optimization: We compare with an on-policy algorithm 52 with both a standard sparse and dense reward to directly optimize the goal-reaching objective.
6. Behavior Cloning: Supervised learning batch of expert trajectories. In our experiments we use expert trajectories.
7. Behavior Cloning + Ours:
We pretrain the policy using imitation learning and we warm start our goal selector by training it from the expert trajectories. Given two random states in the same expert trajectory we add them into the training data for the goal selector, setting the state further in time as closest to the goal.
These baselines are chosen to compare HuGE with methods that perform pure exploration, hindsight relabeling, and human preferences without being goal conditioned to highlight the benefits of combining goal-driven self-supervision with human-in-the-loop exploration guidance.
Inverse
Models
Collapse
(GCSL)
Uniform
Frontier
Overexploration
Expansion
ILEXA.
Go-Explore)
Goa!
Directed
Frontier
Start
Expansion
(ours)
Figure E.9; Failure modes of exploration algorithms for goal-reaching. Inverse models (top) col- lapses and does not discover the target room (second room at the top). Uniform frontier expansion (middle) does reach the target room, but to get there it visits all possible rooms, since exploration indiscriminate. Directed frontier expansion (bottom, ours) reaches the target room much faster by leveraging human signal on direction. Training epochs increase from left right. Each subfigure an aerial view of floor with 9 rooms, with multiple trajectories, each one in different color.
25

---

Four Rooms
Pusher with walls.
Maos
M
200
Roo
0.2
12G
2006
503
6XA
G00k
6oo
1.0
Number
Kitchen
Block Stackking
Bandu
12
Disance br
Disarra to
m
100
500
25
Number of stops
demass
(dense)
Human Preferences
Oracle
DDL
BC Ours 15 demost
(sparse)
LEXV-iike
Figure E. 10:0: Distance to the goal for each method on different benchmarks. We note that the LEXA-like exploration strategy was only implemented on the four rooms benchmark.
For the sake of concreteness, we will study two simple schemes from prior work on solving goal-
reaching problems self-supervision via goal conditioned supervised learning [4 (as described in Section 3 and reinforcement learning with density based exploration [37] Exploration in GCSL relies generalization of the policy across goals, while density based exploration rewards exploring the most novel states. We show these algorithms can fail in different ways for a simple maze environment shown in Fig E.9 where the agent starts in the middle room and must reach goals commanded in the top middle room.
As shown in Fig E.9 GCSL exploration quickly collapses in the maze environment. This can be understood by noticing that self-supervised training on goals in the bottom right corner room or even the bottom left corner are. Instead of navigating the agent around the walls, the policy generalization suggests that the agent simply go into the wall shown in Fig E.9
does not extrapolate to the top right corner, where the commanded goals
Exploration methods are meant tackle this kind degenerate exploration, by encouraging visitation of less frequently visited goals at the "frontier" of visited states. When applied to the goal-reaching problem, Fig E.9 we see that while the exploration is not degenerate, exploration indiscriminate in that it explores both sides of the maze even though commanded goals are only down one particular path. While this will eventually succeed, incurs a significant cost of redundant exploration by going down redundant paths.
This suggests that frontier expansion needed like exploration methods, but should ideally be done in directed way towards goals of interest. In Figure E.9 we see how this directed exploration could be useful and reduce sample complexity, by removing the need for indiscriminate frontier expansion. We show how small amount of relatively cheap human feedback can be leveraged to guide this exploration.
E.1 Detailed training curves
For some the runs the plot the success could be misleading, in the sense that, despite not achieving the goal, the algorithms may still learn how to almost solve the task, or at least gained some knowledge about how to approach it. Figure E. shows for each of the runs, the distance to the goal, which corresponds to where is the reward of the corresponding benchmark, as described in Section
For example, by looking at Figure E. we can see that despite the fact that the Human Preferences wasn't 't able to complete some of the tasks, such as Four Rooms, Pusher with walls or Maze, still got some insight on how approach it, getting much closer to the goal than the other methods that failed.
26

---

Benchmark
Oracle
Ours
GCSL
Human Preferences
DDL
PPO
PPO (dense)
GCSL
(sparse)
+ Explore
Go-
4 rooms
0.02 + 0.01
0.02 0.00
1.15 0.67
0.48 + 0.39
0.45 0.28
1.45 0.13
0.05 0.02
0.13 0.11
Maze
0.4 = 0.3
0.8 + 0.3
2.2
18.5 + 5.6
8.54 + 10.6
30.4 + 0.7
0.0 0.2
1.1 +.1 0.14
Pusher
0.06 + 0.00
0.11 + 0.04
0.85 0.11
0.26 + 0.03
0.69 + 0.06
0.72 = 0.06
0.27 + 0.00
0.50 = 0.29
Kitchen
1.06 + 0.32
0.67 0.21
11.72-0.11
3.43 4.37
11.28
7.63 4.96
2.84 + 2.72
10.97
0.13
Stacking
0.1 = 0.2
0.0 0.0
4.1 2.3
6.5
6.6 +.1.
6.7 0.0
6.6 + 0.0
3.76 2.66
Bandu
1.00 0.53
0.36 + 0.73
12.87=0.01
12.54
12.63+0.21
12.75=0.001
12.75=0.01
10.7 = 1.51
Benchmark
PEBBLE
BC (5 de- mos)
BC Ours (5 demos)
4 rooms Maze
0.59 0.1
0.45 = 0.46
0.04 = 0.00
11.95
+
2.25 + 1.51
0.87 + 1.02
2.56
Pusher
0.185
+
0.25 0.09
0.08 + 0.01
0.04
Kitchen
10.48
11.38
0.87 1.02
0.01
0.00
Stacking
4.24
1.91
1.02
0.01 = 0.00
Bandu
13.5 3.25
4.21
5.47
1.87 + 0.4
Figure E.11: Average distance and standard deviation across 4 seeds for the different baselines implemented to compare against HugRL. We see that HugRL consistently succeeds (in bold) to solve all benchmarks when most other baselines do not. The oracle would be the upper bound that we could hope to achieve, since in this case labels are provided all the time, and the goal selector substituted by a precise distance function.
E.2 Analysis of PPO vs HuGE
Pusher
Block Stacking
0.8
2.5
Sucgess Ralio
Success Ralio
2
0.8
50k
100k Number steps
Number steps
Ours (finetuned)
Preforences (fnctuned)
PPO (fineluned]
Ours (nnt-finctuned)
....
Human Proferances (not-finctured)
.... PPO (nol-fineluned]
Figure 121 Comparison of the PPO and Human Preferences baselines against HuGE when using carefully tuned reward function and not on the pusher and block stacking environments.
As have shown throughout this manuscript, HuGE is robust to underlying noisy human preferences, which means it's robust to noisy underlying reward functions. In Figure [F.22 we provide comparison between the performances of PPO and HuGE when the reward is finetuned against when it's not. We observe that while HuGE succeeds in both cases, PPO only succeeds when the reward function carefully tuned.
In the pusher experiment, the reward finetuning consisted in doing some hyperparameter tuning, basically, modifying the position of the keypoints and adding a bonus for solving each keypoint. In the block stacking, consisted in adding an extra reward term seen in Figure E.13
Finally, wanted to understand why even when the reward function is finetuned and both HuGE and PPO obtain optimal policies, why is HuGE more sample efficient than PPO. To analyze this we ran comparison gradient variance using HUGE compared to gradient variance with PPO, comparing
27

---

sef Firetnol (ist.iistttttiiiioooo glal..96))
for rangeCrun (runsblock)): nene(ot]_ contirus
def standard distance(cb)....o goal..pos): rangoCrum blocks):
norn(obj_pos[t] gool__pos) concinue
none(oR1_poss (a.p31)
reture
somoobjuues0) ec.206)
return
nomi(are_pos pos ab1_pos[L]]
nomm(ohj_oo(11 goo)_
bonus (self. mww.b.
nore(armppe dd).oos((])
norn(obj_ 238l.206)
return
retur
Figure 13: Comparison of the original non-finetuned reward function (left) against the finetuned one (right) for the block stacking experiment.
the average pairwise cosine similarity across mini-batches to measure gradient variance suggested in In Figure E. 14). We find that training the gradient similarity is significantly larger in HuGE than in PPO, potentially explaining improved sample complexity. Besides this, other components in HuGE may also lead to faster learning generalization across states that come from self-supervised hindsight relabeling and hindsight relabeled policy learning off-policy while PPO is on-policy. However, we donot want to claim that the synthetic variant of HuGE is fundamentally better optimization algorithm than PPO. Given a carefully shaped reward function, PPO typically eventually works most decision- maaiing problems, however, HuGE is a method for goal-reaching problems, which is a subset of all possible decision-making problems. In these goal-reaching problems, we have shown that HuGE typically outperform PPO even convergence when the reward functions are noisy. However, PPO is a more general learning paradigm and more careful experimentation and ablations needed before making broad-reachin claims.
PPO vs HuGE Analysis
Cosine Sigilarity
0.6
0.4
PPO
Curs
Training Steps
Figure 1.14: PPO vs HuGE similarity of gradients inside a minibatch during training. We observer HuGE's similarity is much larger, which implies the gradient variance is lower. This could mean that the optimization shape smoother for HuGE and could one of the reasons why HuGE outperforms PPO.
E.3 Analysis of HuGE Human Preferences when learning from noisy feedback
We ran an additional experiment shown in Figure E.tureE.111pp to investigate further the benefits of HuGE when compared to baselines that run RL algorithms over the learned reward functions. In this case we analyze the performance of both HuGE and Human Preferences, when providing noisy synthetic feedback and observe that while HuGE succeeds in learning an optimal policy, Human Preferences does not.
28

---

HuGE vs HP when learning from noisy feedback
0.8
Sucoess Ratio
0.6
Ours
Ours (noisy)
0.4
Human Preferences
0.2
Human Preferences (noisy)
0.5M
1.5M
2M
2.5M
3M
3.5M
Number of steps
Figure E.15; We observe that when adding synthetic noisy feedback HuGE converges to optimality while Human Preference fails
29

---

F Further Analysis and Ablations
F.1 Analysis on learning from comparisons
There is a tradeoff between the frequency of labelling and the speed for the policy to converge. In Figure (left) we observe that if we query more frequently, the policy needs more labels to succeed, however, we also observe (right) that when querying less frequently it takes more timesteps to succeed. Meaning that we provide labels more frequently, the policy is going to converge faster to the optimal policy, but will come the cost of needing more human annotations. On the other hand, if the human annotators provide labels less frequently, it will take longer for the policy to converge to the optimal policy. The query frequency will hence be important parameter to look into depending on what we want to optimize for, number of human labels or timesteps to succeed. We believe that for simulation experiments, since the policy rollouts can be done very fast. However, when working with learning on the real robot, might prefer to have humans label more frequently and reduce the number of rollouts in the real world, which is usually the bottleneck.
might want to optimize for using less human labels
Ablations frequency of annotations
Ablations on frequency annotations
00
0.5
Sucgess Ratio
0.6
0.8
-15
-15
100
400k
500c
Number of labels
Number of steps
Figure F.16: On the left/right we show the number of labels/timesteps needed succeed when varying the query frequency. 1, 15, 100, and 500 are the number of episodes between each period of querying the human for annotations. We observe a clear tradeoff between needing fewer labels to succeed against needing more timesteps. Meaning that if we query more frequently, we will need fewer timesteps to succeed and vice versa. These experiments are done in the Four Rooms benchmark.
Querying few samples per batch is enough. In Figure E.17 (left) we observe that providing more labels every time we query the human leads to needing more labels to have successful policies, as expected. In right, however, we observe that the number of timesteps needed have a successful policy is very similar when querying for 5,20 or 100 annotations, however, when only querying for I the performance drops significantly. This means that 5 labels are already enough to learn how to expand the frontier, and querying more than 5 labels brings useless information.
Ablations on the number of queries per batch
Ablations on number of queries per batch
0.5
Suggess Ratto
0.6
0.5
Sucgess
-100
100
500
1000
1500
2000
2500
0
100k
200k
300
400k
500k
Number of labels
Number of steps
Figure F.17: On the right we show the number of steps needed to succeed in the four rooms benchmark depending the number of comparisons queried per batch. On the left, we show the number of labels needed to succeed, again depending on the query batch size. We observe that we can go as low 5 queries per batch, and the performance is similar to 20 and 100. Showing that many queries bring duplicated information to the goal selector training. Also, we see that providing label is not enough, degrading the performance significantly. These experiments done in the Four Rooms benchmark.
30

---

HuGE is robust to noisy labels. Increasing the noise in human labels leads to an increase in the number of timesteps needed to for the policy learn achieve the goal, as seen Figure [F19 However, this does not decrease the accuracy of the resulting policy. Increased noise in the labels makes exploration become less directed and closer the uniform frontier expansion methods.
Having closer look in Figure F.18 at the shape of the reward function when large noise is added the feedback. We observe that the goal selector becomes less accurate compared the one with perfect feedback in Fig F.23 However, HuGE still successfully reaches the goal. As we can see, there 3 modes in the hinal step (4th subfigure in F.18 This means, the goals will be sampled most frequently from these 3 modes, which will result in a less efficient frontier expansion, since only one of the three modes the target goal. However, since are learning a goal-conditioned policy through self-supervised learning this remains unaffected by this noise and will learn to go to the three modes, one of which is our target location. This would not the case for methods that use this goal selector reward function to run model-free RI, due to its convergence to local maxima without reaching the target goal.
ORACLE
RLHF
Ours
HODE2
MODE
MODES
MODEL
MODE
Figure F.18: Evolution of the learned goal selector when the distance for the synthetic human has noise of 1. We observe that the goal selector is not accurate, however, our method still successfully reaches the goal, hence, is robust to inaccurate goal selectors. This would not be the case for methods that use this goal selector as reward function to run model-free RL, due to the noise on it and multiple local minimas and maximas.
Time to succes for different injected noise on labels
250k
Timestep success
200k
150k
Noise
100k
0.05
0.1
0.3
Ours
Method
Figure F.19: Show the effect of adding Gaussian noise in the labels provided by the human on the Four Rooms benchmark. We observe that our method robust to different amounts of added noise, however, as noise increases, distance function used by the synthetic human to provide labels, which means that with higher noise the probability of the comparison being wrong will increase. For context, the distance between the initial state and the goal is around 1.6.
will the timesteps needed succeed. Noise injected into the
31

---

HuGE is robust to underlying simple reward functions. In Figure F.20 we show the performance of our method in the Four Rooms environment when dealing with simplified version of feedback. In particular, we only return feedback if the given queried states have distance difference of at least d with respect to the goal. For context, in this environment 0.5 is approximately the distance between the center of two consecutive rooms, so using d 2 0.5 is roughly similar to using the room number as reward function. Therefore, in this experiment, we can see that, even with very simple reward functions, we can still get some insight on how to solve the task, though at the expense of clearly slower convergence. performances. This also helps understand what happens in scenarios in which it is hard for humans to compare states that are similarly good for the purpose of achieving the required goal.
particular, we can see how coarser reward functions lead worse
Four Rooms
0.8
Success
0.8
0.4
0.1
0.2
>= 0.25
-d >= 0.5
50k
100k
150k
200k
250k
Number of steps
Figure F.20: Comparison on the effect of simplified reward functions providing the synthetic human annotations.
HuGE can learn when no labels are provided. This property of HuGE is because of the self- supervised learning used to train the policy but also result of using parametric goal selector as compared to directly selecting goals of interest as done [27]. which will not have this advantage. From Figure H23 observe that parametric goal selector has the capacity generalize while, by definition a non non- pon-paramttric goal selection [27] will not. Thereafter, using parametric reward model that has non-degenerate extrapolation can lead significantly more frontier expansion. In Figure F22T we show how our method succeeds in reaching the final goal room even the goal selector has stopped training when the agent enters any of the previous rooms. However, this comes cost in much slower convergence.
Analysis Learning Policy from an Incomplete Goal Selector
700k
Timestop of SucD8sS
600k
400k
300k
100k
Oradle
no stopping
room
third room Moment
second room
first room
Stopping
Figure F.21: Effect of freezing the goal selector at different points in the learning of the policy how long it takes to learn successful policy on the Four Rooms benchmark. We see that an earlier stop in the training leads an increase in the timesteps needed succeed. However, even if stop in the second room, our method is still very good at quickly finding successful policy, which shows how robust it is against incomplete goal selectors. This would not be the case for methods that RL on the learned reward functions (as DDL, and RL from Human Preferences). The policy still succeeds thanks the added random exploration, the self-supervised nature of GCSL, and small probability of sampling the final goal.
32

---

F.2 Goal selector Analysis
Learning a goal selector more feedback efficient than directly using the human feedback. In figure F22 show comparison of the number of labels needed succeed when using parametric goal selector (Ours) against directly using the goal selected by the human (DDL). We show the comparison between different frequencies of human querying. 15, 100, 500 episodes the number episodes we wait before querying the human annotator for more labels. We observe that when learning goal selector, we obtain reduction in the number of labels needed of 40% when querying every 15 or 100 episodes and a reduction of 59% when querying every 500 episodes. Furthermore, if don't learn this parametric model, with low frequencies successful policy, as happens for the non-parametric version at 100, 500 episodes of frequency. When using the non-parametric goal selector (DDL) not all trials succeed, for querying every 100 episodes, 2 seeds out of 4 fail and for 500 episodes between querying 3 out of the fail, which another reason why parametric goal selectors are better.
might not learn
Increased feedback efficiency by learning a goal selector
Labels succeed
400
300
200
Method
100
DDL-like
Ours
0
15 episodes
100 episodes Frequency
500 episodes
Figure F.22: Comparison the number of labels needed to succeed when using parametric goal
selector (Ours) against directly using the goal selected by the human (DDL).
ORACLE
Figure F23: Progress of goal selector learning in the four rooms environment learning progresses it gets closer to the target (oracle on the right). The purple area represents the visited states by the agent at that point. We observe that the goal selector provides extrapolation which will help the training with fewer annotations.
In figure F.23 we show the goal selector will have non-trivial generalization, allowing us to continue expanding the frontier even when no human is present.
Furthermore, in F24 we explore how accurate the goal selector is, depending on the number of queries it has been trained with. In particular, tested it in the Four Rooms environment by training the goal selector using pairs of states sampled uniformly. During evaluation, given two states which are less than dl units apart, we compute the accuracy for which the model is able to pick the closest state to the goal. This allows us to see that the model is able to, given two states, determine which one the closest to the goal, even when the given states are very close together and even when trained with just a handful of queries. For context, bear in mind that the distance from the initial state to the goal is 1.6 units.
Qualitative analysis of the generalization the goal selector. In this qualitative analysis, we show visualizations of the learned goal selector different rooms are discovered during the learning
33

---

Four Rooms
0.8
0.6
Accuracy
0.4
d
-
0.5
0.2
0.1
0.05
20
40
60
80
100
Number queries
Figure F.24: Accuracy of the goal selector depending on the number queries and dependent the distance d between the states compared in the labels.
process in the four-rooms domain Fig F.25
The goal selector model shows nontrivial extrapolation and can potentially provide guidance even beyond the of states is trained
ORACLE
Figure F.25: The goal selector learns and converges to a result close to the oracle (rightmost) as epochs increase (left to right). We observe how this goal selector gets updated iteratively as the frontier expands, Colder colors mean lower reward for that state, whereas warmer colors mean higher reward for that state, in this case, this is equivalent the distance the goal.
Compatibility of HuGE with Learning from Trajectory Demonstrations
As we mention in Section 4.3 HuGE is compatible with learning from trajectory demonstrations. In Figure F.26, we show how HuGE can improve the performance of simple imitation learning starting from different amounts of demonstrations. Given the number of demonstrations. imitation learning fails on less than 10 demonstrations, and with HuGE we can improve the policy to succeed in all cases.
34

---

Four Rooms
1.4
Distance goal
0.8
0.6
2
0.2
BC 5
BC 10
20k
40k
60k
80k
100k
Number of steps
Figure F:26: The figure depicts the distance the goal in the Four Rooms environment when using policy pre-trained via Behaviour Cloning with 0, 2, 5, and 10 demonstrations, respectively. We see that using BC on a small number of demonstrations helps to boost the performance of our method. Also, notice that BC wouldn't 't achieve success (distance 0.05) in any of the cases due compounding errors which leads to covariant shift. However, HugRL solves these compounding errors within small number of steps.
35

---

G Implementation Details
G.1 Algorithm: stopping before exploration
One important detail of the algorithm is knowing when to stop rolling out the policy and start exploration. The idea is that since the policy learned in self-supervised manner. we do not want to have too many redundant steps in a single state. If this happens then because of an imbalance of redundant actions going over this single state, the training set, will bias the policy get stuck and not transition to other areas. For it not to happen, we need to stop rolling out the policy at the right moment and launch the next step the algorithm, the exploration. We propose three different cases where we should stop the rollout, first, once the breadcrumb state Js is reached, second, once the time-horizon for the episode reached, third, once the policy not making any progress and gets stuck in some place. Detecting this can be trivially done looking at the euclidean distance between last states when the state space is in point space, however, it can also be done in image space and we explain further in Appendix A.I A.1
G.2 Hardware
For training the models and running the experiments, had access several workstations with one GeForce RTX 2080 Ti or one GeForce RTX 3090. It took on average 8 hours on these machines to run seeds for each one of the baselines and our method. We account the total amount of compute hours would be around 1 1440 hours for the whole project, taking into account, experimentation and testing the algorithms.
G.3 Networks with Fourier Features
Seeing the complexity of our benchmarks. where we can have non-smooth reward landscapes for the goal selector. For example, the four rooms environment, between one side and the other of the right rooms, the reward changes significantly and abruptly. Adding Fourier Features has been shown to work well for fitting these landscapes (56). For this reason, we used them in some of our experiments, as detailed in Section G More precisely, when used, we added an additional layer with Fourier features of size times the input dimension.
G.4 Training details
The details of the parameters with which the results have been obtained will be disclosed in this section. In particular, Table [.4]depicct the parameters used for the different benchmarks, while Table contains the hyperparameter configuration used for the different algorithms.
36

---

Parameter
Value
Shared (1o those that apply)
Optimize
Adam
Discount factor (?)
0.99
Reward model architecture
MLP(256, 256)
Use Fourier in the reward model
True
Buffer size reward model
1000
Steps per reward model update
1000
GCSL, Oracle and Ours
Learning rate
10
Batch size
100
Policy architecture
MIPP (400, 600, 20,,6000,000 300)
Steps per policy update
5000
Use Fourier in the policy model
True
Buffer size rollout
1000
Max gradient norm
Last trajectories to be labeled
1000
Human preferences Same parameters
52
plus/except
Learning rate
5 10
Batch size
100
Policy architecture
MLP (256, 64)
Steps per policy update
5000
Use Fourier in the policy model
False
Buffer size rollout
1000
Max gradient norm
Last trajectories to be labeled
1000
DDL
Learning rate
5.10 10-1
Batch size
256
Buffer Size
2 10
Policy architecture
MLP (256, 256)
Steps per update
1000
PPO Same parameters 52 plus
Buffer size
8192
Policy architecture
MLP (400, 600, 300)
Table G.3: Hyperparameters setting for the algorithms
Environment
Four rooms
Maze
Pushing around Obstacles
Kitchen
Block Stacking
Bandu
Steps per trajectory
50
250
100
100
10
12
Label from last steps
10
50
10
20
10
12
Table G.4: Benchmark-related parameters
37


Reviews:
[{"review_id": "Uoct17ssV5", "reviewer": "Reviewer_MJKZ", "summary": "Broadly, the paper addresses the challenge of learning multi-stage robotic navigation and manipulation tasks in simulation. \n\nThe paper frames several benchmark tasks as goal-conditioned RL (GCRL), and they present a novel method (\"HUGE\") for leveraging human preferences collected during learning. In particular, the preferences are used to train a goal-selector model that guides (biases) exploration. For policy-learning, they use goal-conditioned supervised learning on the collected replay buffer, without requiring a handcrafted task-specific reward.\n\nCompared to [1], HUGE also uses hindsight relabeling to learn from a replay buffer despite a sparse reward, but it differs in that it biases sampling from the replay buffer based on human preferences. Compared to [11], HUGE also collects human preferences iteratively (intertwined with learning). However, HUGE doesn't use its collected human feedback to learn a reward or otherwise directly bias the policy. Instead, the feedback is only used to bias exploration. Compared to [17], HUGE also has an exploration phase that builds an archive of trajectories. However, HUGE chooses promising goal states and generates trajectories using its goal-condition policy, while [17] chooses promising start states based on novelty and can generate trajectories with various exploration policies (random, epsilon-greedy, novelty-based).\n\nThe authors evaluate on four simulated manipulation tasks, two simulated navigation tasks, and two real-robot tasks. They compare to several baselines including the related work mentioned above as well as PPO (with dense and sparse reward variants).", "questions": "Can you improve your comparison to baselines based on the feedback I gave above? In particular, I am most interested in a robust, fair comparison to [11] (\"Human Preferences\"), because I consider that work the most directly comparable to yours: it doesn't require a handcrafted reward signal but it does require collection of human preferences intertwined with learning.\n\nI hope you'll address the weaknesses in your PPO baseline I mentioned earlier: train to convergence with a fine-tuned reward function. Figure 5 currently shows that HUGE (with synthetic preferences derived from a reward function) outperforms PPO using the same reward function. If this result holds after you improve the PPO baseline, can you analyze this? Would it mean the synthetic variant of HUGE is fundamentally a better optimization algorithm compared to PPO?\n\nI notice your two real-world tasks are fully-observable. How would you apply your approach to a real-world task that is only partially-observable, such as room-scale manipulation with visual perception?", "limitations": "No concerns here", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "Their method doesn't require a handcrafted reward and it's able to leverage noisy, infrequent human feedback. These strengths are relevant and valuable to this domain.\n\nEvaluation on a variety of simulated and real tasks.\n\nImpressive crowdsourcing infrastructure and diversity of annotators.\n\nInteresting, well-explained adaption of real-world tasks to fit their use of goal-conditioned RL.", "weaknesses": "Lack of clear, fair numeric results on task performance across HUGE and the baselines:\n\n= In Figure 5, I can only compare task performance after a given number of steps. A table with a performance number after training each approach to convergence would be more clear and would better support the claim from your text \"HUGE beats prior work on the proposed long horizon tasks in simulation\".\n\n= It's not clear why some tasks were trained for millions of steps and others not.\n\n= The paper text conveys that Figure 5's various approaches are using a poorly-tuned reward function (\"we did not do any reward finetuning\"). In particular, this makes the PPO results questionable.\n\n= This section also uses synthetic human preferences derived from your reward function, instead of real human preferences. This confuses me and contradicts the earlier premise of HUGE. HUGE with this synthetic source of preferences would appear to be another RL method that utilizes a handcrafted reward function and not human feedback. This muddies your comparison with other approaches, especially [11] which is intended to be guided by real (not synthetic) human feedback.\n\n= It seems like other baselines might benefit from the addition of behavior-cloning pretraining, as you did for \"BC + Ours\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and for taking the time to review our work. Please find answers/additional experiments to address your concerns below\n\n> “Poorly-tuned reward function”\n\nA key point of our paper is the robustness to noisy human feedback, which means that even with a simple guiding human proxy, HuGE can still solve the proposed tasks, as we showed in Appendix E.15. It is known that this is not the case with PPO which needs careful reward shaping to succeed AM[3,4], and also reflected in our results in Fig 5 in the original paper. \n\nWe provide a new analysis experiment on this result (Fig AM 5), where we test HuGE, PPO, and Learning from Human Preferences (using PPO) on two reward functions, one that we very carefully shaped during the rebuttal period, and another that is as in the original submission (note that the reward function serves to inform the synthetic human proxy for HuGE). We observe: \n      \n- HuGE succeeds with both reward functions \n\n- PPO only succeeds when the reward function is carefully shaped.\n\n- Human Preferences only partially succeeds when the reward function is carefully shaped \n\n- When PPO and Human Preferences succeed, they are significantly less sample efficient than HUGE. \n\nThis demonstrates that HuGE is robust to simple underlying reward functions, which makes it robust to learning these reward functions from Human Feedback. This is also reinforced by additional experiments with real human feedback on all domains as shown in Fig AM 2-3. \n\n>“Going from Synthetic Human Feedback to Real Human Feedback” & “Fair comparison with Human Preferences”\n\nIn Figure 5, all baselines that rely on human feedback were trained using a proxy-human for a fair comparison among methods. However, as the reviewer requested in Figure AM 2-3 we provide additional results of HuGE working from real human feedback and results on Human Preferences from real human feedback on the four rooms and empty rooms environments. With this robust, fair comparison between HuGE and Human Preferences, we observe that when learning from real humans:\n\n- HuGE is more sample efficient in the number of human labels compared to learning from   human preferences [11] with real humans\n\n- HuGE is more sample efficient in the number of timesteps to succeed compared to [11]. \n\n>“Analysis of PPO vs HuGE (from synthetic feedback)”\n\nAs noted, we observe that HuGE is more sample efficient than PPO. To analyze why this is, we ran a comparison on gradient variance using HUGE as compared to gradient variance with PPO, comparing the average pairwise cosine similarity across mini-batches to measure gradient variance (as suggested in AM[1]). We find that training the gradient similarity is significantly larger in HuGE than in PPO, potentially explaining improved sample complexity. Besides this, other components in HuGE may also lead to faster learning:\n      \n1. Generalization across states that come from self-supervised hindsight relabeling,\n2. Hindsight relabeled policy learning is off-policy while PPO is on-policy. \n\n> “Does it mean the synthetic variant of HuGE is fundamentally a better optimization algorithm compared to PPO?”\n\nNot necessarily, given a carefully shaped reward function, PPO typically eventually works on most decision-making problems, however, HuGE is a method for goal-reaching problems, which is a subset of all possible decision-making problems. In these goal-reaching problems, we have shown that HuGE can typically outperform PPO even at convergence when the reward functions are noisy. However, PPO is a more general learning paradigm and more careful experimentation and ablations are needed before making broad-reaching claims. \n\n>“Other Baselines would benefit from BC pretraining”\n\nTo provide a fair comparison, we also added learning from human preferences with the same amount of BC pretraining (implemented using stable baselines AM[2]). We found that despite this pretraining, learning from human preferences [11] and PPO were outperformed by HUGE. \n\n>“Table of performance at convergence:”\n\n We would like to refer the reviewer to Table D.10 with the results after convergence in the Appendix. \n\n>“Number of training steps per task”\n\n Each task has different action and observation spaces as well as different ranges of timesteps per episode, we refer the reviewer to Appendix C for more details. Also, we ran each baseline until convergence. \n\n>“How would you apply your approach to a real-world task that is only partially observable”\n\nAs of now, the framework is largely structured around fully observable problems, as is a significant body of work on reinforcement learning in MDPs [11,21,24,37]. Training recurrent policies, rather than MLPs, as well as recurrent goal selectors, may make the system applicable to partially observable settings. \n\nAM[1] Ilyas, Andrew, et al. \"A closer look at deep policy gradients.\" arXiv preprint arXiv:1811.02553 (2018)\n\nAM[2] Raffin, Antonin, et al. \"Stable-Baselines3: Reliable Reinforcement Learning Implementation\" http://jmlr.org/papers/v22/20-1364.html (2021)\n\nAM[3] Learning robust perceptive locomotion for quadrupedal robots in the wild, Takahiro Mki, et al, Science Robotics, 2022.\n\nAM[4] Visual Dexterity: In-hand Dexterous Manipulation from Depth, Tao Chen, et al. 2022, arxiv.org/abs/2211.11744\n"}, {"review_id": "SLTzxgRKiC", "reviewer": "Reviewer_EF3C", "summary": "This paper introduces HUGE, a Reinforcement Learning algorithm that makes use of human preferences to guide the selection of partial goals.\n\nHUGE expands upon Goal-Conditional Supervised Learning (GCSL) by improving the goal selection method using noisy labels from humans to form a model, $f_\\theta$, of distances to a goal state. This model $f_\\theta$ is then used set goals for the GCSL algorithm to learn in a self-supervised fashion, thus bootstrapping learning.\n\nThe paper features an extensive evaluation of HUGE, with 6 synthetic tasks, 2 real robot tasks and one task trained from crowd-sourced feedback. Results show that HUGE outperforms a large number of baselines.", "questions": "1. How were the annotations for the real-world tasks obtained?\n2. Why does BC+Ours finish early in Figure 5 Kitchen & Bandu?\n\n**Minor nitpicks and suggestions:**\n* $\\mathcal{G}$ is undefined in equation 4.\n* In algorithm 1, step 5, the second argument to PolicyExploration should be $f_\\theta$.\n* In figure 6 left, what is the difference between \"human + 5 demos\" and \"crowd-source + 5 demos\". Moreover, the colours of their curves are indistinguishable to my eyes.\n* When defining $f_\\theta$, I would clarify that \"closer\" is not necessarily meant as a mathematical distance, but rather as a human intuition of closeness, which I belive is what $f_\\theta$ is supposed to capture.\n\n\n**Post-rebuttal update**\n\nSee discussion for answers to the above questions.", "limitations": "The limitations have been adequately addressed.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* HUGE combines intuitions from GCSL and GoExplore, and can take advantage of asynchronous human preferences.\n* The paper is really well written and is easy to follow.\n* The paper presents a thorough evaluation. I find it particularly that HUGE Was able to train a real robot policy in under 30 hours with just 130 annotations.\n* HUGE can take advantage, but does not require, expert demonstrations through Behaviour Cloning.", "weaknesses": "* It would have been more representative to choose PEBBLE [1] rather than Christiano et al. (2017) as a more representative example of the performance of Preference-based Reinforcement Learning.\n* Some of the results are buried in the appendix. For instance section 5.2 does not mention what was the task analysed are. From Figure 6. left, I assume it's the Kitchen task from D4RL. Still, neither the final performance nor the task are mentioned in the main text. \n    * Similarly, the main results of the analysis comparing the final performance and the number of annotations should be included in the main text, since it provides readers with an estimation of how many labels they may need for their task.\n\n**Post-rebuttal update**\n\nAuthors added a comparison against PEBBLE, where HUGE outperforms it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your feedback and for taking the time to review our work. Please find the answers to your comments and concerns below.\n\n> “More representative to choose PEBBLE”\n\nWe implemented PEBBLE (using author provided GitHub repository) and provide the new curves with this implementation in Fig AM1. We find that while PEBBLE can learn in certain environments (block stacking, bandu, maze), however, the performance is worse than Human Preferences and HuGE. \n\nSince the reviewer seemed particularly keen on comparing to PEBBLE we tried to understand better why it was failing and we find the following:\n\n- We observe that when the initial unsupervised exploration step does not cover a big enough set of the state space, PEBBLE gets stuck and does not manage to reach the goal. \n\n- We spent a decent amount of time trying to make it work better (hyperparameter tuning on entropy, number of unsupervised steps, increasing the stochasticity in the rollout actions) however nothing seemed to make it work on our benchmarks. We used the implementation provided by the authors and we also implemented our own version, in both cases this faced the same issues of getting stuck mid-way (meaning exploration collapsed and no progress was made).\n\n> “Some of the results are buried in the appendix”\n\nAs suggested, we will move the final performance of the baselines, and the number of annotations for the real human experiments into the main text of the paper and provide more clarity on the problem setting. \n\n> “For instance, section 5.2 does not mention what the task analyzed are.”\n\nThank you for pointing this out! It was only mentioned in small in Figure 6’s title and we will clarify this in the camera-ready version. In the meantime for more information, the crowdsourced data collection was performed on the kitchen manipulation task, where the franka arm needs to open 3 objects (cabinet, microwave, slider), more details about the kitchen environment on Appendix C.\n\n> “The main results of the analysis comparing the final performance and the number of annotations should be included in the main text”\n\nIndeed. We will include this in the main text in Section 5.1. In the meantime, we refer the reviewers to Appendix D and Table D.10 for more details. \n\n> “How were the annotations for the real-world tasks obtained?”\n\nWe obtained the annotations for the real-world task in the same manner as in the simulated experiments, querying three annotators, no crowdsourcing was done in these experiments. We will clarify this in the text, and for more information on how the feedback was collected on the simulated experiments we refer the reviewers to Appendix B.\n\n> “Why does BC+Ours finish early in Figure 5 Kitchen & Bandu?”\n\nWe will update these two curves for the camera-ready, they were not fully completed due to computing limitations.  \n\n>“In figure 6 left, what is the difference between \"human + 5 demos\" and \"crowd-source + 5 demos\". ”\n\nThe difference is in the number of humans that collected the labels, for “human + 5 demos” only one annotator provided the labels, for “crowd-source + 5 demos” 109 annotators provided the feedback, we provide more details about this crowdsourcing experiment in Appendix B.1. \n\n> Minor corrections\n \nWe thank you for this feedback, and we will incorporate this into the manuscript. \n\n"}, {"review_id": "2jWvrXrYhw", "reviewer": "Reviewer_erJy", "summary": "This paper focuses on the exploration problem in decision-making tasks. Previous works try to leverage human guidance with constant synchronous high-quality human feedback, which is expensive and impractical to obtain. In this paper, the authors propose Human Guided Exploration (HUGE), which is able to leverage low-quality feedback from non-expert users. Specifically, the key idea is to separate the challenges of directed exploration and policy learning. Human feedback is only used to direct exploration, while self-supervised policy learning is used to independently learn unbiased behaviors from the collected data. The task for human annotators is just to select which of two states is closer to a particular goal. Then this ranking will be used to train a distance function for goal selection. The experimental results on robotic navigation and manipulation tasks in both simulation and real-world robots demonstrate the advantages of the proposed method.", "questions": "No particular questions.", "limitations": "No limitation is discussed in the paper. One potential limitation is that the human annotation of the preference of states may be difficult for tasks that require logic reasoning. The experiment environments used in this paper mainly focus on L2 distance, which is easy for humans to quickly select the state close to the goal.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of the proposed method is simple yet effective. Separating the exploration and policy learning dramatically reduces the annotation effort and influence of noisy annotation on policy learning. Using human preference to learn the distance function is also better than directly learning the reward function.\n2.\tThorough evaluation of multiple environments, including both simulation and real-world tasks in robotic navigation and manipulation domains. The experimental results show that HUGE outperforms other methods by a large margin. \n", "weaknesses": "1.\tFigure 1 provides limited information. It is hard to find that human feedback is noisy and asynchronous. The tasks of pick-and-place and drawing do not contain detailed explanations. This figure can be improved by adding a comparison between previous works that require high-quality data and their method.\n2.\tThe method may need to be evaluated on more complex real-world tasks. The authors argue that Novelty-based exploration performs task-agnostic exploration of the entire state-space, thereby over-exploring the environment. But it seems that the environments used in this paper do not contain very large exploration space. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and for taking the time to review our work. Please find answers/additional experiments to address your concerns below\n\n> Figure 1 provides limited information. \n\nWe will improve this as suggested to include comparisons with prior methods. Thank you for your suggestions!\n\n> The authors argue that Novelty-based exploration performs task-agnostic exploration of the entire state-space, thereby over-exploring the environment. But it seems that the environments used in this paper do not contain very large exploration space.\n\nTo verify this, we scaled up the experiment of Go-Explore + Novelty to test task-agnostic exploration via undirected frontier expansion (in Fig AM1). We find that while this can solve certain tasks (maze, four rooms), it takes significantly longer and is unable to solve the more challenging tasks (kitchen, block-stacking, pusher). This shows the presented environments have a large exploration space and also shows why directing frontier expansion is important. Finally, prior work on exploration methods such as LEXA also test their algorithms on some of the same benchmarks, such as the kitchen and block stacking, but they solve shorter horizon problems :\n\n1. In the kitchen, LEXA only manipulates 2 objects -> we manipulate 3 sequentially,\n2. In block manipulation LEXA only moves 2 blocks -> we assemble a castle-like structure with 4 blocks\n\n> The method may need to be evaluated on more complex real-world tasks\n\nWhile this was challenging to perform in the short rebuttal period, we will evaluate this further and we are excited to continue as future work the evaluation on more real-world experiments such as object rearrangement in a household setting, drawing a variety of shapes, etc. \n\n> “One potential limitation is that the human annotation of the preference of states may be difficult for tasks that require logical reasoning. The experiment environments used in this paper mainly focus on L2 distance, which is easy for humans to quickly select the state close to the goal.”\n\nWhile it is true that many of the environments use distance as an oracle, this is not fundamental to the method, we would like to point out the drawing experiment that we performed in the real world. It is very hard to specify a distance notion for this experiment, however, it remains very easy for the human to quickly select which state is closer to the goal. We will note this potential limitation in the manuscript. \n"}, {"review_id": "a0RHGFrkFf", "reviewer": "Reviewer_4hMa", "summary": "The paper introduces Human Guided Exploration (HUGE), a system designed to integrate human feedback within the Goal-Conditioned Reinforcement Learning (GCRL) environment, offering a cost-effective and straightforward method for learning diverse robotic tasks through actual human interaction. The authors' primary objective is to strike a balance between overexploration and underexploration in GCRL tasks by incorporating human-in-the-loop assistance. Consequently, human feedback is utilized to ascertain the states requiring further exploration, building on the traditional GCRL algorithms and the GO Explore exploration framework. In particular, a target distance estimation function is trained using binary feedback, facilitating the generation of a more precise subgoal. Subsequently, the Goal-Conditioned Supervised Learning (GCSL) training paradigm is implemented to establish an interaction strategy with the environment. Experimental results in Mujoco and Pybullet demonstrate that HUGE outperforms the preceding GCRL algorithm and Human-in-the-loop algorithm.", "questions": "Other comments:\n- The spelling sparce to sparse in Figure 5.", "limitations": "-  The work lacks novelty.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The integration of human feedback into the GCRL setting is well-conceived. Hence, the motivation is clear, and the proposed method is, in general, logical and sensible.\n- This technique facilitates a simple interface between the human labeler and the algorithm, in which the human supervisor provides binary evaluations to establish which state-goal pairings are closer in comparison to others.", "weaknesses": "- The paper presents a degree of innovation that is somewhat restrained, as it essentially combines modified versions of existing methodologies (GCRL, Go Explore, and Human Preference). The foundational GCRL algorithm used is the previously established GCSL, while the exploration component is derived from the Go-Explore paradigm. It is important to note that GCSL paired with Go-Explore could have operated independently without the need for human feedback. \n- As per the previous discussion, it's not unexpected to see improved results when human feedback is integrated into the Go-Explore process, as it introduces more a priori knowledge. The implementation of binary feedback can help to decrease the exploration space by learning to rank and establishing a subgoal model.\n- This strategy essentially transforms the behavior cloning challenge into a binary human feedback problem. While this appears logical, it makes the motivation somewhat ambiguous and makes specific presumptions about the task at hand. Consequently, the work lacks a degree of originality and novelty. The three fundamental components of the proposed method: the \"goal-conditioned policy\", \"hindsight relabeling\", and \"learning reward models\" from human preference, have all been previously suggested in earlier papers.\n-  I think the tasks for evaluation are borderline novel and perhaps not hard enough when we consider a setting where human feedback can be utilized (these tasks are still challenging for RL-based methods learning with dense/sparse rewards). The maze tasks, robotic manipulation tasks are commonly seen in literature (in fact I am surprised PPO is completely failing on some of these tasks, I have personally trained PickNPlace tasks with a RL-based method with a properly shaped reward - Otherwise the authors can also explore other SOTs such as TD3 or SAC) . These tasks are not seen as a fundamental challenge for present learning methods. The main result of the paper is comparison of the proposed method against RL algorithms like PPO and LEXA in terms of sample efficiency. I prefer to see longer horizon tasks where an RL agent with a dense reward cannot easily solve.  I also did not see comparison results for the Sim2Real tasks nor for PickNPlace.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback and for taking the time to review our work. Please find answers/additional experiments to address your concerns below\n\n> “GCSL paired with Go-Explore could have operated independently without human feedback”\n\nGo-Explore + GCSL suffers from the issues of undirected frontier expansion. To illustrate this, we run this baseline (Go-Explore + GCSL) across all domains in Fig AM1. We observe:\n\n- The baseline (purple line, GoExplore+GCSL) does not always succeed and takes more samples than  HuGE. \n- Longer horizon goals are not discovered. In multistep tasks such as the kitchen, Go-Explore + GCSL only manages to discover how to do 1 task (interacting with one of the objects) at a time, it never manages to combine 2 or 3 tasks as HuGE does.\nIn contrast, we can see the benefit of HUGE in guiding exploration and reducing search space.\n\n> “It is not unexpected to see improved results when human feedback is integrated into the Go-Explore process”\n\nIt is *very important* how we integrate human feedback into the learning process. As we explain in Section E.1 [paragraph: Huge is robust to noisy labels] and show in Fig AM 7 prior work that introduces human feedback into the learning pipeline fails when this human feedback is noisy. We show that using this learned reward function to sample goals and guide exploration makes this method more robust to human feedback.\n\n> “The three fundamental components of the proposed method: the \"goal-conditioned policy\", \"hindsight relabeling\", and \"learning reward models\" from human preference, have all been previously suggested in earlier papers”\n\nWhile the terms - goal-conditioned policies, hindsight relabeling and learned reward models have been seen in prior papers, we would like to emphasize that:\n\n1.  We propose a unique way of integrating learned reward models into the exploration of learning goal-conditioned policies.\n2. *Separating* policy learning from human feedback, and using it for soft frontier expansion (Ours) is crucial to both enable the solving of harder tasks and learning from noisy human feedback. Our results on Go Explore + GCSL [21,24,37], Human Preferences [11], and GCSL [24] emphasize this. \n3. We enabled capabilities that were not possible in prior work solving long horizon tasks *without* careful reward engineering, using occasional and noisy (crowdsourced) human feedback\n\n> “I think the tasks for evaluation are borderline novel and perhaps not hard enough when we consider a setting where human feedback can be utilized”\n\nWe propose and solve harder tasks than any of the prior related work:\n1. PEBBLE benchmarks: opening single drawers/doors -> HuGE benchmark (kitchen): three sequential tasks in the kitchen involving open drawers, and sliders \n2. LEXA benchmarks: \n      - two sequential tasks in the kitchen-> HuGE benchmark (kitchen): three sequential tasks in the kitchen involving open drawers, and sliders \n      - Pick and place tasks with two blocks ->Huge benchmark (bandu, block-stacking): assembling a castle-like structure with 4 blocks\n\nHuGE beats both PEBBLE and LEXA (Go-Explore+GCSL) on these harder benchmarks. Moreover, we see that algorithms like PPO/SAC do not learn or learn significantly slower.\n\n> “Surprise that PPO is completely failing on some of these tasks”\n\nThe goal in Figure 5 was to provide a fair comparison between all of the benchmarks using the same underlying reward. Since HuGE is robust to noisy underlying reward functions, this works with unshaped reward functions. For further analysis, we performed a new experiment with a more carefully tuned underlying reward function, Fig AM5. We see that PPO and HP fail with totally reasonable reward functions. After engineering effort finetuning these reward functions, these methods succeed, but still slower than HuGE. On the other side, we show HuGE’s robustness to the underlying reward function since it works in both cases.\n\n> “These tasks are not seen as a fundamental challenge for present learning methods” “I prefer to see longer horizon tasks where an RL agent with a dense reward cannot easily solve”\n\nWe refer the reviewer to the drawing task in the real world where the reward function for the robot to draw something is very hard to specify. In this case, we do not have access to an oracle dense reward function and yet HuGE is able to succeed from easy-to-provide human feedback. \n\nIn addition to this, we have also run state-of-the-art baselines - PPO/SAC and several baselines for learning from human preferences on the environments in simulation. We find that HUGE is both more efficient and performant in these baselines across tasks. This suggests that without significant reward engineering, these tasks *are* challenging for current methods. \n\n> “I also did not see comparison results for the Sim2Real tasks nor for PickNPlace.”\n\nBecause of HuGE’s sample efficiency, we can learn the policies directly in the real world, hence, no modeling nor sim2real is required. While this could certainly be performed, the point of the real-world experiments is to show that HUGE is both feedback and sample efficient for real-world learning.\n\n> “This strategy essentially transforms the behavior cloning challenge into a binary human feedback problem.” \n\nThe problem we consider is learning from human comparative feedback, not converting behavior cloning into a feedback problem. We propose an algorithm for learning from comparative feedback, and then we show that this algorithm can benefit from some pretraining using behavior cloning. We would appreciate a clarification if we have misunderstood. \n\n> SAC comparison\n\nWe also conduct an experiment to compare with other off-policy RL methods like SAC. We see that HuGE and the previous baselines outperform SAC from dense rewards in these benchmarks, Fig AM 1. A small caveat is that these results could be improved somewhat with further tuning of SAC, but will still require reward engineering. \n"}]

Author rebuttal:
Dear reviewers,

Thank you for your constructive feedback. In response to reviewer concerns, we have conducted a number of new experiments and analyses. We describe these briefly below and refer reviewers to individual responses for detailed clarifications: 
 
**[DISCLAIMER 1]**: In the following experiments we talk about reward functions and RL methods that use reward functions. However, HuGE does *NOT* need any reward function, it learns uniquely from human feedback. Where specified, we use human proxies, which have an underlying reward function, to provide a fair comparison among different baselines. 

**[DISCLAIMER 2]**: Figures in the Aditional Material (PDF) attached to the rebuttal are referred in the text as Figure AM ${figure number}.
 
1. **More carefully tuned rewards for PPO**: (reviewer MJKZ): We compare the performance of HuGE, PPO, and Human Preferences from finetuned/not finetuned reward functions in order to clarify how important careful reward engineering is to the success of these methods. Findings:
    - HUGE succeeds even without careful reward engineering, 
    - PPO and Human preferences struggle without careful reward engineering. 
    - With careful reward engineering - PPO and Human preferences can succeed, but they learn significantly slower than HUGE. (Fig AM 5)

2. **HUGE (Ours) with real human feedback (reviewer MJKZ)**: We provide additional results on running HuGE from real human feedback across more benchmarks (see Fig AM 2-3). HuGE matches performance when feedback is provided by a synthetic human. 

3. **Human preferences [11] with real human feedback (reviewer MJKZ)**: To induce a fair comparison, we also run learning from human preferences[11] with real human feedback, as shown in Fig AM2. Human preferences:
     - Needs more human labels to converge 
     - More timesteps to converge 
     - converges to lower performance compared to HuGE

4. **HuGE is more robust to noisy feedback compared to Human Preferences** (reviewer 4hMa)
     - HuGE matches the behavior obtained with perfect annotations whilst Human Preferences does not solve the task at hand, see Fig AM 7

5. **BC pre-training with human preferences** (reviewer MJKZ): To induce a fair comparison when pretraining data is available, we introduced BC pretraining on a couple of benchmarks for PPO and Human Preferences (see Fig AM 1) and compared this to HUGE with pretraining. We found that even if each method improves when adding demo pretraining, these still take longer than HuGE with and without demos to converge.

6. **Analysis of PPO vs HuGE** (reviewer MJKZ): Since HUGE is much more sample efficient than PPO, we conducted analysis to understand why this might be the case for goal reaching problems. We hypothesize this may be (atleast in part) due to lower variance gradients, and show this in Fig AM 6. 

7. **PEBBLE comparison** (reviewer EF3C): We conducted an experiment to compare with an off-policy RL method learning from human preferences such as PEBBLE [31]. We find:
      - HuGE and Human Preferences methods outperform PEBBLE

8. **Go-Explore + GCSL** (reviewer 4hMa): We conducted an experiment to understand whether GCSL + go-explore by itself is effective enough to solve the proposed tasks. We find in Fig AM1 that Go-Explore + GCSL:
      - Longer horizon goals are not discovered (such as in the kitchen benchmark), which is also seen in prior work (LEXA)[34]. 
      - When it solves the task it takes much longer than HuGE to discover target goals since exploration is undirected.
      - SAC comparison (reviewer 4hMa): We conducted an experiment to compare with other off-policy RL methods like SAC. We see that HuGE and the previous baselines outperform SAC from dense rewards in these benchmarks, Fig AM 1. 

Please let us know if other experiments/clarifications can help with the discussion!


Meta-review: The submission here describes an interesting, novel, and practical method for leveraging human feedback to help with exploration in reinforcement learning. The presentation of the work is excellent and the empirical evaluation and results are impressive. The work is of sufficient quality for presentation at NeurIPS. The authors should now spend the time to succinctly update the submission to include the extra results and discussion points that proved so effective during the author-reviewer discussion period.
Decision: Accept (poster)