SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Marcel Torne<sup>1,2</sup> Max Balsells<sup>3</sup> Zihan Wang<sup>3</sup> Samedh Desai<sup>3</sup>
Tao Chen $^{1}$  Pulkit Agrawal $^{1}$  Abhishek Gupta $^{3}$
<sup>1</sup>Massachusetts Institute of Technology <sup>2</sup>Harvard University <sup>3</sup>University of Washington
{marcelto,taochen,pulkitag}@mit.edu
{balsells,avinwang,samedh,abhgupta}@cs.washington.edu
Exploration and reward specification are fundamental and intertwined challenges for reinforcement learning. Solving sequential decision-making tasks requiring expansive exploration requires either careful design of reward functions or the use of novelty-seeking exploration bonuses. Human supervisors can provide effective guidance in the loop to direct the exploration process, but prior methods to leverage this guidance require constant synchronous high-quality human feedback, which is expensive and impractical to obtain. In this work, we present a technique called Human Guided Exploration (HuGE), which uses low-quality feedback from non-expert users that may be sporadic, asynchronous, and noisy. HuGE guides exploration for reinforcement learning not only in simulation but also in the real world, all without meticulous reward specification. The key concept involves bifurcating human feedback and policy learning: human feedback steers exploration, while self-supervised learning from the exploration data yields unbiased policies. This procedure can leverage noisy, asynchronous human feedback to learn policies with no hand-crafted reward design or exploration bonuses. HuGE is able to learn a variety of challenging multi-stage robotic navigation and manipulation tasks in simulation using crowdsourced feedback from non-expert users. Moreover, this paradigm can be scaled to learning directly on real-world robots, using occasional, asynchronous feedback from human supervisors. Project website at https://human-guided-exploration.github.io/HuGE/.
Figure 1: HuGE leverages noisy and asynchronous feedback from multiple non-expert humans to train robot control policies directly on the real world.
How should we teach agents a new task? A general method is to provide agents with a reward function and employ reinforcement learning. This may appear to be a straightforward plug-and-play
procedure, yet in reality, it usually means iterative human intervention in designing reward functions to overcome exploration obstacles and circumvent potential reward hacking. The cyclical process of a human meticulously crafting a reward function, assessing the resulting behavior of the learned policy, and then refining the rewards to enhance performance can be a painstaking and inefficient process.
If an agent is able to effectively and autonomously explore its environment, it can learn new tasks using an easily designed sparse reward function [50, 20]. This would make the training process less reliant on humans designing detailed reward functions for "dense" supervision. Several strategies incentivize state coverage in exploration by expanding the frontier of visited states and explicitly rewarding discovering novel states.[46, 5, 21, 43]. When purely optimizing for state coverage, the agent overexplores parts of the state space that are not relevant to the task at hand [13]. This quickly becomes a futile effort when exploring in environments with large state spaces, suggesting the need for a more directed exploration strategy.
A promising avenue for directed exploration, without requiring a prohibitive amount of reward design, is in developing techniques that leverage non-expert human feedback to adjust the agent's training [55, 30, 11, 36]. This work is a step towards the overarching objective of learning new skills in the real-world through meaningful human-robot interactions. A challenge with techniques that optimize policies over learned reward functions is that humans may provide noisy feedback, or may not even know the optimal solution for the task at hand [18], rendering their feedback unreliable and biased, a problem which is exacerbated with a large number of non-expert supervisors. When this noisy and incorrect feedback is used to learn reward functions for RL, it can lead to learning biased and incorrect policies. Requiring human supervisors to provide high-quality feedback synchronously and frequently during the training process becomes prohibitively expensive. If we want to reduce the burden on supervisors it is important to ensure that inaccuracies, biases in the reward function are not reflected directly in the learned policy.
We note that reward functions in reinforcement learning serve two distinct purposes. One, determining which goal or behavior is optimal, and two, determining how to explore the environment to achieve this goal. In this work, we make the observation that these two independent purposes of reward functions can be decoupled. Human feedback can be used to guide exploration, while policy learning techniques like self-supervised reinforcement learning can learn unbiased optimal behaviors for reaching various goals from the data collected via exploration. This provides the best of both worlds: a way to guide exploration using non-expert human supervisors, while avoiding inheriting the bias from providing asynchronous, low-quality feedback.
Human Guided Exploration (HuGE) uses noisy, infrequent, and asynchronous binary human comparative feedback to guide exploration towards subsets of the state space closer to goal, while learning optimal policies via self-supervision. Rather than having humans directly label which states to visit, HuGE solicits users for binary comparisons between visited states to determine which are closer to the goal. These comparisons help train a goal selector which biases exploration towards chosen states that are more proximal to the objective. The process of exploration involves returning to these selected states and subsequently expanding the set of visited states from there. From this exploration data, we learn optimal policies for goal-reaching in a self-supervised manner using ideas from "hindsight" relabelling [24, 2, 28], hence avoiding reliance on any explicit reward function.
Iterating between exploration guided by the goal selector and self-supervised policy learning, HuGE expands the frontier of visited states until the desired target goals can be reached reliably. Opposed to directly using human feedback as a reward [16, 45], HuGE lets humans provide noisy, asynchronous, and infrequent feedback, as it is simply used to guide exploration, not directly as an objective for policy learning. In the worst case, when human feedback is incorrect or absent, the exploration is hampered but the policy learning scheme is still convergent to the optimal solution. This way, humans are able to essentially "drop bread crumbs", incrementally guiding agents to reach distant goals, as shown in Fig 3. This helps us solve more complex tasks than if solely relying on self-supervision and is able to overcome the optimization bias that is often encountered when optimizing over a learned and noisy reward function [16]. Overall, this work presents the following contributions:
Guided Exploration from Noisy and Minimal Human Feedback: We propose an algorithm for separating (noisy) human feedback from policy learning by utilizing comparative, binary human feedback to guide exploration while learning a policy through self-supervision. This ensures that we can derive optimal goal-reaching policies even from imperfect human feedback.
Capability to learn from crowdsourced non-expert human feedback: We show HuGE can learn from noisy, asynchronous and infrequent human feedback. We validate this by collecting crowdsourced pilot data from over a hundred arbitrarily selected, non-expert annotators who interact with the robotic learning through a remote web interface.
Demonstration with real-world policy learning: We show that HuGE can learn policies on a robotic learning system in the real world, demonstrating effective exploration via human feedback and compatibility with pretraining from non-expert trajectory demonstrations as seen in Figure 1.
Our work builds on techniques for exploration methods, goal-conditioned, and human-in-the-loop RL, but presents a technique for learning from noisy, asynchronous and infrequent human feedback. We present the connections to each of these sub-fields below:
Exploration in RL: While exploration is a widely studied subfield in RL [10, 43, 5, 37, 9, 21], this is typically concerned with either balancing the exploration-exploitation tradeoff [4] or maximizing state coverage [5, 10, 37]. This contrasts with our goal of performing targeted exploration, informed by human feedback. Our work is related to ideas from Go-Explore [21], which explores by maintaining an archive of visited states, returning to states with a low-visitation count, and performing random exploration from there. However, this results in redundant overexploration shown in Figure 2 under the novelty-based exploration paradigm. In contrast, HuGE is able to leverage human feedback to only explore in promising directions as directed by human supervisors, significantly improving sample efficiency.
For goal-reaching problems, alternatives for exploration include self-supervision techniques such as hindsight relabeling [2, 24, 35]. By learning goal-reaching behavior in hindsight for reached states, these methods obtain dense supervision, albeit only for the states that were actually reached. This has been shown to empirically aid with exploration in certain cases, via policy generalization across goals. However, as shown in Figure 3, these methods suffer from underexploration, since the policy generalization across states can be challenging to control. In contrast, HuGE does not rely on arbitrary policy generalization, instead revisiting already reached states at the frontier of visited states and performing further exploration from these states.
Goal-conditioned reinforcement learning: Goal-conditioned RL algorithms [34, 24, 35, 2, 33, 28, 37] are multi-task RL methods where various tasks are defined as reaching different goal states. A number of different approaches have been proposed for goal-conditioned RL - learning with hindsight relabeling [2, 19, 33, 28], learning latent spaces for reward assignment [42, 48], learning dynamical distances [27, 22] and goal conditioned imitation learning [25, 35, 24, 40]. While these algorithms are able to solve tasks with a simple component of exploration, they can struggle with tasks with complex sequential nature, requiring complex machinery such as hierarchical architectures [33, 39]. In contrast, our work shows the ability to solve sequential goal-reaching problems with standard techniques, just using occasional comparative feedback from human supervisors to guide exploration.
Perhaps most closely related to our work is the paradigm introduced in DDL [27] where a human supervisor occasionally selects which states to explore from. This state is used to define the reward function, which when combined with self-supervised Q-learning, can aid with exploration for goal-conditioned RL problems. However, exploration is limited to exactly the state selected by the human, whereas in HuGE, learning the parametric goal selector allows the exploration frontier to continue expanding and decreases the overall amount of human feedback needed (Section 5, Figure 8).
Reinforcement Learning from Human Feedback (RLHF): RLHF typically characterizes a class of methods that learn reward models from binary human comparative feedback for use in RL. These techniques have seen use in better aligning language models, and guiding learning of embodied agents in simulation with expert feedback [16, 7, 8]. In much of this work, humans are assumed to provide high-quality and frequent feedback to learn rewards. In HuGE, we take a complementary view and show that by considering human feedback in a self-supervised policy learning setting, much of the burden can be placed on self-supervision, rather than on receiving high-quality human feedback. More broadly, human in the loop RL is a well explored concept, with various interfaces ranging from preferences [16, 7, 8, 31] to scalar rewards [29], language based corrections [53], binary right/wrong signals [12] and even sketching out rewards [11]. While we restrict the study
in this work to binary comparative feedback, exploring how the underlying principles in HuGE (of decoupling human feedback and self supervised policy learning) can be extended to other forms of feedback is an exciting avenue for future work.
Figure 2: Comparison of exploration types. hindsight relabelling: suffers from exploration collapse, novelty-based exploration suffers from overexploration, we propose directed exploration from human feedback Ours.
3 Problem Setup and Preliminaries
This work solves goal reaching tasks by using goal-conditioned policy learning methods. The goal reaching problem is characterized by the tuple  $\langle S, \mathcal{A}, \mathcal{T}, \rho(s_0), T, p(g) \rangle$ , adopting the standard MDP notation with  $p(g)$  being the distribution over goal states  $g \in S$  that the agent is tasked with reaching. We assume sampling access to  $p(g)$ . We aim to find a stationary goal-conditioned policy  $\pi(\cdot | s, g): S \times S \to \Delta(\mathcal{A})$ , where  $\Delta(\mathcal{A})$  is the probability simplex over the action space. We will say that a goal is achieved if the agent has reached the goal at the end of the episode (or is within  $\epsilon$  distance of the goal). The learning problem can be characterized as that of learning a goal conditioned policy that maximizes the likelihood of reaching the goal  $\pi \gets \arg \max_{\pi} J(\pi) = \mathbb{E}_{g \sim p(g)}[P_{\pi_g}(s_T = g)]$ .
Goal-conditioned Policy Learning: Goal-conditioned RL methods approach this problem using the reward function  $r_g(s) = \mathbb{1}(s = g)$ , defining a sparse reward problem:  $\pi \gets \arg \max_{\pi} \mathbb{E}_{\tau \sim \pi(\cdot | s, g), g \sim p(g)} \left[ \sum_{t=1}^{H} \gamma^t r_g(s_t) \right]$ . This problem can be difficult to solve with typical RL algorithms [26, 52, 23] because the training process is largely devoid of learning signals.
3.1 Why is exploration in goal-conditioned reinforcement learning challenging?
Hindsight relabelling: To circumvent the challenge of sparse rewards in goal-conditioned reinforcement learning, the structure of goal-reaching problems can be exploited using the hindsight relabeling technique [28, 1, 3]. Hindsight relabeling leverages the insight that transitions that may be suboptimal for reaching "commanded" goals  $g$ , may be optimal in hindsight had the actually reached states  $s_T$  been chosen as goals. This allows us to relabel a transition tuple  $(s, a, s', g, r_g(s))$ , with a hindsight tuple  $(s, a, s', g', r_{g'}(s))$ , where  $g'$  can be arbitrary goals chosen in hindsight. When  $g'$  is chosen to be states  $s$  actually visited along a trajectory, the reward function  $r_s(s) = 1$  provides a dense reward signal for reaching different goals  $g = s$ , which can be used to supervise an off-policy RL algorithm [3] or a supervised learning algorithm [1, 40, 47, 24].
The key to exploration in these methods is generalization - even though the desired goal distribution  $p(g)$  is not accomplished, learning policies on the accomplished goal distribution  $p(g')$  influences exploration towards  $p(g)$  via generalization between  $g'$  and  $g$ . However, this form of exploration is unreliable across domains and goal representations. As shown in Figure 2 and in Appendix E, this often underexplores the environment without accomplishing the target goal.
Novelty-based exploration: On the other hand, pure novelty-based exploration [21, 51, 5, 44] is also ineffective, since it performs task agnostic exploration of the entire state-space, thereby overexploring the environment (Fig 2). This becomes intractable with large state and action spaces.
Directed exploration (Ours): A practical solution would explicitly encourage exploration but directed towards the goal. In the following sections, we will argue that providing noisy, asynchronous and infrequent binary comparison feedback by a variety of human users can provide this directed exploration, without inheriting the noise and bias of the provided feedback.
4 HuGE: Guiding Exploration in Goal-Conditioned RL with Human Feedback
Figure 3: Overview of HuGE. We train a goal selector, from human feedback through state comparisons, to perform directed exploration in self-supervised learning.
The key idea behind HuGE is to leverage (noisy) human feedback for guiding exploration and data collection, but to decouple this from the process of learning goal-reaching policies from the collected data. We show how this can be done in a self-supervised way, allowing unbiased goal-reaching policies to be learned, while human feedback can still be useful in guiding exploration. We describe each component of our proposed algorithm below.
4.1 Decoupling Human Feedback from Policy Learning for Goal Reaching Problems
While human feedback has been shown to be an effective tool to guide exploration [16, 8], policies obtained by maximizing reward functions learned from noisy and infrequent human feedback can be very suboptimal, often getting trapped in local optima. This places a significant burden on human supervision, making it challenging for non-expert, casual supervisors to guide robot learning. How can we enable policy learning from infrequent and noisy human feedback?
Our key insight is that human feedback and policy learning can be disentangled by decoupling the process of exploration from the policy learning, while only leveraging the human feedback on the exploration process. In this new paradigm, where the generation of exploration data does not necessitate optimality, we can harness guiding signals originating from noisy human feedback, yet still effectively learning optimal goal-conditioned policies.
In particular, for human guided exploration we build on the idea of frontier expansion [21, 37, 14], where exploration is performed by revisiting states at the frontier of visited states and then continuing exploration from these. This technique would maintain a frontier  $\mathcal{F}$  that consists of the set of all visited states. We can then select a previously visited state for the policy to revisit, which we will call breadthcrumb state  $g_{b}$ , by querying a "goal selector" on  $\mathcal{F}$ , that is learned from (noisy) human feedback and defined in detail in 4.2. In this way, human feedback can be used to "softly" guide exploration towards promising states  $g_{b}$ . Frontier expansion can then be performed by executing random exploration from the reached breadthcrumb state  $g_{b}$ , and adding the collected data to the replay buffer for self-supervised policy learning.
Given data collected by frontier expansion directed using human feedback, goal-reaching policies can then be learned by leveraging self-supervised policy learning techniques [24, 2, 35] that are able to learn policies from collected data using "hindsight" relabeling and supervised learning. Importantly, this self-supervised policy learning procedure does not depend on the human feedback at all and is purely self-supervised. The decoupling of exploration and policy learning implies that even if this exploration is biased and imperfect, the self-supervised policy learns unbiased paths to reach all goals that were visited. In this way, cheap low-quality supervision from non-expert human feedback can
be used to ensure guided frontier expansion towards relevant subsets of the state space, while still being able to learn unbiased goal-reaching policies. Below, we delve into each of these components - guided frontier expansion and self-supervised policy learning and describe the overall algorithm in Algorithm 1.
4.2 Guiding Exploration from Human Comparative Feedback by Learning Goal Selectors
To direct exploration more effectively, we hypothesize that receiving small amounts of occasional guidance from non-expert human users can be beneficial, even if the feedback is biased and noisy. To leverage this feedback, we learn a parametric goal selection function directly from binary human comparison feedback. Learning a parametric goal selection function allows this feedback to be useful even when humans are providing feedback infrequently and asynchronously.
Learning State-Goal Distances from Binary Comparisons: We propose a simple interface between the human and the algorithm that simply relies on the human supervisor to provide binary comparisons of which of two states  $s_1, s_2$  is closer (given a human's judgment) to a particular goal  $g$  (as demonstrated in Fig 3 and Appendix Fig B.4). These binary comparisons can be used to train an unnormalized estimate of distances  $f_{\theta}(s, g)$  by leveraging the Bradley-Terry model of choice [16, 8, 7]:
$$
\begin{array}{l} \max  _ {\theta} \sum \mathbb {1} \left(s _ {1} > s _ {2} | g\right) \log \frac {\exp f _ {\theta} \left(s _ {1} , g\right)}{\exp f _ {\theta} \left(s _ {1} , g\right) + \exp f _ {\theta} \left(s _ {2} , g\right)} + \tag {1} \\ (1 - \mathbb {1} (s _ {1} > s _ {2} | g)) \log \frac {\exp f _ {\theta} (s _ {2} , g)}{\exp f _ {\theta} (s _ {1} , g) + \exp f _ {\theta} (s _ {2} , g)} \\ \end{array}
$$
This objective encourages states  $s$  closer to particular goals  $g$  to have smaller  $f_{\theta}(s, g)$ . While this estimate may be imperfect and noisy, it can serve to bias exploration in promising directions.
Using the Learned State-Goal Distances for Biasing Exploration: To guide exploration, we can select which breadth must be directed to the goal. We first consider the case where  $g_{b}$  is a command and start exploring from the current state  $g_{b}$ . By sampling states inversely proportional to their distance to the goal, we can find that the goal is either directed or not. This allows us to explore the optimal location of the goal in terms of the current state  $g_{b}$ . By using the learned state-space as a starting point, we can find the optimal location of the goal in terms of the current state  $g_{b}$ . By using the learned state-space as a starting point, we can find the optimal location of the goal in terms of the current state  $g_{b}$ .
4.3 Self-Supervised Policy Learning: Hindsight Relabeled Learning for Goal-Conditioned Policies
Given exploratory data collected by guided frontier expansion (Algorithm 2, Section 4.1), we can leverage a simple self-supervised learning scheme building on [1, 47, 24] for goal-conditioned policy learning. Given trajectories  $\tau = \{s_0, a_0, s_1, s_2, \ldots, s_T, a_T\}_{i=1}^N$  we to construct a dataset of optimal tuples using:  $\mathcal{D}_{\tau} = \{(s_t, a_t, g = s_{t + h}, h) : t, h > 0, t + h \leq T\}$  (3). This relabeled optimal dataset can then be used for supervised learning:  $J_{\text{Policy}}(\pi) = \mathbb{E}_{\tau \sim \mathbb{E}_g[\pi_{\text{old}}(\cdot | g)]} \left[ \sum_{t=0}^{T} \log \pi(a_t | s_t, g) \right]$  (4). This process can be repeated, iterating between collecting data, relabeling it, and performing supervised learning. As policy learning continues improving, the learned policy can be deployed to solve an expanding set of goals, eventually encompassing the desired goal distribution. The overall pseudocode of HuGE is shown in Algorithm 1. We note that this relabeled supervised learning scheme has been studied in past works [1, 47, 24], the unique contribution here being how low-quality human comparative feedback is used to guide exploration for self-supervised policy learning. While prior techniques such as [1, 47, 24] often struggle with exploration, the occasional human feedback can lead to significantly more directed exploration behavior for HuGE as compared to prior work.