SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Boostrapping Learning from Trajectory Demonstrations: While the system thus far has described a strategy that is applicable for learning policies from scratch, training HuGE from scratch can be prohibitively slow in the real world, and instead we may look to finetune already pre-trained "foundation" models or existing goal-directed policies using HuGE, or to pretrain from human demonstrations as considered in a huge plethora of prior work [49, 32, 6, 41].
Given that the self-supervised policy learning method in HuGE is fundamentally based on iterated supervised learning, a natural way to ensure the practicality of the method is by simply initializing the policy  $\pi$  using supervised imitation learning on trajectory demonstrations (or from other sources if available). While this technique is effective on policy pretraining, it fails to account for the goal selection model  $f_{\theta}(s,g)$ . A simple modification to the training process allows also for effective pre-training the state-goal distances model  $f_{\theta}(s,g)$  from demonstrations. Concretely, since demonstrations are typically monotonically decreasing in terms of effective distance to the goal, a single trajectory  $\tau$  can be reinterpreted as a set of  $\frac{n(n + 1)}{2}$  comparisons, where later states in the trajectory are preferred to earlier ones:  $\{s_{t_1} < s_{t_2}: t_1 < t_2 \forall s_{t_1}, s_{t_2} \in \tau\}$ . These comparisons can then be used to pre-train the goal selector, as discussed in Equation 1. Our proposed technique is not a replacement for learning from demonstrations, but rather serves a complementary role. As we show experimentally, this combination of HuGE and pre-training from non-expert demonstrations actually enables learning in the real world under practical time constraints.
Algorithm 1 HuGE: Guided Exploration with Human Feedback
1: Input: Human  $\mathcal{H}$ , goal distribution  $p(g)$
2: Initialize policy  $\pi$ , goal selector  $f_{\theta}$ , data buffer  $\mathcal{D}$ , goal selector buffer  $\mathcal{G}$
3: while True do
4: Sample goal  $g \sim p(g)$
5:  $\mathcal{D}_{\tau} \gets \text{PolicyExploration}(\pi, \mathcal{G}, \mathrm{f}_{\theta}, \mathrm{g}, \mathcal{D})$
6:  $\mathcal{D}\gets \mathcal{D}\cup \mathrm{RelabelTrajectory}(\mathcal{D}_{\tau})$  (3)
7:  $\pi \gets$  TrainPolicy  $(\pi ,\mathcal{D})$  (4)
8:  $\mathcal{G}\gets \mathcal{G}\cup \mathrm{CollectFeedback}(\mathcal{D},\mathcal{H})$  (4.2)
9:  $f_{\theta}\gets \mathrm{TrainGoalSelector}(f_{\theta},\mathcal{G})$  (1)
Algorithm 2 PolicyExploration
1: Input: policy  $\pi$ , goal selector  $f_{\theta}$ , goal  $g$ , data buffer  $\mathcal{D}$
2:  $g_{b}\sim$  SampleBreadcrumb  $(f_{\theta},g,\mathcal{D})$  (2)
3:  $\mathcal{D}_{\tau}\gets \{\}$
4: for  $i = 1,2,\dots ,N$  do
5:  $s\gets s_0$
6: while NOT stopped (see G.1) do
7: Take action  $a \sim \pi(a|s, g_b)$
8: Execute  $\pi_{\mathrm{random}}$  for  $H$  timesteps
9: Add  $\tau$  to  $\mathcal{D}_{\tau}$  without redundant states
0: return  $\mathcal{D}_{\tau}$
In this work, we show that HuGE learns to successfully accomplish long-horizon tasks, and tasks with large combinatorial exploration spaces through little human supervision. To demonstrate these experimentally, we test on several goal-reaching domains in simulation, shown in 4, in the MuJoCo [54] and PyBullet [17] simulators where we compare against state-of-the-art baselines. Furthermore, we show the benefits of our method by learning policies directly on a real-world LoCoBot robot. We additionally show the ability for HuGE to be used with an uncurated, large-scale crowdsourcing setup with non-expert human supervisors. With these experiments, we show 1) HuGE outperforms prior work on solving long horizon goal-reaching tasks in simulation; 2) HuGE is suitable for collecting crowdsourced, noisy and asynchronous feedback from humans all over the world with different backgrounds and education levels; 3) HuGE is suited to learning in the real world with a robotic platform in practical time-scales, directly from visual inputs.
Bandu
Figure 4: Six simulation benchmarks where we test HuGE and compare against baselines. Bandu, Block Stacking, Kitchen, and Pusher, are long-horizon manipulation tasks; Four rooms and Maze are 2D navigation tasks, see Appendix D
Block stacking
Pusher two walls
Figure 5: Success curves of HuGE on the proposed benchmarks compared to the baselines. HuGE outperforms the rest of the baselines, some of which cannot solve the environment while converging to the oracle accuracy. For those curves that are not visible, it means they never succeeded and hence are all at 0 (see E.10 for distance curves). The curves are the average of 4 runs, and the shaded region corresponds to the standard deviation.
5.1 Learning Goal-Conditioned Policies with Synthetic Human-in-the-Loop Feedback in Simulation
We consider goal-reaching problems in the six domains shown in Fig 4. These are domains with non-trivial exploration challenges—the agents must assemble a structure in a specific order without breaking it, navigate around walls, etc., and purely random exploration is unlikely to succeed. We evaluate HuGE compared to the baselines described in Appendix E on these domains. These baselines were chosen to compare HuGE with methods that perform different types of exploration, hindsight relabeling, and methods that use human preferences to learn a reward function instead, to highlight the benefits of decoupling exploration from human preferences and policy learning. We report the number of final goals reached successfully in Fig 5 as learning progresses. Only in these and the analysis experiments, the human feedback is synthetic (see Appendix D).
In Figure 5 and in Appendix E with further detail, we show HuGE learning a goal selector model to perform goal selection (Ours) matches, at convergence, the performance of using an Oracle always choosing the best goal to explore. Guiding exploration (Ours) is significantly better than techniques that perform indiscriminate exploration (Go-Explore+GCSL) [21, 37]. Ours also beats methods that purely rely on policy generalization and stochasticity to perform exploration and do not explicitly expand the frontier (Inverse Models) [1, 47, 24] which fail in complex exploration domains. Goal-conditioned reinforcement learning methods with PPO [52] and sparse rewards do not experience enough reward signals to actually learn directed behavior. In PPO with dense reward, we use the same reward as for generating the synthetic human labels and we observe, that in most cases the performance is lower. The reason is that we did not do any reward engineering for either PPO or HuGE, showing that Ours is more robust to simpler underlying reward functions than PPO, we analyze these results further in Appendix E.2. Similarly, using human feedback to bias exploration Ours beats using it to learn a reward function (Human Preferences)[16]. Finally, in Appendix C we show that pretraining HuGE with 5 noisy trajectories  $(BC + Ours)$  gives a significant decrease in the number of timesteps needed to succeed and beats plain Behavior Cloning  $(BC)$  and also compared to other baselines with demonstrations.
5.2 Learning Goal-Conditioned Policies with large-scale crowdsourced data collection
In this section, we show HuGE works on the kitchen environment from a crowdsourced data collection of 109 non-experts annotators labeling asynchronously and from all over the world, as shown in Figure 6. We spanned across three continents, having annotators living in 13 different countries, with ages ranging from 18 to  $65+$  and a variety of academic backgrounds, and we collected 1600 labels, we refer the reader to B.1 and B.2 for more details about the demographics. Each annotator
Figure 6: left: Crowdsourcing experiment learning curves for the kitchen, middle: human annotators spanned 3 continents and 13 countries, right: screenshot of the interface for data collection.
could provide labels at any time during the day, our recommendation was to provide 30 labels, which took on average less than 2 minutes of their time. We collected a total of 2678 labels for this crowdsourcing experiment beating Ours when labels are synthetically generated. This indicates that the crowdsourced data might provide more information than the simplistic reward function, designed to provide the synthetic feedback. We repeated the experiment collecting just 1670 labels from 4 annotators and saw that the crowdsourcing experiment also yield better results, suggesting that a wider variety of feedback does not damage performance. We refer the reader to Appendix B for additional results of HuGE from real human feedback on other simulated benchmarks, details on the platform we developed for these experiments, and where we clarify that this study was approved by the Institutional Review Board.
5.3 Learning Goal-Conditioned Policies in the real world
Figure 7: Accomplished goals at the end of 5 different evaluation episodes throughout training in the real world.
HuGE's qualities of being robust to noisy feedback and requiring minimal and asynchronous human supervision together with its self-supervised policy learning nature and the capability to be pretrained from trajectory demonstrations makes it suitable for learning in the real world. As shown in Fig 7, HuGE can learn policies for pick and place and drawing in the real world with a LoCoBot [38]. Some adaptations were made to run HuGE on real hardware such as changing the state space to images, instead of point space. In Appendix A we provide more results on HuGE from image space in the real world and simulation. We also pretrained our policy with 5 trajectory demonstrations (Appendix C. For both experiments, the robots learned directly on the real hardware. In the pick and place experiment, we collected around 130 labels across 20 hours of real-world robot training, whereas in the drawing one, we collected 150 labels across 6 hours.
5.4 Ablation Analysis
Lastly, we conducted a number of quantitative analysis experiments to better appreciate and convey the benefits of HuGE. In Figure 8, we show that learning a parametric goal selector (Ours) is more sample efficient than directly using the human-selected goals (DDL [27]). More concretely, learning a parametric goal selector needs  $50\%$  less human annotations than DDL [27]. The underlying
idea is that by deriving a goal selector from human feedback, we extract recurrent patterns and generalizations that can be redeployed in future rollouts. Conversely, without learning from this goal selector, we would overlook these patterns, leading to more frequent queries for human input.
One of the properties of HuGE is that it works from noisy human feedback. In Figure 8, we observe that despite adding large amounts of noise on the human annotations, HuGE still converges to the optimal solution. This supports the idea that when the human feedback is noisier, exploration will be less efficient and will take longer to converge nevertheless, still converging to an optimal policy.
Furthermore, in Figure 8 (right), we provide a visualization explaining why optimizing over the reward function (Human Preferences [16]) can provide non-optimal solutions. We observe reward functions learned from noisy human feedback are suboptimal, with multiple local maxima, and hence optimizing over them will potentially lead to the policy getting stuck in local optima. On the other hand, HuGE, which samples goals from this noisy learned reward function will still converge to the optimal solution. The explanation is that the goal selector will steer exploration towards the three modes uniformly, making exploration suboptimal but still converging to an optimal goal-conditioned policy reaching each one of the modes, the goal included.
In Appendix F, we explain further analysis as the trade-off between the amount of human querying and sample complexity of HuGE, as well as showing that HuGE is also robust to incomplete feedback and further details of its robustness to noisy underlying reward functions compared to PPO.
Figure 8: (left): Learning a goal selector (Ours) needs on average  $50\%$  fewer labels than not (DDL) (middle): As the noise in the feedback increases, so will the number of timesteps to succeed, however HuGE still finds a solution. (right): HuGE is robust to noisy goal selectors since trajectories going to each mode will be sampled while if we ran RL the policy would become biased and fail. See Appendix F for more details.
In this work, we build from the insight of decoupling human feedback from policy learning. We introduced HuGE, that guides exploration by leveraging small amounts of noisy and asynchronous human feedback, improving upon indiscriminate exploration or relying on hindsight generalization. Moreover, the policy training using self-supervised learning remains disentangled from the human feedback and makes it robust to asynchronous and noisy human feedback. We firmly believe this insight is key to learn policies from crowdsourced human feedback, where the noise in the labels due to non-expertise and variety of the annotators will be significant. As we show, when the annotations are noisy, optimizing over the learned noisy reward function using standard RL techniques will fail. However, once we disentangle the human feedback from the learned policy, and only use this to softly guide exploration, we can still learn optimal policies despite the noise. We rigorously demonstrate that HuGE is able to solve difficult exploration problems for various control tasks learning in both the real world and in simulation. Moreover, we run a crowdsourced data collection experiment with 109 human annotators living across three continents, with a wide variety of backgrounds, education level, ages, and we show HuGE succeeds in learning optimal policies. Finally, we provide ablations and analysis on the design decisions and better show the benefits of HuGE.
There are a number of directions for future work that are very exciting building on HuGE. On an immediate note, adapting HuGE to work in reset-free environments so that we can scale up the applications on learning on the real robot is the most promising. It is with excitement that we would like to see this system scaled to fine-tune foundation models in robotics. In the future, aiming to learn behaviors not just from bianry comparative feedback, but from richer forms of communication such as language or physical corrections or vector valued feedback would be intriguing as well.
We thank all of the participants to our human studies that gave us some of their time to provide labels. We thank the members of the Improbable AI Lab and the WEIRD Lab for their helpful feedback and insightful discussions. We also thank the reviewers for their insightful comments and helping us make the paper stronger.
The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper. This research was supported by MIT-IBM Watson AI Lab.
8 Contributions
Marcel Torne and Abhishek Gupta jointly conceived the project. Marcel set up the simulation and training code, designed and conducted experiments in simulation, designed and implemented the interface for collecting human feedback, led the human experiments, conducted the ablations, and made the figures. Marcel led the manuscript writing together with Abhishek. Max Balsells designed and conducted the experiments in the real-world, integrated the vision models in the code, helped running some of the simulation experiments and ablations and helped writing the manuscript. Zihan Wang provided feedback on the manuscript. Samedh Desai helped Max setting up the real-world experiments. Tao Chen was involved in the initial research discussions and provided feedback on the paper. Pulkit Agrawal was involved in research discussions, contributed some of the main ideas behind the project and provided feedback on the writing and positioning of the work. Abhishek Gupta conceived the project jointly with Marcel, led the manuscript writing together with Marcel, and provided the main overall advising.
[1] P. Agrawal, A. V. Nair, P. Abbeel, J. Malik, and S. Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
[2] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In I. Guyon, U. von Luxburg, S. Bengio, H. M. Wallach, R. Fergus, S. V. N. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5048-5058, 2017.
[3] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048-5058, 2017.
[4] M. G. Azar, I. Osband, and R. Munos. Minimax regret bounds for reinforcement learning. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 263-272. PMLR, 2017.
[5] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pages 1471-1479, 2016.
[6] A. Billard and G. Hayes. Learning to communicate through imitation in autonomous robots. In W. Gerstner, A. Germond, M. Hasler, and J. Nicoud, editors, Artificial Neural Networks - ICANN '97, 7th International Conference, Lausanne, Switzerland, October 8-10, 1997, Proceedings, volume 1327 of Lecture Notes in Computer Science, pages 763-768. Springer, 1997.
[7] E. Biyik. Learning preferences for interactive autonomy. CoRR, abs/2210.10899, 2022.
[8] E. Biyik and D. Sadigh. Batch active preference-based learning of reward functions. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages 519-528. PMLR, 2018.
Next, we provide additional details of our work. More concretely:
- Appendix A Real-Robot experiments: provides more details on the real-robot experiments and on HuGE working from images.
- Appendix B Real-Human experiments: shows all the details of the real human experiments, presenting results for both the crowdsourcing experiment as well as experiments on more benchmarks with fewer annotators.
- Appendix C HuGE from demos: we show HuGE is compatible with warm starting from demonstrations.
- Appendix D Simulated Benchmarks: we provide further details on the simulated benchmarks used in this work.
- Appendix E Baselines: we provide details about the baselines together with more detailed learning curves of the baselines on the simulated benchmarks.
- Appendix F Further Analysis and Ablations: we provide more insights on where the benefits of HuGE come from, as well as provide some ablations on the method.
- Appendix G Implementation Details: we provide further implementation details, hyperparameters and resources used.
The code is available at github.com/Improbable-AI/human-guided-exploration
A Real-Robot Experiments
HuGE's qualities make it suitable to learn policies directly in the real world. However, we adapted the method with respect to the simulated experiments in Fig 5. The main change consisted in changing the state space to image space instead of point space. Next, we show HuGE works from image space in two of the simulated environments, four rooms and block stacking.
A.1 HuGE from images
Adapting HuGE to work from image space was a trivial process, the goal selector and policy networks were modified introducing a first encoding network consisting of 3 convolutional layers (with stride 2 and kernel size 5) to map the input image to a lower dimension space and then we passed it through an MLP to predict the score and action respectively.
Learning stopping criteria: When using the point state space, we could easily detect whether the policy stopped, indicating it reached the target goal, or that it got stuck, to then start random exploration from there. This could be done by computing the Euclidean distance and setting a small enough threshold. This is more difficult when working from image state space. What we did was train an image classifier  $\phi(s_1, s_2)$  that predicts whether the two images correspond to states close in space (i.e., the state corresponding to image  $s_2$  can be reached from the state corresponding to image  $s_1$  within  $t_{\mathrm{close}}$  timesteps). We trained  $\phi$  by using contrastive learning [15]. In particular, we sampled images from our replay buffer and assigned the corresponding label based on their distance in timesteps:  $l(s_i, s_j) = 1$  if  $|i - j| \leq t_{\mathrm{close}}$  and 0 if  $|i - j| \geq t_{\mathrm{far}}$ . Based on the premise that, in most cases, images obtained far away in time, will probably correspond to states that take longer than  $t_{\mathrm{close}}$  timesteps to reach, if we were to act optimally.
A.2 Results in the real-world
For the real robot experiment, we used a LoCoBot with a WX-200 arm.
Pick and place in the real world: The state space consisted of RGB images of  $64 \times 64$  pixels, and the action space was continuous with dimension 2, representing an absolute position in the space  $(x, y)$  from which to predict a grasping point in even timesteps, or a dropping point in odd timesteps. For the experiment to be succeed in a reasonable amount of time, we pretrained the policy and the goal selector by using 5, sub-optimal demonstrations. The robot was trained for around 30h, during which, 130 labels were provided via the interface shown in B.4 by one annotator. Finally, we used a reset mechanism to pull the socks to the same corners, though, it had some stochasticity.
Figure A.1: Success rate for the four rooms (left) and block stacking (right) using images as input space for both the policy and goal selector.
Figure A.2: Accomplished goals at the end of 5 different evaluation episodes along training on the real world to pick and place, and draw the letter U in the real world.
Drawing in the real world: The state space consisted of RGB images of  $64 \times 64$  pixels, and the action space was discrete, encoding a total of 5 actions: no movement and moving across the two axis on the plane in polar coordinates (i.e. increasing  $r$ , decreasing  $r$  and moving a fixed amount clockwise or counterclockwise), to move the end effector with the brush. An episode consisted of 12 timesteps. For the experiment to be ran in a reasonable amount of time, we pretrained the policy and the goal selector by using 5, sub-optimal demonstrations. The robot was trained for around 6h, during which, 150 labels were provided via the interface shown in B.4 by one annotator. Finally, the reset was done by using the erase mechanism in this drawing boards and moving it with the arm by using a script. As a final note, in this environment we had to perform few exploration steps and slowly increase the frontier. This is because in this environment there is only one optimal solution (the actions taken must be exactly the optimal ones, due to the fact that all past actions within the episode will affect the current state of the board), in particular, any non-optimal action will leave a trace, making that trajectory not that useful for the policy to learn from it.