SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Human Alignment evaluation for drawing in the real world: Designing a reward function for drawing is a hard and tedious labor. HuGE does not need a reward function and we can fully leverage human feedback to learn this behavior as shown in A.2. Without a reward function evaluation cannot be performed either. For this reason, we defined this "Human Alignment Score" that basically consists in querying humans and asking them for a score between 0 and 10 of how well the robot draw the target picture. In the case of the drawing experiments, we asked 2 annotators to label the performance of the robot drawing the letter U with a score from 0 to 10. This score was only used for evaluation and is the metric used to plot the drawing plot in A.2.
B Real-Human experiments
Figure B.3: Learning Progress with Human-in-the-Loop Feedback for the kitchen, block stacking, four rooms navigation, pusher walls and complex maze.
In this section, we give more details on how we ran the human experiment. We designed a simple interface shown in Figure B.4. We can see the two states to be compared in blue and red and the goal we aim to achieve in green. Then the annotator has to decide which one of the two states is closer to the given goal and provide feedback by clicking either on the blue or red button. In the case in which the annotator is undecided, they can click on the gray button that simply skips the current case. Finally, if the annotator does not provide any feedback after 30 seconds of being presented with the scenario, we skip the current batch of labeling and continue with training the policy. With this, we can take advantage of the properties of our method and continue training the policy even when no labels are given.
In Figure B.3 we share again the results obtained with the human experiment on a larger scale. We ran both experiments using the same frequency of labeling and number of labels per batch. In particular, We labeled every 50 rollout trajectories and queried the annotators for 20 labels. These parameters were identified through empirical experiments.
Human Guided Exploration (HUGE)
Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-loop Feedback
Figure B.4: Screenshot of the interface from our proposed crowdsourcing platform. It shows a comparison of two image states of the kitchen environment, and the user needs to click one of the three buttons below depending on their answer of which one is best: Left/I don't know/right
B.1 Details about the crowdsourcing experiment
Subject 109 subjects participated in this pilot crowdsourcing study. Subjects were recruited from the acquaintances of the collaborators. The average time to complete the study was about 2 minutes. The subjects participated voluntarily without financial remuneration. The participants age ranged from 18 to  $65+$  years old. Gender Male  $= 58.7\%$ , Female  $= 39.4\%$ , Non-binary  $= 1.8\%$ . The participants were from 21 nationalities and participated from 13 distinct countries. More detailed information is presented in tables B.1 and B.2. There is no reason to believe that subjects experienced any physical or mental risks in the course of these studies.
Procedure This study was approved by the Institutional Review Board of the Massachusetts Institute of Technology protocol E-4967.
We provide the participants with the following detailed instructions:
Thank you very much for participating in this study. It should not take you more than a couple of minutes to complete. First of all click on the link we sent you to get directed to the main page (B.4), you can either use your phone or your computer. Please, start by filling out the form for us to get an overview of the participants' demographic. The task consists of controlling a robot to do different things in the kitchen: 1) open the slider on the right, 2) open the microwave on the right 3) open the hinge cabinet on the left. [We show a video of a successful trajectory]. To help us, we will present you two images and you need to tell us which one of the two images is closer to achieving the task. Click on the left/right button depending on whether the left/right image is better. If at some point you don't know which one is best please click the "I don't know" button. [We present a couple of examples demonstrating this]. We will show 30 pairs of images and after that, you will receive a message saying you completed the task. You can stop at any moment before that if you want.
Figure B.5: Heatmap on the country representation during our crowdsourcing experiment.
Metric	Percentage
Current country of Residence
USA	41.3% (45)
Spain	30.3% (33)
India	8.3% (9)
Germany	6.4% (7)
Canada	2.8% (3)
France	2.8% (3)
Singapore	1.8% (2)
China	1.8% (2)
Andorra	0.9% (1)
Austria	0.9% (1)
Ireland	0.9% (1)
Switzerland	0.9% (1)
United Kingdom	0.9% (1)
Prefer not to say	0% (0)
Gender
Male	58.7% (64)
Female	39.4% (43)
Non-binary	1.8% (2)
Prefer not to answer	0% (0)
Age group
18-24	48.6% (53)
25-34	24.8% (27)
35-44	7.3% (8)
45-54	11.0% (12)
55-64	7.3% (8)
65+	0.9% (1)
Prefer not to answer	0% (0)
Education
Graduate or professional degree	39.4% (43)
College degree	33.9% (37)
High school or some college	20.2% (22)
Other	12.8% (14))
Prefer not to say	2.8% (3)
Table B.1: Demographics on the participants of the crowdsourced data collection experiment
Metric	Percentage
Nationality
Spain	26.6% (29)
USA	20.2% (22)
India	9.2% (10)
Germany	8.3% (9)
China	7.3% (8)
France	4.6% (5)
Mexico	3.7% (4)
Colombia	2.8% (3)
Switzerland	1.8% (2)
Hong Kong	1.8% (2)
Canada	1.8% (2)
Uruguay	0.9% (1)
Singapore	0.9% (1)
Russia	0.9% (1)
Ireland	0.9% (1)
Lebanon	0.9% (1)
South Korea	0.9% (1)
Sweden	0.9% (1)
Andorra	0.9% (1)
Puerto rico	0.9% (1)
Israel	0.9% (1)
Prefer not to say	0.9% (1)
Ethnicity
Hispanic, Latino or Spanish	38.5% (42)
Asian	28.4% (31)
White or Caucasian	24.5% (27)
Middle Eastern or North African	3.7% (4)
South-east Asian	2.8% (3)
Black or African American	0.9% (1)
Perfer not to say	0.9% (1)
Table B.2: Demographics on the participants of the crowdsourced data collection experiment
C HuGE from demonstrations
As explained in Section 4.3, HuGE is suited to easily incorporate demonstrations and therefore use these to warm start the method providing a significant reduction in sample complexity over HuGE without demonstrations. Below we show a comparison of HuGE against other baselines that use demonstrations.
Figure C.6: Comparison of HuGE warmstarted with demonstrations. We observe that a significant speed up is provided compared to running HuGE without demonstrations. Furthermore, we observe an improvement over simply learning from demonstrations (Behavior Cloning, BC) baselines. Similarly, HuGE with demonstrations continues to outperform the PPO and Human Preferences with demonstrations baselines.
D Simulated Benchmarks
Bandu
Red piece is place and picks blue piece
Red and blue pieces placed, picks green piece
Places green piece on top of blue and red pieces
Places final piece and reaches goal configuration
Start Configuration of Pieces
Block Stacking
Start Block Stacking
Picks red block
- Drops red on target and picks green
Stacks green and picks blue
Three blocks stacked
Kitchen
Start Kitchen
Reaches slider
Opens slider
Moves puck around second obstacle
Opens cabinet
Pushers with walls
Start Pusher Walls
Figure D.7: Results of our method on four of the hardest benchmarks. From left to right, the timestep in the trajectory increases.
Reaches puck
Moves puck around first obstacle
Puck reaches goal
Pusher two walls
Figure D.8: Results of our method on four of the hardest benchmarks. From left to right, the timestep in the trajectory increases.
In this section, we give more details on the benchmarks used to compare our method with the baselines. All of these benchmarks are variations of benchmarks presented in previous work. In general, we have made them harder to showcase the benefits of our method. More concretely, for each method, we will give an overview of the difficulties it has and we will present the reward function we designed to provide synthetic labels in our experiments.
1. Four rooms (small 2D Navigation): We consider goal-reaching problems in a 2-D navigation problem in a four rooms environment, shown in Fig D.8. The challenge in this environment is navigation through obstacles, which are unknown without exploration. The agent is initialized in the bottom right room and the goal is sampled from the top right room. The state observation of this environment is the absolute position of the agent in the world, i.e. a vector  $(x,y)$ , and the action space is discrete with 9 possible actions, encoding 8 directions of movement (parallel to the axis and diagonally), plus a non-move action. To solve this benchmark the agent needs to traverse the two intermediate rooms to get to the target room, traversing a total of four rooms.
The reward function in this case is the shaped distance between the state and the goal. This benchmark is a modification of the benchmarks proposed by [24].
2. Maze (large 2D Maze Navigation): We consider a second 2-D navigation problem in a maze environment. The additional challenge in this environment compared to the previous one relies upon having a longer horizon (see Figure G.4). The agent is initialized in the green dot and has to reach the red dot. The state space is the absolute position of the agent in the maze, i.e. a vector  $(x,y)$ , and the action space is the same as in the Four rooms one, i.e. discrete with dimension 9.
The reward function in this case is the shaped distance between the state and the goal.
3. Pusher two walls: This is a robotic manipulation problem, where a Sawyer robotic arm pushes an obstacle in an environment with multiple obstacles. The puck and arm start in the configuration seen in Fig D.8. The task is considered successful if the robotic manipulator brings the puck to the goal area, marked with a red dot.
The state space of this environment consists of the position of the puck and the position of the arm. The action space is the control of the position of the robotic arm. It is also a 9-dimensional discrete action space where each one corresponds to a delta change in the position in 2D. This benchmark is a modification of the benchmarks proposed by [24]. The reward function designed for this environment is the following:
$$
r = \max  (d i s t a n c e \_ p u c k \_ f i n g e r, 0. 0 5) + d i s t a n c e \_ p u c k \_ g o a l
$$
4. Sequential Kitchen Manipulation: This benchmark is a harder robotic manipulation task where apart from being long horizon the agent needs to show three different skills to solve the task. We operate a 7 DoF Franka robot arm in a simulated kitchen, manipulating different cabinets, sliding doors, and other elements of the scene. The observation space consists of the position of the end effector and its rotation together with the joint states of the target objects. The action space consists in controlling the end effector position in 3D, we discretize it so the dimension is 27, and the control of the gripper and rotation of the arm. In our evaluation, we consider tasks where the goal is to sequentially manipulate three elements in the kitchen environment - the sliding cabinet, the microwave and the hinge cabinet to target configurations. The reward function we use is the following:
$$
r = \left\{ \begin{array}{l} - \text {d i s t a n c e} (\text {a r m}, \text {h i n g e c a b i n e t}) - | \text {h i n g e c a b i n e t} \text {t a r g e t} \text {j o i n t} - \text {h i n g e c a b i n e t} \text {c u r r e n t} \text {j o i n t} |, \text {i f s l i d e c a b i n e t a n d m i c r o w a v e o p e n e d} \\ - \text {d i s t a n c e} (\text {a r m}, \text {m i c r o w a v e h i n g e}) - | \text {m i c r o w a v e} \text {t a r g e t} \text {j o i n t} - \text {m i c r o w a v e} \text {c u r r e n t} \text {j o i n t} | - \text {b o u n s}, \text {i f s l i d e c a b i n e t o p e n e d} \\ - \text {d i s t a n c e} (\text {a r m}, \text {s l i d e c a b i n e t h i n g e}) - | \text {s l i d e c a b i n e t} \text {t a r g e t} \text {j o i n t} - \text {s l i d e c a b i n e t} \text {c u r r e n t} \text {j o i n t} | - 2 \text {b o u n s}, \text {o t h e r w i s e} \end{array} \right. \tag {5}
$$
5. Block Stacking: This domain is another long horizon robotic manipulation task, we operate a 6 DoF UR5 robot arm with a suction gripper as an end effector in a simulated tabletop configuration, stacking blocks. The observation space consists of the position of the end effector and the position of each block in 2D, and a bit indicating whether the hand is holding a block. This is a continuous action space domain with dimension 2, where the agent will predict a grasp position if it does not hold an object and a drop position if it is holding an object. We consider the goal to be accomplished if the three blocs are stacked in the correct
order (red, green, blue) on the correct fixed place on the table. Our code was inspired from the Ravens benchmark [57]. The reward function is the following:
$$
r = - \left\{ \begin{array}{l} \text {d i s t a n c e (a r m , b l u e b l o c k)} + \text {d i s t a n c e (b l u e b l o c k , t a r g e t g o a l) , i f r e d a n d g r e e n b l o c k a t p o s i t i o n} \\ \text {d i s t a n c e (a r m , g r e e n b l o c k)} + \text {d i s t a n c e (g r e e n b l o c k , t a r g e t g o a l)} + \text {b o u n s , i f r e d b l o c k a t p o s i t i o n} \\ \text {d i s t a n c e (a r m , r e d b l o c k)} + \text {d i s t a n c e (r e d b l o c k , t a r g e t g o a l)} + 2 \text {b o u n s , o t h e r w i s e} \end{array} \right. \tag {6}
$$
$$
r = - \left\{ \begin{array}{l} \mathrm {d} (\text {a r m}, \text {b l u e}) + \mathrm {d} (\text {b l u e}, \text {g o a l}), \text {i f r e d a n d g r e e n a t g o a l} \\ \mathrm {d} (\text {a r m}, \text {g r e e n}) + \mathrm {d} (\text {g r e e n}, \text {g o a l}) + \text {b o u n s}, \text {i f r e d a t g o a l} \\ \mathrm {d} (\text {a r m}, \text {r e d}) + \mathrm {d} (\text {r e d}, \text {g o a l}) + 2 \text {b o u n s}, \text {o t h e r w i s e} \end{array} \right. \tag {7}
$$
6. Bandu: This domain is very similar to the block stacking. We operate a 6 DoF UR5 robot arm with a suction gripper as an end effector in a simulated tabletop configuration. The observation space consists of the position of the end effector and the position of each block in 2D, and a bit indicating whether the hand is holding a block. This is a continuous action space domain with dimension 2, where the agent will predict a grasp position if it does not hold an object and a drop position if it is holding an object. We consider the goal to be accomplished if the four blocs are stacked in the target configuration building the castle like structure seen in Figure D.8. Our code was inspired from the Ravens benchmark [57]. The reward function is the following:
$$
r = \left\{ \begin{array}{l} - \text {d i s t a n c e} (\text {a r m}, \text {y e l l o w s t a r}) - \text {d i s t a n c e} (\text {y e l l o w s t a r}, \text {t a r g e t y e l l o w s t a r}), \text {i f a l l e x c e p t s t a r a t p o s i t i o n} \\ - \text {d i s t a n c e} (\text {a r m}, \text {g r e e n b l o c k}) - \text {d i s t a n c e} (\text {b l u e g r e e b l o c k}, \text {t a r g e t g r e e b l o c k}) - \text {b o u n s}, \text {i f r e d a n d b l u e b l o c k s a t p o s i t i o n} \\ - \text {d i s t a n c e} (\text {a r m}, \text {b l u e t r i a n g l e}) - \text {d i s t a n c e} (\text {b l u e t r i a n g l e}, \text {t a r g e t b l u e t r i a n g l e}) - 2 \text {b o u n s}, \text {i f r e d c y l i n d e r a t p o s i t i o n} \\ - \text {d i s t a n c e} (\text {a r m}, \text {r e d c y l i n d e r}) - \text {d i s t a n c e} (\text {r e d c y l i n d e r}, \text {t a r g e t r e d c y l i n d e r}) - 3 \text {b o u n s}, \text {o t h e r w i s e} \end{array} \right. \tag {8}
$$
More details about how these benchmarks were run, such as the number of episodes we ran the benchmarks for, are presented in Section G
E Baselines
We compare HuGE to relevant baselines from prior work.
1. Inverse Models: We compare with the iterative supervised learning algorithm for goal-reaching introduced in [24], consisting of hindsight relabeling without additional exploration (GCSL).
2. Learning from Human Preferences: We consider the technique introduced in [16], which learns a goal-agnostic reward model using binary cross-entropy. This learned reward is then combined with an on-policy RL algorithm [52] to learn the policy.
3. DDL: Dynamical Distance Learning [27] proposes a method to learn a goal-conditioned reward function by regressing on the time distance between states achieved in the same trajectory. A human synchronously provides preferences on which state brings the agent closest to the goal, note that no goal selector is being learned. The policy is then trained to maximize the learned reward to get to this selected state.
4. Go-Explore/LEXA: We compared with a version of goal-reaching with indiscriminate exploration. In particular, we perform frontier goal selection by identifying goals with the lowest densities. The policy returns to these states and perform random exploration from there. This is equivalent to performing indiscriminate exploration.
5. Proximal Policy Optimization: We compare with an on-policy algorithm [52] with both a standard sparse and dense reward to directly optimize the goal-reaching objective.
6. Behavior Cloning: Supervised learning on a batch of expert trajectories. In our experiments we use 5 expert trajectories.
7. Behavior Cloning + Ours: We pretrain the policy using imitation learning and we warm start our goal selector by training it from the expert trajectories. Given two random states in the same expert trajectory we add them into the training data for the goal selector, setting the state further in time as closest to the goal.
These baselines are chosen to compare HuGE with methods that perform pure exploration, hindsight relabeling, and human preferences without being goal conditioned to highlight the benefits of combining goal-driven self-supervision with human-in-the-loop exploration guidance.
Figure E.9: Failure modes of exploration algorithms for goal-reaching. Inverse models (top) collapses and does not discover the target room (second room at the top). Uniform frontier expansion (middle) does reach the target room, but to get there it visits all possible rooms, since exploration is indiscriminate. Directed frontier expansion (bottom, ours) reaches the target room much faster by leveraging human signal on direction. Training epochs increase from left to right. Each subfigure is an aerial view of a floor with 9 rooms, with multiple trajectories, each one in a different color.
Figure E.10: Distance to the goal for each method on different benchmarks. We note that the LEXA-like exploration strategy was only implemented on the four rooms benchmark.
For the sake of concreteness, we will study two simple schemes from prior work on solving goal-reaching problems—self-supervision via goal conditioned supervised learning [24] (as described in Section 3) and reinforcement learning with density based exploration [37]. Exploration in GCSL relies on generalization of the policy across goals, while density based exploration rewards exploring the most novel states. We show these algorithms can fail in different ways for a simple maze environment shown in Fig E.9, where the agent starts in the middle room and must reach goals commanded in the top middle room.
As shown in Fig E.9, GCSL exploration quickly collapses in the maze environment. This can be understood by noticing that self-supervised training on goals in the bottom right corner room or even the bottom left corner room does not extrapolate to the top right corner, where the commanded goals are. Instead of navigating the agent around the walls, the policy generalization suggests that the agent simply go into the wall as shown in Fig E.9.
Exploration methods are meant to tackle this kind of degenerate exploration, by encouraging visitation of less frequently visited goals at the "frontier" of visited states. When applied to the goal-reaching problem, in Fig E.9, we see that while the exploration is not degenerate, exploration is indiscriminate in that it explores both sides of the maze even though commanded goals are only down one particular path. While this will eventually succeed, it incurs a significant cost of redundant exploration by going down redundant paths.
This suggests that frontier expansion is needed like exploration methods, but should ideally be done in a directed way towards goals of interest. In Figure E.9 we see how this directed exploration could be useful and reduce sample complexity, by removing the need for indiscriminate frontier expansion. We show how a small amount of relatively cheap human feedback can be leveraged to guide this exploration.
E.1 Detailed training curves
For some of the runs the plot of the success could be misleading, in the sense that, despite not achieving the goal, the algorithms may still learn how to almost solve the task, or at least gained some knowledge about how to approach it. Figure E.10 shows for each of the runs, the distance to the goal, which corresponds to  $-r$  where  $r$  is the reward of the corresponding benchmark, as described in Section D.
For example, by looking at Figure E.10, we can see that despite the fact that the Human Preferences wasn't able to complete some of the tasks, such as Four Rooms, Pusher with walls or Maze, it still got some insight on how to approach it, getting much closer to the goal than the other methods that failed.
Benchmark	Oracle	Ours	GCSL	Human Preferences	DDL	PPO (sparse)	PPO (dense)	GCSL + Go- Explore
4 rooms	0.02 ± 0.01	0.02 ± 0.00	1.15 ± 0.67	0.48 ± 0.39	0.45 ± 0.28	1.45 ± 0.13	0.05 ± 0.02	0.13 ± 0.11
Maze	0.4 ± 0.3	0.8 ± 0.3	29.6 ± 2.2	18.5 ± 5.6	8.54 ± 10.6	30.4 ± 0.7	0.0 ± 0.2	1.1 ± 0.14
Pusher	0.06 ± 0.00	0.11 ± 0.04	0.85 ± 0.11	0.26 ± 0.03	0.69 ± 0.06	0.72 ± 0.06	0.27 ± 0.00	0.50 ± 0.29
Kitchen	1.06 ± 0.32	0.67 ± 0.21	11.72±0.11	3.43 ± 4.37	11.28±0.02	7.63 ± 4.96	2.84 ± 2.72	10.97 ± 0.13
Stacking	0.1 ± 0.2	0.0 ± 0.0	4.1 ± 2.3	6.5 ± 0.1	6.6 ± 0.1	6.7 ± 0.0	6.6 ± 0.0	3.76 ± 2.66
Bandu	1.00 ± 0.53	0.36 ± 0.73	12.87±0.01	12.54±0.01	12.63±0.21	12.75±0.01	12.75±0.01	10.7 ± 1.51
Benchmark	PEBBLE	BC (5 de-mos)	BC + Ours(5 demos)
4 rooms	0.59 ± 0.1	0.45 ± 0.46	0.04 ± 0.00
Maze	11.95 ±2.56	2.25 ± 1.51	0.87 ± 1.02
Pusher	0.185 ±0.04	0.25 ± 0.09	0.08 ± 0.01
Kitchen	10.48 ±0.01	11.38 ±0.00	0.87 ± 1.02
Stacking	4.24 ± 1.51	1.91 ± 1.02	0.01 ± 0.00
Bandu	13.5 ± 3.25	4.21 ± 5.47	1.87 ± 0.4
E.2 Analysis of PPO vs HuGE
Figure E.11: Average distance and standard deviation across 4 seeds for the different baselines we implemented to compare against HugRL. We see that HugRL consistently succeeds (in bold) to solve all benchmarks when most other baselines do not. The oracle would be the upper bound that we could hope to achieve, since in this case labels are provided all the time, and the goal selector is substituted by a precise distance function.
Figure E.12: Comparison of the PPO and Human Preferences baselines against HuGE when using a carefully tuned reward function and not on the pusher and block stacking environments.
As we have shown throughout this manuscript, HuGE is more robust to underlying noisy human preferences, which means it's robust to noisy underlying reward functions. In Figure E.12 we provide a comparison between the performances of PPO and HuGE when the reward is finetuned against when it's not. We observe that while HuGE succeeds in both cases, PPO only succeeds when the reward function is carefully tuned.
In the pusher experiment, the reward finetuning consisted in doing some hyperparameter tuning, basically, modifying the position of the keypoints and adding a bonus for solving each keypoint. In the block stacking, it consisted in adding an extra reward term seen in Figure E.13.
Finally, we wanted to understand why even when the reward function is finetuned and both HuGE and PPO obtain optimal policies, why is HuGE more sample efficient than PPO. To analyze this we ran a comparison on gradient variance using HUGE as compared to gradient variance with PPO, comparing