SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure E.13: Comparison of the original non-finetuned reward function (left) against the finetuned one (right) for the block stacking experiment.
the average pairwise cosine similarity across mini-batches to measure gradient variance (as suggested in In Figure E.14). We find that training the gradient similarity is significantly larger in HuGE than in PPO, potentially explaining improved sample complexity. Besides this, other components in HuGE may also lead to faster learning as generalization across states that come from self-supervised hindsight relabeling and hindsight relabeled policy learning is off-policy while PPO is on-policy. However, we donot want to claim that the synthetic variant of HuGE is fundamentally a better optimization algorithm than PPO. Given a carefully shaped reward function, PPO typically eventually works on most decision-making problems, however, HuGE is a method for goal-reaching problems, which is a subset of all possible decision-making problems. In these goal-reaching problems, we have shown that HuGE can typically outperform PPO even at convergence when the reward functions are noisy. However, PPO is a more general learning paradigm and more careful experimentation and ablations are needed before making broad-reaching claims.
PPO vs HuGE Analysis
Figure E.14: PPO vs HuGE similarity of gradients inside a minibatch during training. We observer HuGE's similarity is much larger, which implies the gradient variance is lower. This could mean that the optimization shape is smoother for HuGE and could be one of the reasons why HuGE outperforms PPO.
E.3 Analysis of HuGE vs Human Preferences when learning from noisy feedback
We ran an additional experiment shown in Figure E.15 to investigate further the benefits of HuGE when compared to baselines that run RL algorithms over the learned reward functions. In this case we analyze the performance of both HuGE and Human Preferences, when providing noisy synthetic feedback and we observe that while HuGE succeeds in learning an optimal policy, Human Preferences does not.
HuGE vs HP when learning from noisy feedback
Figure E.15: We observe that when adding synthetic noisy feedback HuGE converges to optimality while Human Preference fails
F Further Analysis and Ablations
F.1 Analysis on learning from comparisons
There is a tradeoff between the frequency of labelling and the speed for the policy to converge. In Figure F.16, (left) we observe that if we query more frequently, the policy needs more labels to succeed, however, we also observe (right) that when querying less frequently it takes more timesteps to succeed. Meaning that if we provide labels more frequently, the policy is going to converge faster to the optimal policy, but will come at the cost of needing more human annotations. On the other hand, if the human annotators provide labels less frequently, it will take longer for the policy to converge to the optimal policy. The query frequency will hence be an important parameter to look into depending on what we want to optimize for, number of human labels or timesteps to succeed. We believe that for simulation experiments, we might want to optimize for using less human labels since the policy rollouts can be done very fast. However, when working with learning on the real robot, we might prefer to have humans label more frequently and reduce the number of rollouts in the real world, which is usually the bottleneck.
Figure F.16: On the left/right we show the number of labels/timesteps needed to succeed when varying the query frequency. 1, 15, 100, and 500 are the number of episodes between each period of querying the human for annotations. We observe a clear tradeoff between needing fewer labels to succeed against needing more timesteps. Meaning that if we query more frequently, we will need fewer timesteps to succeed and vice versa. These experiments are done in the Four Rooms benchmark.
Querying a few samples per batch is enough. In Figure F.17, (left) we observe that providing more labels every time we query the human leads to needing more labels to have successful policies, as expected. In right, however, we observe that the number of timesteps needed to have a successful policy is very similar when querying for 5,20 or 100 annotations, however, when only querying for 1 the performance drops significantly. This means that 5 labels are already enough to learn how to expand the frontier, and querying more than 5 labels brings useless information.
Figure F.17: On the right we show the number of steps needed to succeed in the four rooms benchmark depending on the number of comparisons queried per batch. On the left, we show the number of labels needed to succeed, again depending on the query batch size. We observe that we can go as low as 5 queries per batch, and the performance is similar to 20 and 100. Showing that too many queries bring duplicated information to the goal selector training. Also, we see that providing 1 label is not enough, degrading the performance significantly. These experiments are done in the Four Rooms benchmark.
HuGE is robust to noisy labels. Increasing the noise in human labels leads to an increase in the number of timesteps needed to for the policy to learn to achieve the goal, as seen in Figure F.19. However, this does not decrease the accuracy of the resulting policy. Increased noise in the labels makes exploration become less directed and closer to the uniform frontier expansion methods.
Having a closer look in Figure F.18 at the shape of the reward function when large noise is added to the feedback. We observe that the goal selector becomes less accurate compared to the one with perfect feedback in Fig F.23. However, HuGE still successfully reaches the goal. As we can see, there are 3 modes in the final step (4th subfigure in F.18). This means, the goals will be sampled most frequently from these 3 modes, which will result in a less efficient frontier expansion, since only one of the three modes is the target goal. However, since we are learning a goal-conditioned policy through self-supervised learning this remains unaffected by this noise and will learn to go to the three modes, one of which is our target location. This would not be the case for methods that use this goal selector as a reward function to run model-free RL, due to its convergence to local maxima without reaching the target goal.
Figure F.18: Evolution of the learned goal selector when the distance for the synthetic human has a noise of 1. We observe that the goal selector is not accurate, however, our method still successfully reaches the goal, hence, it is robust to inaccurate goal selectors. This would not be the case for methods that use this goal selector as a reward function to run model-free RL, due to the noise on it and multiple local minimas and maximas.
Figure F.19: Show the effect of adding Gaussian noise in the labels provided by the human on the Four Rooms benchmark. We observe that our method is robust to different amounts of added noise, however, as noise increases, so will the timesteps needed to succeed. Noise is injected into the distance function used by the synthetic human to provide labels, which means that with higher noise the probability of the comparison being wrong will increase. For context, the distance between the initial state and the goal is around 1.6.
HuGE is robust to underlying simple reward functions. In Figure F.20 we show the performance of our method in the Four Rooms environment when dealing with a simplified version of feedback. In particular, we only return feedback if the given queried states have a distance difference of at least  $d$  with respect to the goal. For context, in this environment 0.5 is approximately the distance between the center of two consecutive rooms, so using  $d \geq 0.5$  is roughly similar to using the room number as a reward function. Therefore, in this experiment, we can see that, even with very simple reward functions, we can still get some insight on how to solve the task, though at the expense of clearly slower convergence. In particular, we can see how coarser reward functions lead to worse performances. This also helps us understand what happens in scenarios in which it is hard for humans to compare states that are similarly good for the purpose of achieving the required goal.
Figure F.20: Comparison on the effect of simplified reward functions providing the synthetic human annotations.
HuGE can learn when no labels are provided. This property of HuGE is because of the self-supervised learning used to train the policy but also a result of using a parametric goal selector as compared to directly selecting goals of interest as done in [27], which will not have this advantage. From Figure F.23 we observe that a parametric goal selector has the capacity to generalize while, by definition a non-parametric goal selection [27] will not. Thereafter, using a parametric reward model that has non-degenerate extrapolation can lead to significantly more frontier expansion. In Figure F.21 we show how our method succeeds in reaching the final goal room even if the goal selector has stopped training when the agent enters any of the previous rooms. However, this comes at a cost in much slower convergence.
Figure F.21: Effect of freezing the goal selector at different points in the learning of the policy on how long it takes to learn a successful policy on the Four Rooms benchmark. We see that an earlier stop in the training leads to an increase in the timesteps needed to succeed. However, even if we stop in the second room, our method is still very good at quickly finding a successful policy, which shows how robust it is against incomplete goal selectors. This would not be the case for methods that run RL on the learned reward functions (as DDL, and RL from Human Preferences). The policy still succeeds thanks to the added random exploration, the self-supervised nature of GCSL, and a small probability of sampling the final goal.
F.2 Goal selector Analysis
Learning a goal selector is more feedback efficient than directly using the human feedback. In figure F.22 we show a comparison of the number of labels needed to succeed when using a parametric goal selector (Ours) against directly using the goal selected by the human (DDL). We show the comparison between different frequencies of human querying. 15, 100, 500 episodes are the number of episodes we wait before querying the human annotator for more labels. We observe that when learning a goal selector, we obtain a reduction in the number of labels needed of  $40\%$  when querying every 15 or 100 episodes and a reduction of  $59\%$  when querying every 500 episodes. Furthermore, if we don't learn this parametric model, with low frequencies we might not learn a successful policy, as happens for the non-parametric version at 100, 500 episodes of frequency. When using the non-parametric goal selector (DDL) not all trials succeed, for querying every 100 episodes, 2 seeds out of 4 fail and for 500 episodes between querying 3 out of the 4 fail, which is another reason why parametric goal selectors are better.
Figure F.22: Comparison of the number of labels needed to succeed when using a parametric goal selector (Ours) against directly using the goal selected by the human (DDL).
Figure F.23: Progress of goal selector learning in the four rooms environment as learning progresses it gets closer to the target (oracle on the right). The purple area represents the visited states by the agent at that point. We observe that the goal selector provides extrapolation which will help the training with fewer annotations.
In figure F.23, we show the goal selector will have non-trivial generalization, allowing us to continue expanding the frontier even when no human is present.
Furthermore, in F.24 we explore how accurate the goal selector is, depending on the number of queries it has been trained with. In particular, we tested it in the Four Rooms environment by training the goal selector using pairs of states sampled uniformly. During evaluation, given two states which are less than  $d$  units apart, we compute the accuracy for which the model is able to pick the closest state to the goal. This allows us to see that the model is able to, given two states, determine which one is the closest to the goal, even when the given states are very close together and even when trained with just a handful of queries. For context, bear in mind that the distance from the initial state to the goal is 1.6 units.
Qualitative analysis of the generalization of the goal selector. In this qualitative analysis, we show visualizations of the learned goal selector as different rooms are discovered during the learning
Figure F.24: Accuracy of the goal selector depending on the number of queries and dependent on the distance  $d$  between the states compared in the labels.
process in the four-rooms domain Fig F.25. The goal selector model shows nontrivial extrapolation and can potentially provide guidance even beyond the set of states it is trained on.
Figure F.25: The goal selector learns and converges to a result close to the oracle (rightmost) as epochs increase (left to right). We observe how this goal selector gets updated iteratively as the frontier expands. Colder colors mean a lower reward for that state, whereas warmer colors mean a higher reward for that state, in this case, this is equivalent to the distance to the goal.
F.3 Compatibility of HuGE with Learning from Trajectory Demonstrations
As we mention in Section 4.3, HuGE is compatible with learning from trajectory demonstrations. In Figure F.26, we show how HuGE can improve the performance of simple imitation learning starting from different amounts of demonstrations. Given the number of demonstrations, imitation learning fails on less than 10 demonstrations, and with HuGE we can improve the policy to succeed in all cases.
Figure F.26: The figure depicts the distance to the goal in the Four Rooms environment when using a policy pre-trained via Behaviour Cloning with 0, 2, 5, and 10 demonstrations, respectively. We see that using BC on a small number of demonstrations helps to boost the performance of our method. Also, notice that BC wouldn't achieve success (distance  $< 0.05$ ) in any of the cases due to compounding errors which leads to covariant shift. However, HugRL solves these compounding errors within a small number of steps.
G.1 Algorithm: stopping before exploration
One important detail of the algorithm is knowing when to stop rolling out the policy and start exploration. The idea is that since the policy is learned in a self-supervised manner, we do not want to have too many redundant steps in a single state. If this happens then because of an imbalance of redundant actions going over this single state, in the training set, will bias the policy to get stuck and not transition to other areas. For it not to happen, we need to stop rolling out the policy at the right moment and launch the next step in the algorithm, the exploration. We propose three different cases where we should stop the rollout, first, once thebreadcrumb state  $g_{b}$  is reached, second, once the time-horizon for the episode is reached, third, once the policy is not making any progress and gets stuck in some place. Detecting this can be trivially done looking at the euclidean distance between last states when the state space is in point space, however, it can also be done in image space and we explain it further in Appendix A.1.
G.2 Hardware
For training the models and running the experiments, we had access to several workstations with one GeForce RTX 2080 Ti or one GeForce RTX 3090. It took on average 8 hours on these machines to run 4 seeds for each one of the baselines and our method. We account the total amount of compute hours would be around 1440 hours for the whole project, taking into account, experimentation and testing the algorithms.
G.3 Networks with Fourier Features
Seeing the complexity of our benchmarks, where we can have non-smooth reward landscapes for the goal selector. For example, in the four rooms environment, between one side and the other of the right rooms, the reward changes significantly and abruptly. Adding Fourier Features has been shown to work well for fitting these landscapes [56]. For this reason, we used them in some of our experiments, as detailed in Section G. More precisely, when used, we added an additional layer with Fourier features of size 40 times the input dimension.
G.4 Training details
The details of the parameters with which the results have been obtained will be disclosed in this section. In particular, Table G.4 depicts the parameters used for the different benchmarks, while Table G.3 contains the hyperparameter configuration used for the different algorithms.
Parameter	Value
Shared (to those that apply)
Optimize	Adam
Discount factor (γ)	0.99
Reward model architecture	MLP(256, 256)
Use Fourier in the reward model	True
Buffer size reward model	1000
Steps per reward model update	1000
GCSL, Oracle and Ours
Learning rate	5 · 10-4
Batch size	100
Policy architecture	MLP (400, 600, 600, 300)
Steps per policy update	5000
Use Fourier in the policy model	True
Buffer size rollout	1000
Max gradient norm	5
Last trajectories to be labeled	1000
Human preferences Same parameters as [52] plus/except
Learning rate	5 · 10-4
Batch size	100
Policy architecture	MLP (256, 64)
Steps per policy update	5000
Use Fourier in the policy model	False
Buffer size rollout	1000
Max gradient norm	5
Last trajectories to be labeled	1000
DDL
Learning rate	5 · 10-4
Batch size	256
Buffer Size	2 · 104
Policy architecture	MLP (256, 256)
Steps per update	1000
PPO Same parameters as [52] plus
Buffer size	8192
Policy architecture	MLP (400, 600, 600, 300)
Table G.3: Hyperparameters setting for the algorithms
Environment	Four rooms	Maze	Pushing around Obstacles	Kitchen	Block Stacking	Bandu
Steps per trajectory	50	250	100	100	10	12
Label from last k steps	10	50	10	20	10	12
Table G.4: Benchmark-related parameters