Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulation
Authors: Todor Davchev
Oleg Olegovich Sushkov
Jean-Baptiste Regli
Stefan Schaal
Yusuf Aytar
Markus Wulfmeier
Jon Scholz
Venue: ICLR 2022 Poster (2022)
Keywords: goal-conditioned reinforcement learning
learning from demonstrations
long-horizon dexterous manipulation
bi-manual manipulation

Abstract: Complex sequential tasks in continuous-control settings often require agents to successfully traverse a set of ``narrow passages'' in their state space. Solving such tasks with a sparse reward in a sample-efficient manner poses a challenge to modern reinforcement learning (RL) due to the associated long-horizon nature of the problem and the lack of sufficient positive signal during learning. 
Various tools have been applied to address this challenge. When available, large sets of demonstrations can guide agent exploration. Hindsight relabelling on the other hand does not require additional sources of information. However, existing strategies explore based on task-agnostic goal distributions, which can render the solution of long-horizon tasks impractical. In this work, we extend hindsight relabelling mechanisms to guide exploration along task-specific distributions implied by a small set of successful demonstrations. We evaluate the approach on four complex, single and dual arm, robotics manipulation tasks against strong suitable baselines. The method requires far fewer demonstrations to solve all tasks and achieves a significantly higher overall performance as task complexity increases. Finally, we investigate the robustness of the proposed solution with respect to the quality of input representations and the number of demonstrations.

Published as a conference paper at ICLR 2022
WISH YOU WERE HERE: HINDSIGHT GOAL SELECTION FOR LONG-HORIZON DEXTEROUS MANIPULATION
Todor Davchev Oleg Sushkov Markus Wulfmeier' Jon Scholz'
Beannor Jean-Baptiste Regli', Stefan Schaal', Yusuf Aytar',
'DeePMind, University of Edinburgh, 'Intiinsic LLC
ABSTRACT
Complex sequential tasks in continuous-control settings often require agents to successfully traverse set of "narrow passages" their state space. Solving such tasks with a sparse reward in sample-efficient manner poses challenge to modern reinforcement learning (RL) due to the associated long-horizon nature of the problem and the lack of sufficient positive signal during learning. Various tools have been applied to address this challenge. When available, large sets demonstrations can guide agent exploration. Hindsight relabelling on the other hand does not require additional sources of information. However, existing strategies explore based on task-agnostic goal distributions, which can render the solution of Long-horizon tasks impractical. In this work, we extend hindsight relabelling mechanisms guide exploration along task-specific distributions implied by a small set of successful demonstrations. We evaluate the approach on four complex, single and dual arm, robotics manipulation tasks against strong suitable baselines. The method requires far fewer demonstrations to solve all tasks and achieves a significantly higher overall performance as task complexity increases. Finally, we investigate the robustness of the proposed solution with respect to the quality of input representations and the number of demonstrations.
INTRODUCTION
Recent advances in model-free reinforcement learning (RL) have enabled successful applications in
variety of practical, real-world tasks (Gu al. 2017 Levine al. 2018 Riedmiller et al.] 2018
OpenAl et al., 2019 Sharma et al. 2020: Gupta et al.
2021). Given the complexity of these tasks,
additional information such as demonstrations often plays an important role (Vecerik et al.) 2019
Zuo et al. 2020 Davchev et al., 2020 Sutanto et al.
2020 Luo et al.) 2021 These methods are particularly useful robotics as they enable efficient learning with only sparse rewards, which
easier to define. However, scaling such solutions to more complex long-horizon sequential settings with limited number of demonstrations remains an open challenge (Gupta al. 2020).
Solving long-horizon dexterous manipulation tasks is an im- portant problem as it enables a wide range of useful robotic applications ranging from collaborative settings, through contact-rich dual arm manip- ulations, often seen in manufacturing. Consider the bi-manual 3.5mm jack cable insertion problem as illustrated in Figure [ Solving such tasks with vanilla demo-driven RL often fails due to the difficulty of assigning credit over long horizons in the presence of noisy exploration and sparse rewards.
tasks, common in
Self-supervision mechanisms like Hindsight Experience Replay (HER) (Andrychowicz al., 2017) offer an alternative to ease the sparse reward problem by providing a stronger learning sig- nal through additional goal-reaching tasks which generated from the agent's 's trajectories. While the more frequent reward can be beneficial, the agent's trajectory
Figure 1: HinDRL solving 3.5mm jack cable insertion task.
Work done during intersship DeepMind; Denotes equal contribution.

---

Published as a conference paper at ICLR 2022
distribution is often considerably more complex and non-stationary in comparison to the target-task distribution. When the final task is very complex and the main task reward is not perceived, this can lead to learning capable, goal-reaching policy for states close to the agent's initial states while unable to complete the actual task.
Our contribution is based the insight that demonstration states can be viewed samples of the target task distribution, and can therefore be used to constrain the self-supervision process to task-relevant goals. We introduce a framework for task-constrained goal-conditioned RL that flexibly combines demonstrations with hindsight relabelling. Unlike HER, which learns a general goal- conditioned agent, we train coal-conditiones agent specialized at achieving goals which directly lead to the task solution. The approach further allows to smoothly vary the task relevance of the relabelling process. Unlike conventional goal-conditioned RI., we enable agents to solve tasks through utilising abstract goal formulations, such as inserting 3.5mm jack, common in the context of complex sequential tasks. We achieve this through using continuously improving target goal distribution for the online goal selection stage. We refer to our method as Hindsight Goal Selection for Bemo-Driven RL or HinDRL for short. We demonstrate that the proposed solution can solve tasks where both demonstration-die -riven RL and its self- self-supervised version with HER struggle. Specifically, we show that HinDRL can dramatically reduce the number of required demonstration by an order of magnitude on most considered tasks.
RELATED LITERATURE
Using demonstrations in RL is a popular tool for robot learning, Classic approaches. such as (Atkeson Schaal 1997 Peters & Schaal 2008: Tamosiunaite et al.) 2011), use expert demonstrations to [Chemali extract good initial policies before fine-tuning with RL. Kim et al. ( & Lazaric (2015) use demonstrations to learn an imitation loss function to bootstrap learning. However, those solutions not applied to complex sequential tasks. Follow up work that use deep neural networks enable the application of demo-driven RL to more complex tasks such as Atari(Aytar et al., 2018: Pohlen et al.,
2018; Hester et al. 2018), or robotics (Vecerik et al. 2017 2019 Paine et al. 201 We build upon Deeerk et al. 2019), which we explain in more details in Section
The Atari literature often requires large number of environment steps (Salimans & Chen 2018 Bacon et al. 2017). This can result in hardware wear and tear if applied on a physical robotics system. Alternatively, number of robotics solutions assume the existence of manually defined structure, e.g. through specifying an explicit set of primitive skills (Stulp et al.) 2012: Xie et al., 2022)), or a manually defined curricula (Davchev et al. 2020 Luo et al. 2021). However, hand-crafung structure can be sub-optimal in practice as it varies across tasks. In contrast, restricting the search space by greedily solving for specific goals (Foster & Dayan 2002 Schaul et al. 2015) can be combined with implicitly defined curricula through retroactive goal selection (Kaelbling) 1993 Andrychowicz et al. 2017). Hindsight goal assignment has been successfully applied to multi-task RL (Li et al.] 2020 Eysenbach et al. 2020), reset-free RL (Sharma et al. (202T) but also for batch RL (Kalashnikov et al. 2021] Chebotar el al. (02T) and hierarchical RL (Wulfmeier et al. 2020 2021). However, these approaches explore based on task-agnostic goal distributions and compensate with additional policy structure such as allowing for longer training to concurrently learn a hicrarchy, or using large amounts of offline data. In this work, we extend hindsight relabelling mechanisms to guide exploration along task lask-specific distributions implied from a limited number of demonstrations and study its performance against strong task-agnostic goal distributions.
Combining hindsight relabelling with demonstrations is not a novel concept. Ding et al. (0019) relabels demonstrations in hindsight improve generative adversarial imitation learning. The proposed solution is used to learn goal-conditioned reward function that can be applied on demo- driven RL for hindsight relabelling. Guupt al.) (2020) propose a hicrarchical data relabelling mechanism that uses demonstrations to extract coal-condditioed hierarchical imitation amitation-based policies and apply them to multi task RI. Nair al. (0088)) fuses demo-driven RL similar to us but uses HER's 's final-goal sampling mechanism and combines it with structured resets to specific demo states to overcome the difficulties of exploration for long-horizon tasks, which would be difficult to achieve in a continuous robotics task in practice. Zuo et al. (2020) uses demonstrations to bootstrap TD3 and combines it with HER. However, in all those works, hindsight goals were always chosen directly from the agent's 's own trajectory. Instead, consistently select goals directly from task distribution implied from a collection of successful demonstrations. Hindsight goal selection has previously been generalized via adopting representation learning. Florensa et al. (209)) applied HER
2

---

Published as a conference paper at ICLR 2022
Online Goal
Selection Stage
Agent Rollout
T
Teak Demomatration
Hindsight Goal
Selection Stage
Patk-specffc goal
Ageni rol
Select K goals hindsight for each timestep
Replay Buffer
Cand Sand
leget
Candibee Gool
Suncde zghom veperer
20-ategy
Sivtegg
The HinDRL pipeline performing 3.5mm jack cable insertion. The input to the policy are encoded target goal and the current robot state. At train time, the produced episodes relabelled in hindsight using choice of sampling strategies. We consider two main strategies for sampling goals in hindsight. Strategy a) shows the standard hindsight goal selection strategies that select goals from states within the trajectory being relahelled; and strategy b) shows a mechanism that facuses on sampling goals directly from some collection of successful trajectories. The resulted relabelled data is fed into the replay buffer.
Figure 2:
to mixels-to-torquee reaching task where sample efficiency was not a direct constraint. Nair et al. (2018b) used a B- balaanece variational auto encoder (B-VAD) to enable learning from visual inputs with special reward. While ß-VAE is broadly a powerful tool for unsupervised representation learning, it does not take into account the temporal nature of sequential robotics tasks. This makes them sub-optimal in the context of our work (Chen et al.) (021). In contrast, contrastive-leane leaaning based techniques such as temporal cycle consistency (TCC) (Dwibedi et al., 2019) can provide more informative representations as previously discussed in concurrent work (Zakka et al.) 2022). In this work we combine HinDRL., framework for goal-conditioned demo-driven RL with both learnt and engineered representations and provide a comprehensive sensitivity analysis the quality of the encoder and the dependency against the number of demonstrations used.
PRELIMINARIES
Consider a knite-horizon discounted Markov decision process (MDP), - (S.A,,,,,Y)) with transition probability P: x x [0, 1]. Let the current state and goal E Rdh be elements in the state space S, and a E C Rh denote the desired robot action. We define a sparse environmental reward r(s) that assigns 0/1 reward only the final state of an episode. Let vr()) be an encoding function that embeds a given state and goal to latent state z (s)) and latent goal g ((g)) Let { = (X......, .r) be a trajectory with a discrete horizon T, a discount function 7(.), a state- action tuple x, = (5, ar.g) .)) and a trajectory return R(5) Σ (r(s;) In this setting, transition (sr, 51 ..+ 0) can be 'relabelled" as (Nr, (5,,4, RJ s,-1, 1) and both the original and relabelled transitions can be used for training. separate, goal conditioned reward r(z)+1,2g) 1 [z 2g] uses the latent state and goal and assigns reward during relabelling. Finally, we also assume access D demonstration trajectories 9 { that reach the goals { retrieved from the final state of the demonstration. Then, goal-conditioned deterministic control policy parameterised by 0, Ng 2g)) selects an action given a latent state Z and goal zz. In this context, define an optimal policy π² argmaxzea ((T), where J(T)) Exp--psa)ss [R()]]-
DPGID DPGrD(Vecerik et al. 2019) is a powerful algorithm that bootstraps learning through a BC loss applied directly to the actor. This loss is typically defined as the L2 distance between the predicted by the policy actions and the true actions of a demonstration and is decayed proportionally to the number of environmental steps. Gradual conversion from using BC loss to using the actor-critic loss is typically done using Q filtering or gradually decaying the BC loss. We employ the latter. The algorithm uses an 1.2 regularised actor-critic loss with distributional critic, introduced in (Bellemare)
3

---

Published as a conference paper at ICLR 2022
et al. 2017). Using BC loss with DDPG was recently applied goal-conditioned setting (Ding et al. 2019) but coal-conditioned policy learning was never used with demo- driven distributional RL.
4
METHODOL.OGY
This section describes our method, HinDRL. First, we extend DPGfD to its goal-conditioned inter- pretation and introduce a mechanism that deals with intermediate goals. Next, introduce two demonstration-driven goal sampling strategies and discuss how to use them for goal-conditioned policy learning. Finally, discuss how extract temporally consistent representations and define distance-based goal-conditioned reward function. We summarise our framework in Figure
4.1 GOAL-CONDITIONEL DEMO-DRIVER REINFOR LEARNING
Similar to DPGID we combine reinforcement and imitation learning. We utilise a data set of demonstrations, 9 { (so, 55.5.....)) .))]] 12
which are used seed a replay buffer for RL, and for supervised training of the policy. We used both 6D Cartesian velocity and binary open/close actions, which we modelled using a standard regression for the predicted velocity and a classification for the binary prediction. We use encoded state and encoded goal zg as input to the policy network,
LBC = [x(2, - Σ (z)
The BC loss is applied only to transitions from successful trajectories.
For RL we utilized the standard deterministic volicy-gradient (DPG, Silver et al. (2014)) loss for the deterministic actions (velocities), and stochastic value- gruuieg (SVG, Heess et al. (2013) for the stochastic actions (gripper positions). We used the Huubel-Soffmaa trick (Jang et al. 2016 Maddison et al., (0166) to reparameterize the binary actions for the SVG loss. Following Vecerik et al. (2019) applied both the BC and RL losses train the policy, and annealed the BC loss throughout training to allow the agent to outperform the expert.
We distributional critic (Bellemare al. 2017), modeling the Q-function with categorical distribution between O and (with 60 evenly spaced bins). The learning loss is the KI--divegeence between our current estimate and projection of the 1-step return on the current bins. Therefore, the loss becomes,
LTD (L( (D y(t) Qtarger (2,, (2, 2g),22)))
(2)
where 4 is an operator that projects a distribution on the set of bins.
4.2 GOAL SELECTION FROM DEMONSTRATIONS
Typically, goal-conditioned policy learning has two separate stages for goal selection (Andrychowicz et al. 2017): ) an online stage where a policy is conditioned
specific goal during policy rollout, and a hindsight goal selection stage where the produced trajectory is being relabelled in hindsight (See Figure 2). While Ahhle/AAndrychowizz et al. (2017) only consider the original goal in the online stage, we describe below how this perspective can be extended to increase robustness.
Online goal selection A target goal of sequential task, such as inserting a plug into socket, is in principle more abstract than the physical skill of reaching a specific configuration. That is, being an distance away from target configuration might still result in a failed insertion. We mitigate this issue by maintaining a separate goal database, G.b. comprised of final states over successful executions. We bootstrap this database with the final states from 9 but also continuously grow Cab as learning progresses. Conceptually, the more examples of final goal states we collect, the better understanding of the target goal of the current task we will have. In this context, for every episode we sample a new &g from a distribution over the set of all possible goals in Gab- We denote this as RG = {zg E Gab p(zg) > 0} and we refer to this process as online goal-conditioning always store the resulted 2 Ey-conditioned trajectory, Ç, in the replay buffer. As in classic RL, the last step of [ is rewarded using the environment reward r(s;).
this work,
Hindsight goal selection In contrast, hindsight goal selection is the process of retroactively sampling candidate goal states from some goal distribution p(zg) after an episode rollout. Here, Zg are not conceptually related to the abstract formulation of target task &g is. Instead, they represent different stages from the behaviour that results in solving the abstract task. The way this stage works is sample goals for each time step of the trajectory 5 using some candidate goal distribution
4

---

Published as a conference paper at ICLR 2022
Algorithm HinDRL
1: Initial 0, 9, 3 [T RB
2: Policy Tg, encoder Ve. p(so)
3: if trainable, Iearn encoder offline here
Algorithm 2 Generate samples
4: // encode demos, add to replay buffer
1: Input: trajectory, Ç. task demos Ø
5: RB t -RBU@
2: 0 relabelled transitions
6: // relabel trajectories and add to RB
3: for sampler E sampling strategies list do
7: RB + -RBU {generate, [generate_samples(ff
1-11-11
4: data <- 0 support data
8: // add final state from demos to goal db
5: if sampler.name is Rollout-conditione then
9: Cab Uf 1
6: data HER
10: for iter iterSmax do
7: else if sampler.name is Task-condittoned then
11:
50 < p(so) // initial state
8: data 2 // Task
12:
$ G~Ga // target goal
9: end if
13:
// trajectory & (2..0,2211.1
10:
I/ compose support for goal distribution
$ 5+ -o (. v())
11:
RG = (2g data p(zg) 0}
15:
if success then
12:
sample new goals, T - -51
16:
I/ if successful, store last state in goal db
13:
fāg p(2g) 1r-1
17:
Gab- + GabUEr // er 5 end if
14:
I/ store relabelled transitions using Eg
18:
15:
end 2(t
X
19:
RB + RBUŠ add rollout to RB
16:
20:
#/ relabel [ and add to RB samples((,. 9)
17:
Returns: X
21:
RB <- RBU generate_ generate_samplees[[]
22:
Il opttionally, retrain here
23:
0 train_x(R)
24:
end for
Figure 3: Algorithm for Hindsight goal selection for Demo-driven Reinforcement learning (HinDRL) p(zg) implied from some given behaviour. Typically we sample candidate goals from the future states in rollout trajectories. Then, re-evaluate the given transition with the newly sampled goal and assign a new reward. This is achieved with the help of a goal conditioned reward r(z,zz) and not the environment reward from the previous paragraph. Thanks the coal-conditioned formulation, fitting a Q function from both of these sources of reward is well-posed. In this section we propose a way for composing a hindsight goal distribution p(zg), that targeted on the specific task at hand.
Candidate goals can be acquired through self-suppervssio (as in HER), where goal distribution is comprised of states from the agent's trajectory Ç. The agent retrospectively samples feasible goals to 'explain' its own behaviour under the current goal-conditioned policy π. However, similar to concurrent work (Gupta al., 2020 Pertsch et al., 2020) we observed that choosing goals from p(zg15) does not work well for long-horizon tasks.
However, hindsight goal selection does not need be constrained to the same trajectory. Goals can be sampled directly from a task distribution, implied from a set of successful trajectories, such as demonstrations. Therefore, selecting task-specific goals can come from p(zg) with support RG = {zg E 2 p(zg) > 0}. We used uniform distribution over the demonstration states. However, similar to HER, this can benefit from alternative formulations too. In practice, the support of the task-distribution. Rg. can be enriched over time with additional positive task completions from the training process too. However, we do not do it in this work.
Using the agent's trajectory is useful when modelling the agent's behaviour. However, it can struggle to scale beyond normal-horizon tasks, Gupta et al. (2200). On the other hand, using demo- driven samplers can speed up training by directly modelling the task distribution. However, it requires certain level of confidence over the quality of the task representation. These two support distributions can complement each other and do not need to disjoint. Joining both types of trajectories can be particularly useful in cases where using just demonstration states can fail to make the sparse reward problem easier, e.g. when the representations fail capture the notion of progress. We provide an ablation against loint-condition samplers in Appendix A.3 including using the union, ÇU2, and intersection $n of the two sets of goals. Our results indicate that doing so can improve the performance of HinDRL with representations that do not preserve the notion of progress and speed up training for complex tasks (see Appendix A.4 and A.5)
4.3 TIME- COM-SSSNSITTNT REPRESENTATIONS
Synthesising goal-conditioned policies from raw state observations can be impractical in high dimensional spaces. Using raw states can not only deteriorate the speed of learning (Hessel et al.
5

---

Published as a conference paper at ICLR 2022
2018), but it also makes it difficult to define comprehensive gondicond reward for relabelling. Instead, we encode our state observations into a lower dimensional latent space R"s using an encoder y obtain latent state ((ss) and latent goal Zz V(g). Next, propose both expert-engineered and learnt time-consistent representations.
Engineered representations In the presence of an expert, low dimensional representations be programmed to include features that bear a notion of progress, e.g. through measuring the distance from target goal, well as the notion of contact, such as whether insertion was successful. These notions vary across each task and we provide additional details for each of the engineered encoders in Appendix(A.9
Learnt representations We propose as an alternative to the engineered encoding, self-supervised learnt representation that can be directly used with the provided demonstrations and refined over time, We ensure consistent and aligned notion of episode progress in our learnt representation without the need of labels by employing a differentiable cycle-consicn consisteen loss (TCC) et al. (019). This allows to learn representation by finding correspondences across time between trajectories of differing length, effectively preserving the notion progress.
Training of TCC involves encoding an observation from trajectory U to a latent 4 and checking for cycle-consistency with another trajectory that is not necessarily of the same length as U. Cycle consistency is defined via neiirbot - if the nearest embedding in V to zl is zy. i.c. nearest V) = = then z! and z are cycle-consistent and only if nearest U) this requires differentiable version of yycle-consistency measure, such as regression-based or classification-uue based one. In this work, we employ the latter.
Learning
We additionally evaluate a non-temporal embedding using B-VAE, similar to (aair al. 2018b) (see Appendix A.5).
4.4 DISTANCE-BARE BASED REWARD SPECIFICATION
Considering local smoothness in the learnt goal representation space,
distance- based goal-conditioned reward function. This allows us to increase the number of positive rewards and therefore ease the hard exploration problem. We particularly use,
(3) for some threshold E. where latent state and latent goal Zg are considered similar if the /2 distance between them is below g. We provide more details on how we obtain the threshold in Appendix A.8 Figure 4 illustrates the t-SNE visualisation temporally aligned represen- tation space learnt with TCC. The notion of progress in TCC is consistent across all 100 plot- ted trajectories. We provide additional details in Appendix A.5) TCC is inherently capable of temporally aligning different in length and mo- tion trajectories that pass through similar stages. This makes it particularly suitable to use with distance-batt based metrics like Eq.
r(z,zg) = 1 1|||- <g||< E],
Figure 4: t-SNE of latent representations obtained with TCC (Dwibedi et al. 2019) against novel successful trajectories. Temporal color-coding, purple is start and yellow is goal.
The final proposed algorithm, together with the relabelling technique, is detailed in Algorithm
EXPERIMENTS
We are interested in answering the following questions: i) is task-conditioned hindsight goal selection useful for long-horizon sequential tasks; ii) does it improve over DPGfD and HER's sample efficiency; iii) is HinDRL demonstration efficient solution and how does it perform in a few-shot setting; and iv) how useful are learnt time-consistent representations.
5.1
TASKS
We answer these questions based on four tasks with increasing complexity, implemented with MuJoCo (ooorov et al. 2012). We consider attempt successful if the agent gets under threshold distance

---

Published as a conference paper at ICLR 2022
(a)
(b)
(d)
(c)
Figure 5: Tasks description a) is the parameterised reach task. Visiting the small yellow goals must happen before reaching the middle goal. b) is dual arm reaching. we combine this with c) lifting, d) aligning in both position and orientation and e) is the dual arm insertion.
from its target. The metries include the running accuracy of completing the tasks and the overall accumulated reward. All policies are executed in 10Hz with 12 random seeds. At test time we sample a random goal from all successful goal states and for the agent. We use 2 = (s, e(s),, and Zg = e(g). for all encoders e(-) and always train with a sparse binary environment reward. All methods use the same representation and rewards as HinDRL. The number K of goal samples is dependent on the complexity of the task with further details in the next paragraphs and in Appendix [A.10] We evaluate HinDRL on two different robotic set ups and a total of four different tasks. Each environment assumes a different robot which has different structure, action space and robot dynamics.
Parameterised Reach: Here, we use single 7DoF Sawyer robot to perform parameterised reaching. Parameterised reaching the task of visiting two randomly located in free space way-points before reaching for its target goal, Figure 5p. The continuous action space is comprised of the Cartesian 6 DoF pose of the robot and Dof representing the gripper open/close motion. The goal space is equivalent to the state space. We use total of two hindsight goal samples. Zg. to relabel single step.
Bring Near: This is dual-arm task situated in a cell with two Panda Franka Emika arms and hanging audio cables, Figure 5p-c. The agent has to use both robot arms reach for two cables and bring both tips within an & distance from each other. The action space is 7DoF for each arm, representing pose and grasp. The state and goal spaces are twice as large due to the dual nature of the task. Here we use a total of four hindsight goal samples, Zg. to relabel a single time step.
This is dual-arm task similar to above where the agent needs to reach and grasp the two cables. However, here it is also required to align both tips, that is position and reorient them within certain threshold. This task is significantly more complicated than the one above as it requires an additional planning component. The two cables have rigid components at their tips, 3.5mm jack and socket respectively. In order be manipulated to a specific orientation those cables have to be grasped at the rigid parts as grasping the flexible part of the cable prevents from aligning the two tips (Figure (ld). We used total of six goal samples, Zg. to relabel a single time step.
Bring Near and Orient:
This is the most complex task we evaluate against. It has the same dual-an arm set up as above but requires the complete insertion of the 3.5mm jack as well the reaching, grasping and alignment stages from the previous two bimanual tasks (Figure This task requires not only careful planning but also very precise manipulation in order to successfully complete insertion. We used a total of 12 hindsight goal samples, Zg. to relabel single time step for this task.
Bimanual Insertion:
5.2 HINDSIGHT GOAL SELECTION FOR DEMO-DRIVEN RL: HINDRL
In this section we evaluate the utility of using different strategies for goal sampling in hindsight in the context of demo-driven RL against a hand-engineered encoder. We compare the performance of HinDRL against several strategies for hindsight goal sampling, as well as two non-goal-condition baselines - vanilla DPGfD and Behaviour Cloning (BC).
In addition, we combine DPGID with hindsight relabelling using only the final reached state by the agent, similar to (Nair et al. (018a) although without a curriculum over start configurations and using our goal-conditioned distributional agent as opposed to DDPG. We refer to this as HER (final). We consider using samples from the future trajectory to relabel both the demonstrations and the agent rollouts, similar to (Ding et al., 2019), although we do not consider using a stronger BC component and use our goal-conditioned distributional agent as opposed to DDPG. we refer to this baseline as HER (future). We report findings in Table [ In all cases HinDRL performs best. Notably, assuming sufficient number of demonstrations, the DPGfD agent is able to solve both the parameterised reach and the bring near tasks very well, but struggles on the full cable-insertion task. Both HER-basee samplers struggle this task as well, but HER (future) performs better. HER is particularly good
7

---

Published as a conference paper at ICLR 2022
for shloter-horizzn tasks like Bring Near. We do not expect to have any significant performance benefits against HER on such tasks. In summary, the ability of HinDRL to specialize at achieving goals on track to the specific task solution through the proposed task-constrained self-supervision results in a superior performance across all considered long-horizon tasks.
Environment
BC
DPGID
HER (final)
HER (future)
HinDRL (Our)
Mean (Std)
Mean (Std)
Mean (Std)
Mean (Std)
Mean (Std)
Parameterised Reach (2wp)
82.05% (0.0)
88.62% (7.7)
84.92% (28.6)
86.92 (5.2)
92.61 % (4.7)
Bring Near
88. 88.90% (0.0)
99.74% (1.0)
32.03% (41.8)
97.75% (2.9)
98.79% (4.1)
Bring Near + Orient
57 97.89% (0.0)
83.52% (18.0)
8.33% (27.6)
23.28% (33.5)
90.77% (10.0)
Bimamual Insertion
16.06% (0.0)
4.28% (4.4)
3.38% (3.3)
18.39% (12.3)
78.92% (10.3)
Average
61.23% (0.0)
69.04% (7.8)
32.17% (25.32)
56.59% (12.9)
90.27% (7.3)
Table 1: Records performance in terms of accuracy.
5.3
PERFORMANCE ON FEW-SHOT TASKS
Using demonstrations can significantly speed up the training time while still successfully outper- forming the expert eventually, (Vecerik et al.) 2019). However, the final achieved performance in sparse reward settings depends on the com- plexity of the task and the number of provided demonstrations. In this section we show how using task-conditioned hindsight goal selection can significantly reduce the demonstration re- quirements and relax the dependency of the RL agent to the performance of the BC loss. We evaluate the ability of HinDRL to work in a few-shot setting. We train with 1, 5, 10 and 30 demonstrations and report our findings against the Bring Near Orient task in Figure 6 The figure shows the overall achieved accuracy across different number of demonstrations. There, all solutions fail to solve the task with single demonstration. This shows that all considered DPGfD-based based solutions were dependent on the ability of the BC component to achieve more than 0% accuracy. HinDRL. is able to outperform the BC policy rather quickly on the 10-shot tasks while both DPGfD and HER took significantly longer. Furthermore, HinDRL manages to maintain its final achieved accuracy across the 5, 10 and 30-shot cases for the dedicated training budget while both DPGfD and BC didn't. Table shows the final achieved accuracy on the Bi-manual Insertion task. There, only HinDRL was able to learn in the 10-5-1-shor scenarios. This indicates the ability of task-constrained hindsight relabelling to bootstrap learning for long-horizon dexterous manipulation tasks, particularly with high dimensional continuous action spaces.
Demomnatratio
DEGID
HER (fatue)
HisDRL (Ow)
Mean (Sid)
Mean (Sid)
Mean (Sid)
Mcan (Sad)
55 demes
16.06% (0.0)
4.28% (4.4)
18. 39%
78.92% (10.3)
10 demio
2.94% (0.0)
0,0%% (0.0)
0.0% (0.0)
32.05% % (23.3)
Sakmm
0.52% (0.0)
0.0% (0:0)
0.0% (0:0)
17.78% (18.9)
0.13% (0.0)
(0.00(0.0)
0.0% (0.0)
12.58% (18.3)
Table 2: Performance against using different number of demonstrations on Bimanual Insertion.
demos
10
demos
demos
DPGa
0.8
0.8
DPGIt + HER (ununs)
HnDRLL (Curs)
0.6
6.0
125
150
0.0%
75
100
125
150
0.223
0.22
Steps x1000
Env Steps x1000
Env Steps x1000
Steps x1000
Figure 6: Accuracy against different number of demonstrations on the Bring Near Orient task. Each plot shows the overall achieved accuracy during training. HinDRL consistently outperforms the alternatives. Success with one demonstration depends on the selected demonstration. We choose it on random principle.
5.4 STUDYING THE SPEED OF EXECUTION
We study the dependency of HinDRL on the number of demonstrations and the achieved accuracy of the BC component. The results show a definite improvement over these two factors. However, this does not necessarily mean that HinDRL is strictly better than the expert's performance. In practice, quick and nimble expert can still produce higher total output and be more valuable on average than an RL solution even if it is less successful. Therefore, in this section we study the overall speed of performing a task. We measure the speed of solving a task as a function of the per-step reward accumulated over the course of the entire training. Therefore, increased per-step reward means
8

---

Published as a conference paper at ICLR 2022
that an episode takes less environment steps complete. We report our findings in Figure We provide an additional ablation over using different numbers of demonstrations in Appendix A.6 Our findings indicate that HinDRL was always able to outperform the speed of execution obtained from the BC policy while using only DPGfD or DPGfD with HER did not. HER is beneficial for the easier tasks with insufficient number of demonstrations for DPGfD. However, HER alone is unable to solve the longer-horizon complex sequential tasks.
0.010
Grancllcriseed Moachh 22.sst donssst
Sroas and hoor demmooo)
Baach, Group L.ft an Algnn demoa)
Full Irsattonn demoe)
0.006
Suap- Tisg-Mammne mim.lal00 Peaea11
bodel Phac
3.005
0,0005
Step Mermataa
3.000
0.004
0.0004
2200
0.002
0.0002
0.000
0.000
50
75
100
0.000
100
150
50
1000
150
1.00
200
Em Steps 1e+3
Ery Steps 1e+3
Enn Steps 1e+3
Env
Stede 1e+3
Figure 7: Measuring per-step reward using the smallest number of demonstrations that resulted in learning.
5.5 ROBUSTNESS TO THE QUALITY OF THE ENCODER
In the above sections we study performance when using different hindsight goal samplers, HinDRL sensitivity to the number of demonstrations and its ability to learn more efficient policies than the provided expert. However, in all those cases we assumed access to a hand-engineered state encoder. Informative representations is of central important to HinDRL as the algorithm uses the state representations both as input to the RL agent but also during the relabelling stage to assign reward, as discussed in Section 4 However, access to near perfect representations is not always feasible. Therefore, we study the sensitivity of our solution to the quality of the encoder. We evaluate the achieved performance against using learnt representations and raw input too. We compare learnt representations. One is extracted with a B- for B = 0.5, and one with TCC. We hypothesise that the notion of episode progress is of crucial importance to HinDRL. We report our findings in Figure [8] Representations that preserve temporal consistency were able to achieve higher results on all tasks. The dip in success rate is caused from annealing away the BC term in the actor's loss. Using learnt encoding obtained with TCC was capable of achieving near equivalent performance to using an engineered encoding. We notice that using support, Rg for the goal distribution that is comprised of the union between the condiiondt distribution and HER help improve the B B-VAE performance, indicating that the agent's 's rollout was able to partially substitute the lack of notion of progress. However, we provide this more detailed ablation in Appendix A.5 Unlike the engineered encoding, the TCC-based representation does not contain any manipulation-specifee priors.
1.0 Parameterised wps) Aun candioonad
Bring Near
Bring Near Orient,
Full Insertion
brokk Enginened
Eng conittinnn
tia renaiiionn P-NC Task ponditioeed
0.8
bit
conditooed
B-ME Creoder
0.0
0.2
0.4
0.1
0.2
0.0
20
Ent Steps 10+3
40
60
80
Env Steps 10-3
50
100
150
Env Steps 16+3
50
100
150
0.0
Env Steps 1e+3
100
200
300
Figure 8: Robustness to the quality of the encoder. Comparing the engineered encoder with raw state observations and learnt ß-VAE and TCC based encoders. The engineered encoder performs best with TCC achieving close to commensurate performance.
CONCLUSION
We proposed an efficient coalioondiioneed, demonstration-driven solution to solving complex sequen- tial tasks in sparse reward settings. We introduce a novel sampling heuristic for hindsight relabelling using goals directly from the demonstrations and combine it with both engineered and learnt encoders that consistently preserve the notion of progress. We show that HinDRL outperforms competitive baselines. Moreover, the method requires a considerably lower number of demonstrations. In the future, would like to extend this work to multi-task settings, employ vision and move towards the context of batch RL, e.g. through employing distance learning techniques (Hartikainen et al. 2020 to build more informative goal spaces.
9

---

Published as a conference paper at ICLR 2022
REFERENCES
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong. Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
Christopher G Atkeson and Stefan Schaal. Robot learning from demonstration. In ICML, volume 97, pp. 12-20. Citeseer, 1997.
Yusuf Aytar, Tobias Pfaff, David Budden, Thomas Paine, Ziyu Wang, and Nando De Freitas. Playing hard exploration games by watching youtube. Advances in neural information processing systems, 31, 2018.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Proceedings of the AAAI Conference Artificial Intelligence, volume 31, 2017.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective reinforcement learning. In International Conference on Machine Learning, pp. 449-458. PMLR, 2017.
Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov, Jacob Varley, Alex Irpan, Benjamin Eysenbach, Ryan C Julian, Chelsea Finn, and Sergey Levine. Actionable models: Unsupervised offline reinforcement learning of robotic skills. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 15 1518-1528. PMLR, 18-24 Jul 2021. URL httips:/proceedings. mlr prrss vv3//vvvv//////////// wwwwhharebeeeeee html
Jessica Chemali and Alessandro Lazaric. Direct policy iteration with demonstrations, In Twenty- Fourth International Joint Conference on Artificial Intelligence, 2015.
Xin Chen, Sam Toyer, Cody Wild, Scott Emmons, Ian Fischer, Kuang-Huu Lee, Neel Alex, Steven H Wang, Ping Luo, Stuart Russell, et al. An empirical investigation of representation learning for imitation. 2021.
Todor Davehev, Kevin Sebastian Luck, Michael Burke, Franziska Meier, Stefan Schaal, and Subra- manian Ramamoorthy, Residual learning from demonstration. arXiv preprint arXiv:2008.0r1 076682, 2020.
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances neural information processing systems, 32, 2019.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1801-1810, 2019.
Ben Eysenbach, XINYANG GENG, Sergey Levine, and Russ R Salakhutdinov. Rewriting history with inverse rl: Hindsight inference for policy improvement. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 14783-14795. Curran Associates, Inc., 2020. URL http::tpps... httttps://prrceeeiingn htwp:rceeediinn.g...iiieeeeeeeee
cc/
Carlos Florensa, Jonas Degrave, Nicolas Heess, Jost Tobias Springenberg, and Martin Riedmiller. Self- supervised learning of image embedding for continuous control. arXiv preprint arXiv:1901.00945 2019.
David Foster and Peter Dayan. Structure in the space value functions. Machine Learning, 49(2): 325-346, 2002.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off ofolcy updates. In 2017 IEEE international conference robotics and automation (ICRA), pp. 3399-3396. IEFE, 2017.
10

---

Published as a conference paper at ICLR 2022
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-howe long-horizon tasks via imitation and reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1025-1037, PMLR, 30 Oct-01 Nov 2020. URLhttss::: htttts:://rooeeennnnnggooooe ngs m press/presvvvlvv h0tpt//gguppappppp html
Abhishek Gupta, Justin Yu, Tony 7. Zhao, Vikash Kumar, Aaron Rovinsky, Kelvin Xu, Thomas Devlin, and Sergey Levine. Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 6664-6671. IEEE, 2021.
Kristian Hartikainen, Xinyang Geng, Tuomas Haarnoja, and Sergey Levine. Dynamical distance learn- ing for sumi-super and unsupervised skill discovery. In International Conference on Learning Representations, 2020. URL wtttt: /ttpts::////peeeeiiiiii oowr://iieeeeeeeeeeeeeeeeeeeeee.
net. wwt./eet//uru ?id-H1 lmhaVtvr
Nicolas Heess, Gregory Wayne, David Silver, Timothy Lillicrap, Tom Erez, and Yuval Tassa. Learning continuous control policies by stochastic value gradients. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, neur ps. cc/ pdf
volume 28. Curran Associates, Inc., 2015. URL
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Thirty-second AAAI conference on artificial intelligence, 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-second AAAI conference artificial intelligence, 2018.
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint arXiv: 1.01 144, 2016.
Leslie Pack Kaelbling. Learning achieve goals. In IJCAI, pp. 1099- Citeseer, 1993.
Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski. Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic rein- forcement learning at scale. arXiv preprint arXiv:2104.08212, 2021.
Beomjoon Kim, Amir-massoud Farahmand, Joelle Pineau, and Doina Precup. Learning from limited demonstrations. In NIPS, pp. 2859-2866 Citeseer, 2013.
Sergey Levine, Peter Pastor, Alex Krizhevsky, Julian Ibarz, and Deirdre Quillen. Learning hand- eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4-5):421 -33, 2018.
Alexander Li, Lerrel Pinto, and Picter Abbeel. Generalized hindsight for reinforcement learning. Advances in neural information processing systems, 33:7754-776:: 2020.
Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Wenzhao Lian, Chang Su, Mel Vecerik, Ning Ye, Stefan Schaal, and Jonathan Scholz. Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scall -Sale Study. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi: 10. N..115507RS.2000 102..X1000..... XVI.0888.
Chris J Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint a61iv: 1.007 12, 2016.
Ashvin Nair, Bob McGrew, Marcin Andrychowicz. Wojciech Zaremba, and Pieter Abbeel. Over- coming exploration in reinforcement learning with demonstrations. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 6299-6229. IEFF, 3018a.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. Advances in neural information processing systems, 31, 2018b.
11

---

Published as a conference paper at ICLR 2022
OpenAI, llge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand. arXiv preprint, 2019.
Tom Le Paine, Sergio Gómez Colmenarejo, Ziyu Wang, Scott Reed, Yusuf Aytar, Tobias Pfaff, Matt W Hoffman, Gabriel Barth-Maron, Serkan Cabi, David Budden, et al. One-shot high-fidelity imitation: Training large-scale deep nets with rl. arXiv preprint arXiv:/810.0500 2018.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors. arXiv preprint arXiv:2010. 1944, 2020.
Jan Peters and Stefan Schaal. Natural actor-critic. Neurocomputting, 71(7-9):1180-1133 2008.
Tobias Pohlen, Bilal Piot, Todd Hester, Mohammad Gheshlaghi Azar, Dan Horgan, David Budden, Gabriel Marthn Hado Van Hasselt, John Quan, Mel Večerík, et al. Observe and look further: Achieving consistent performance on atari. arXiv preprint 1rXiv:1805.11595. 2018.
Martin A. Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing solving sparse reward tasks from scratch. CoRR, abs/ abs//1802.1067, 2018. URL
org/abs 1802. 10567
Tim Salimans and Richard Chen, Learning montezuma's revenge from a single demonstration. arXiv preprint 112.v:1812.03331, 2018
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approximators. In International conference machine learning, pp. 1312-1320. PMLR, 2015.
Archit Sharma, Michael Ahn, Sergey Levine, Vikash Kumar, Karol Hausman, and Shixiang Gu. Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020. doi: 10.15607/
Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous reinforcement learning via subgoal curricula. Advances in Neural Information Processing Systems, 34, 2021.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. International conference on machine learning, pp. 387-395. PMLR, 2014.
Freek Stulp, Evangelos A Theodorou, and Stefan Schaal. Reinforcement learning with sequences of motion primitives for robust manipulation. IEEE Transactions robotics, 28(6):1360-133 1370. 2012.
Giovanni Sutanto, Katharina Rombach, Yevgen Chebotar, Zhe Su, Stefan Schaal, Gaurav S Sukhatme, and Franziska Meier. Supervised learning and reinforcement learning of feedback models for reactive behaviors: Tactile feedback testbed. arXiv preprint arXhv:2007.00450 2020.
Minija Tamosiunaite, Bojan Nemec, Aleš Ude, and Florentin Wörgötter. Learning to pour with a robot arm combining goal and shape learning for dynamic movement primitives. Robotics and Autonomous Systems, (11111)990-9999 2011.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe. and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards. arXiy preprint arXiv:1707.08817 2017.
12

---

Published as a conference paper at ICLR 2022
Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester, and Jon Scholz. A practical approach to insertion with variable socket position using deep reinforcement learning. 2019 international conference robotics and automation (ICRA), pp. 754-760. IEEE, 2019.
Markus Wulfmeeer, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Noah Siegel, Tim Hertweck, Thomas Lampe, Nicolas Heess, and Martin Riedmiller. Compositional Transfer in Hierarchical Reinforcement Learning. In Proceedings of Roboties: Science and Systems, Corvalis, Oregon, USA, July 2020. doi: 10.15607/RSS
Markus Wulfmeier, Dushyant Rao, Roland Hafner. Thomas Lampe, Abbas Abdolmaleki, Tim Hertweck, Michael Neunert, Dhruva Tirumala, Noah Siegel, Nicolas Heess, et al. Data-efficient hindsight off- off-policy option learning. In International Conference Machine Learning, pp.
11340-11350. PMLR, 2021.
Fan Xie, Alexander Chowdhury, M De Paolis Kaluza, Linfeng Zhao, Lawson Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. Advances in Neural Information Processing Systems, 33:2327-2333 2020.
Kevin Zakka, Andy Zeng, Pete Florence, Jonathan Tompson, Jeannette Bohg, and Debidatta Dwibedi. Xirl: Cross-embodiment inverse reinforcement learning. In Conference on Robot Learning, pp- 537-546. PMLR. 2022.
Guoyu Zuo, Qishen Zhao, Jiahao Lu, and Jiangeng Li. Efficient hindsight reinforcement learning using demonstrations for robotic tasks with sparse rewards. International Journal of Advanced Robotic Systems, (CT]))17298841198999 2020.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.1 PREDICTING PROGRESS
We measure the performance of the learnt representations to encode progress by running a KNN classification. First, we collect 500 demonstrations on the full insertion task using manually defined way points and a PD controller. We vary the starting configuration of each robot arm by 1 for all its joints. Then, we perform a CV split over the collected trajectories and obtain training and validation sets. We train both encoders using the training data. Once training is complete, we process all training trajectories with each encoder which results in two separate encoded data sets. We use those to train two separate 10-fold KNN classifiers one for each type of encoding. Then, we process the never seen before validation set using each of the encoders and evaluate the accuracy of predicting the results. It can be seen that the VAE achieved much higher accuracy when evaluated on the training data as opposed to test, indicating it has overfitted to it. In contrast, the TCC was much better at predicting the episode progress.
Model
Train
Test
VAE TCC
88.6%
63.6%
86.2%
79.2%
Predicting episode progress with 10-fold KNN.
Figure 9:
the episode progress with the KNN classifiers.
A.2 ABLATION OF THE ONLINE GOAL SELECTION STAGE
Goal-conditioned (ge) policy learning takes in input desired goal state to solve for. However, in cases where the goal is as abstract 'solve the task' there may be number of different goals describing the same problem. Therefore, restricting the ge policy to a single target goal state may negatively impact the learning process. Additionally, using a small subset of goals, e.g. goals corresponding the final states of all demonstrations Nair et al. 201 Sa), may be insufficient too. In contrast, conditioning on wider range of goal states that solve the same task can better capture the goal distribution describing the task at hand. As result, we randomly sample a plausible target goal state for each roll-out during training (see Figure 2) We propose to continuously grow a target goal distribution as training evolves and the agent starts solving the training task. In this section, compare the performance of our agent when using a single target goal to represent solving the task, using as target goals only the goal states from the provided near-optimal demonstrations, and continuously growing the goal database as learning progresses. Our findings reported in Figure show that continuously growing the goal database allows for better and faster learning in the context of vaguely formulated goals such as the 3.5mm jack insertion considered in this work.
1.0
Full Inseriton (55 demoos)
Jemagn ond cenor
0.8
0.6
done in
0.4
0.2
200
400
600
Env Steps 1c+3
Figure 10: Different types of on- line goal selections.
A.3 MIXING THE GOAL DISTRIBUTION SUPPORT
Mixing goal candidates taken from the agent's rollout and the provided demonstrations can help for retroactive goal selection. A potential scenario is the one discussed in Appendix A.44 Having noisy, sub-optimal representations that do not preserve the notion of progress can be problematic, An
Pirta neterted 2e and derceo
1.0
Grasp and Bring Noze 120 demooo
Bring Bear Orlent demoos)
Insertion demros).
0.8
0.8
0.8
0.8
age nor
3.6
0.6
20.6
0.8
8.4
0.2
0.0
100
0.0
150
0.0
0.0
200
400
600
Steps
Steps
Staps 1es3
(a)
(b)
(c)
(d)
Figure 11: Measuring per-step reward using the smallest number demonstrations that resulted in learning.
14

---

Published as a conference paper at ICLR 2022
alternative scenario could involve a relatively narrow set of demonstrations. This is particularly evident in complex task settings where having a set of successful demonstrations can still be insufficient to solve the task as there are vast number of failure modes, like in the full insertion task, for example. In this subsection, we focus on studying different candidate heuristics that can help relax these constraints. We build upon the formulation for retroactive goal selection introduced in Section 4.2 which allows us to fuse together the HER-style relabelling strategies and demo-driven ones too.
Using demonstration states as candidate goals: The simplest version of this is a union over both sets where the agent gets to sample a goal that comes from the rollout or the demonstration data. This can be expressed as R (zy E UU p(zz) > 0}. Such formulation can be useful, for example when the provided demonstrations are only partially useful to solving the task. Implicitly letting the agent to sample goals that are part of its own rollout can help model useful behaviours that over time could help get us closer to the provided demonstrations.
An alternative version is when the goal distribution p(zg) is composed of the intersection over the two types of goal distributions discussed in Section 4.2 In this case, the support of the goal distribution becomes R {2g <N 9: p(2g) > 0}. We find this intersection to be useful in cases where the adopted representation does not have an encoded notion of progress. Therefore, choosing goals that are c-close to states produced by the agent's dynamics over the choice of goals we use but still ensure staying close to the target task. We can collect all qualifying goals for a time step / by iterating over the data set of successful trajectories obtained from demonstration and compare to Zz Since z and all Eg are temporally consistent, we can use Eq. to prune the data set and pick the closest 2g for each z, We used a cask-conditioned sampler where we sample directly from distribution implied from demonstrations. However, we can also compose distributions comprised of mixing goals from the agent's 's rollout and the demonstrations. Here we consider two different versions of this.
hypotherically act regularisation
We report our overall results in Figure 11 Our results indicate that using the intersection over the rollout trajectory and the demonstrated goal results in broadly similar resutls as the task conditioned approach. However, the intersection based solution had much higher variance indicating that the agent is is less stable where some seeds achieved near perfect performance and others were closer to failure. We noticed that this type of goal conditioning can be useful when training with a VAE. That is, this sampling strategy can be useful in cases where the quality of the representations and also of the implied task distribution is poor, e.g. when they do not contain notion of progress or in low data regimes. We report these details in Appendix A.4
Using relevant agent states as candidate goals: Utilising the demonstration states to collect can- didate goal distributions can be a powerful tool as we demonstrate in this work. However, in complex tasks, such as the full insertion task (Figure D. relabeling the goals with just demon- stration states can sometimes fail to make the sparse-reward problem casier (as intended by HER) since the resulting goal distributions are still relatively narrow. This is particularly true in the beginning of the training process when there is relatively large mismatch between the agent's 's Q function and the true underlying dynamics the provided demonstrations follow.
1.0
Dut Ensertionn on (SS OWTICE)
0.005
0.8
Stop Noraalzy Roaer
0.004
80.8
0.001
0.4
0.002
0.001
6
200
430
600
0.000
200
400
800
Env Steps Ic+3
Env Steps 1e-3
(a)
(b)
Figure 12: Comparing different demo-driven support distributions.
Therefore, we consider an alternative method that can help speed up the training process. To this end, we can form a collection of candidate goals that is jointly conditioned on both the agent and the demonstrator's ' behaviours but is comprised of all z that fall under the E threshold defined in Appendix A In this setting, we focus on modelling the agent's behaviour directly by focusing only on relevant to the task states as opposed to strictly targeting actual demonstration states. Figure [2 summarises our findings. While using this type of joint conditioning can speed up training (plot on the right), it does not necessarily result in improved performance (plot on the left). We suspect that mixing the goal distribution support can be potentially very useful to using less demonstrations or partially useful demonstrations. We leave this study for future work.
15

---

Published as a conference paper at ICLR 2022
A.4
NOISE SENSITIVITY OF THE ENCODER: CHALLENGING THE NOTION OF PROGRESS
We compare the performance of both the joint- and task- conditioned samplers on the Bring Near + Orient task and report the overall accuracy. We trained for total of 150K environmental steps and report our findings in Figure [3] Relying on task-conditioned samples results in higher ac- curacy for the less noisy observations. This indicates that having a stronger representation is directly related the agent's 's confidence in 'understanding" the actual task it has to solve. This study further confirms our conjecture that notion of progress is paramount to solving complex sequential tasks. Note that the relationship between the level of noise and performance depicted in Figure [3 does not affect the jointly conditioned relabelling strategies as much it does for task-conditioned relabelling. In fact, little bit of noise leads to improved performance for the former while it slows down training for the latter. This indicates that relying on alternative mecha- nisms for indicating progress, such as conditioning on the agent's 's own trajectory can be useful when progress is not successfully encoded in the representations used. This aligns with our motivation from Section 4 that task- coskiiondiiinnin works when we are confident in the quality of the task distribution. However, the proposed ablation this section points towards an alternative mode that can potentially compensate for this. Namely, implicitly informing the agent for the notion of progress e.g. through building heuristics for hindsight goal selection that utilise both demonstrations and agent motion can be useful. We hypothesise that retroactive relabelling using only goals that are both similar to the demonstrations and aligned with the current agent's s trajectory can be useful with respect to the agent's 's current understanding of the dynamics, represented through its current function. Next, consider three different strategies for extracting candidate goals and discuss some of their benefits and limitations.
NIO.
N/O, 2.01
0.3
0.0
0.5
0.4
0.4
0.2
NIO.
9-
Success
0.8
jaint Cand
0.5
0.4
0.4
0.2
50
100
150
Eny Stegs 1e+3
Env Steps 1e+3
Figure 13: Injecting noise to the engineered encoder. Reach, Grasp, Lift, Orient Task.
A.5 QUALITY OF ENCODER: EXTENDED STUDY
1.0
Parameterised Reach wps)
Reach Grasp Lift
Bring Mear + Orient.
Grasp, Bring Near, Orient and Insert Task.
Ong
kng condiieed TCC difenct condioore
Ing coliilo
0.8
TCC coalriued
0.6
givl jaic conditinad
30.4
0.1
0.2
0.0
40
80
50
100
150
50
300
150
0.0
100
200
300
Steps 1e+3
Env Steps
Env Steps 10+3
Env Steps 1e+3
Figure 14: Learnt representations.
The previous section suggests that a joint-conditioned sampler can be more useful when the used representations do not encode notion of progress. In this section we compare using task-conditioned and joint-conditioned encoders using learnt representations instead. We compare using TCC and VAE and benchmark the results against a hand-engineered encoder. Even though TCC with a task- conditioned sampler achieved the closest results the best hand-engineered solution across all tasks. we can see that joint-conditioned encoder can work better for states that do not encode notion of progress.
Figure 14 illsstraasss the achieved results. We can see that using B- B-VAE encoder worked best with goal sampling from a joint-conditioned goal distribution, RG = {2g E in 9 p(zz) > 0). Note that the intersection between both distributions still results in a data set comprised of goal candidates that still belong to the implied from demonstrations task distribution. However, we only used the goals that were similar to the agent rollout. This result is connected to our observations from Appendix [A.4| [A.4] that a joint-cone cantt-onditioned sampler implicitly introduces a notion of progress via utilising the agent's s own motion at the retroactive goal selection stage.
16

---

Published as a conference paper at ICLR 2022
Another interesting observation is the final full insertion's 's task performance. Although TCC- representation came closest to the engineered representation, it was still much lower. The full insertion task relies the most on the contact-rich manipulation to be completed when compared to the rest. The engineered representation contains information relevant to the manipulation which is why we suspect the gap between both learnt and engineered representation is much larger than the rest of the tasks. A potentially exciting future direction is attempting to extract representations that preserve the notion of contact as well as progress.
A.6 PER-STEP REWARD
Srirg Near 120 demaaL
Dring Sear: (12 darren)
DrNng SE demoal
0.020
Bring Scmes
0.020
0.022
29 -ttt SowArS
Reward
0.015
0.015
0.015
0.010
3.010
9.010
0 0.005
2.005
0.000
0.000
100
150
300
100
100
150
Er Stops 1er
Ery Steps 1013
Env Steps 1ef3
Eni Stops J013
(a)
(b)
(c)
(d)
Sring Noor Sa1 donosl
5/11 Moa. O Son domsst
bring Mozr 1 Orient (S demosst
Brio New Oriet (1 demos)
0.05
0.006
berolar Bewwrrd
0.004
3.004
0.004
0.03
0.02
2.002
0.002
0.002
0.01
100
150
0.000
100
0.000
300
150
Em Steps 1e+3
Ery Steps 1e+3
Ene Steps 1e+3
Ery Steps 1e+3
(e)
(I)
(g)
(h)
Pul Insectio LS5 (eres))
Inwettion (10 demac
0.0004
Fut Inwettor demrol
M Insertee derress
0.0000
0,00100
0000000 0.0004
0. 0.0003
0 0.00000
0.000b
0,0008
0.0002
0.00006
7.00000
0.3002
00004
0.00025
0.0003
0000001
0.00002
1.0
202
100
400
0,0000
100
203
300
420
0.0000
0
15%
220
300
400
300
200
120
400
Steps
30+-3
Enu Staps 1843
Emy Steps
204-3
Ern Stops 1m+-3
(i)
(k)
()
Figure 15: Measuring per-step reward.
A.7 PROGRESS-BASED WEIGHTING
Weighting down the BC and actor-critic losses associated with intermediate states can have slight benefits to im- proving the speed of learning of goal-conditioned DPGID, Although in practice there could be many different ways of reweighing loss values, we found two particular ones useful in our setting. One way of scaling such losses is by choosing a fixed weight @ that scales down all non- terminal states' losses during training by the same fixed weight. An alternative weighting can be defined by using quadratically scaled weight using the episode progress. That is, for batch b, we get (Bc(b) 2 LBC(b) (b) and LTD = 2 * LTD, where * indicates element-wise multi- plication and
(a)
(b)
Figure 16: Measuring per-step reward using the smallest number demonstrations that resulted in learning.
11.0, 00
=T otherwise or 2 = P, for i i
1 T 7 }
AP=
We used @ = 0.1 in our experiments. Figure illustrats an example of re- re-weighting on the Bring Near Orient task. Down-weighting intermediate states can lead to slightly faster learning and
17

---

Published as a conference paper at ICLR 2022
higher variance performance. There is difference between weighting states using fixed value and assigning quadratic weighting proportional to the episode progress.
A.8 COMPUTING THE THRESHOLD
There are multiple ways obtain an threshold for our goal con- ditioned reward. We used the provided demonstrations to compute the average distance
1.0
Bring Near Orient (30 demos).
μ +kk, where μ and were extracted using rolling distance between consecutive states from the encoded demonstrations, e-g. || < ml for demonstration d with an m step gap in between the two states and k standard deviations. In our tasks, m = 10 for all tasks but the bring near and orient and the full insertion where we used = 5. We use rolling distance over a window time steps because we did not want to let time step clusters often situated around the different "narrow phases" of a trajectory influence the average threshold. Broadly, we notice a relationship between the size of the rolling window and the precision and recall of the obtained goal- condiionit sparse reward function. We ablate the importance of rolling window size on the bring near and orient task in Figure 17 There, it can be seen that too small of a rolling window size (such as 1) might have noticeable negative impact on learning due to the clusters situated around the different phases. Effectively, too small of window can affect the recall of our obtained reward which can be detrimental to learning. In contrast, too large of a rolling window can affect the speed learning due to allowing for more flexible threshold function. Using too large of rolling window can reduce the precision of the obtained threshold by rewarding too many false positive states. In terms of learning, this can be detrimental to the speed of learning successful policy and might result in converging to poorer performance too.
0.8
2.66
3.44
50
100
Env Steps
Figure 17: Ablating the rolling window.
A.9 COMPUTING THE ENGINEERED ENCODERS
The engineered goal encoders vary between tasks, but in all cases the encoding captures some notion of progress of the agent through the set task.
Parameterized Reach: The engineered encoder for this task concatenates the robot arm pose and mask of which waypoints have been visited so far.
The state encoder is the concatenation of the distance of the left and right grippers from their corresponding cables, the distance between the cable tips, and of whether the grippers have grasped their respective cables.
Bring Near:
The encoder for this task is similar to Bring Near, but also adds the dot product between the z-axes of the left and right cable tips.
Bring Near and Orient:
Bimanual Insertion: The encoder for this task is similar to Bring Near and Orient, but adds the distance of the cable tip from the socket bottom, along the z-axis of the socket.
A. CHOOSING NUMBER OF HINDSIGHT GOAL SAMPLES
We followed the intuition that the complexity of the task guides the number of samples required to capture the overall task distribution. That is, we used samples for the simplest task of Parameterised Reach, 4 for the Bring Near, for Bring Near and Orient and 12 for the Bi-manual Insertion.
18


Reviews:
[{"review_id": "yB7t2f-eqxA", "reviewer": "Reviewer_qVKc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\nThis paper presents a (to my knowledge), novel goal-conditioned RL method. This approach improves appon Hindsight Experience Replay by incoporating demonstrations and efficiently uses them to set goals. I think this way of using goals will be quite useful for goal-conditioned RL in general. The method is well presented and easy to understand. The results are also well analyzed and HindRL shows strong performance when compared to HER and other RL + LfD approaches. I think the ablations, especially looking at the number of demonstrations needed for each method, are very detailed and provide good intuition about HindRL.\n\n\nWeaknesses: \n\nI think the main weakness of this paper is the lack of diversity in the empircal evaluations. It would be good to see results on more complex continuous control tasks in different settings than the bimanual cable insertion, for example the tasks from the HER paper (Andrychowicz et al., 2017). I am willing to increase my recommendation score if the authors can address this issue. \n\n", "summary_of_the_paper": "The authors present a goal-conditioned RL method, which performs relabelling given a set of demonstrations. The main claim is that demonstrations will help guide exploration a lot more efficiently. This method is inspired from HER + LfD methods. The agent starts with a database of demonstrations. At training time, the agent samples a goal from this database, which it updates with successfully reached goal-states. In the hindsight relabelling process, the goals are sampled from the trajectory or from the demonstration database. This approach is evaluated on a set bimanual cable insertion tasks, where it outperforms prior approaches such as HER and is able to learn with fewer demonstrations than RL + LfD. ", "summary_of_the_review": "The method is novel, useful and well presented, but the paper lacks diversity in the domains evaluated. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "x6hDLOTwtxF", "reviewer": "Reviewer_jLXD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong Points\n\n+ The final task used in the paper (“bimanual insertion”) is challenging enough to convince readers of the authors’ claim that the proposed algorithm is a meaningful improvement on learning long-horizon control tasks. Authors’ breaking it down to several tasks by complexity is very helpful.\n\n+ The empirical results are comprehensive and convincing. Difference between the proposed algorithm and prior works is big.\n\n+ The paper is well motivated. The overall “storyline” is clear and makes sense.\n\nWeak Points\n\n- The amount of novelty is mediocre. The main innovation seems to be the new technique for hindsight relabeling. \n\n- Technical clarity can be improved, especially in the way prior works are introduced in the Methodology section, which seems a bit high-level and requires readers to have substantial knowledge in the prior works to understand the full picture. \n\nAdditional Questions\n\n? I’m curious to learn from the authors why HER (both the “final” and “future” variants) underperforms DPGfD in the Bring Near and Bring Near + Orient tasks, as shown in Table 1?\n", "summary_of_the_paper": "This paper aims to improve the learning effectiveness and data efficiency for long-horizon control tasks under sparse rewards with demonstrations. Building upon prior works on DPGfD and HER, the authors proposed a new hindsight goal relabeling technique, which is referred to as “task-constrained goal-conditioned RL” to differentiate against “general goal-conditioned RL” as in prior works. Specifically, instead of selecting goals from the learned agent’s rollouts, the new technique only selects goals from the successful rollouts (and demonstrations). Rigorous experiments demonstrated that the proposed algorithm outperforms prior works on a very challenging bidextrous peg insertion task in simulation, in the aspects of the learned agent’s final performance (e.g. task completion rate) and data efficiency (e.g. number of demonstrations).", "summary_of_the_review": "Overall I enjoy reading this paper. While the novelty is not steller, the experimental results are convincing and impressive. I recommend acceptance for broader dissemination. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gTPDVp2nSs", "reviewer": "Reviewer_BCwv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper studies a difficult problem in robotic manipulation, and shows results on tasks of varying difficulty. Leveraging demonstrations as task-relevant goals is intuitive and straightforward.\n\n- The experiments are thorough. They study the importance of the goal selection strategy, sensitivity to different numbers of demonstrations, and how quickly different learned policies can solve the task.\n\nWeaknesses:\n- In the hindsight goal selection, relabeling the rewards for goals not part of the rollout trajectory necessitates a different reward function separate from the environment reward. The separate reward function seems to need to follow a sparse-reward structure to learn a Q-value function from both reward functions. This requires choosing a thresholding value \\epsilon (although Appendix A.8 presents a seemingly general way to define what this should be, it ultimately still depends on task-specific knowledge of how large the window should be, from which \\epsilon is computed). It would be nice to include a sensitivity analysis of this hyperparameter.\n\n- Relabeling the goals with just demonstration states can fail to make the sparse-reward problem easier (as intended by HER) if the resulting goal distribution is still relatively narrow. For example, we could imagine tasks with a wide initial state distribution and a large state space that the demonstrations only sparsely cover, and even reaching demo states can be difficult. It seems like mixing the demo-driven goal distribution and distribution over reached states could lead to some benefits as shown in Appendix A.4.\n\n- The choice of encoder seems to be quite important, with the TCC results being the closest to those with the hand-engineered features across the four tasks.\n\nFinally, some of the implementation/experimental details are currently a bit unclear to me. My questions are below:\n\n- Which encoder is used for the results in Table 1? Do the other methods in Table 1 operate over the same latent representation as HinDRL? Is the strong performance due to the proposed relabeling scheme or its combination with the extracted representation?\n\n- What is p(z_g) in the definition of R_G? Is this derived from the demonstration data, e.g., a uniform distribution over the set of demonstration states?\n\n- What does the “number of hindsight goal samples” hyper-parameter refer to? Is it the total number of hindsight goals, i.e., |R_G|?\n\n- In Figure 7, why is there a dip in success rate for the different methods at around 60K environment steps? Is this because the BC term in the actor’s loss is annealed away?\n\nMinor comments:\n- The placement of the figures is a bit awkward. For example, Figure 8 corresponds to Sec. 5.4 but is placed in Sec. 5.5; Table 6 corresponds to Sec. 5.2 but is placed in Sec. 5.3. Some of the figures also appear in the paper out of order.\n\n- It’d be nice to have a consistent legend across the 4 plots in Figure 12.", "summary_of_the_paper": "This paper studies long-horizon manipulation tasks given sparse rewards and a few demonstrations from a hindsight relabeling-based approach. Specifically, it leverages the demonstration states as relevant goals in the goal relabeling process to guide task-specific exploration.\n\nWhen evaluating the policy, the online goals are sampled from a goal dataset, composed of successful states and final states from the demonstrations. When relabeling the goals for training, the goals are sampled from the set of states visited by the demonstrations. Rather than operating over the raw state observations, this method utilizes an encoder, which can be either engineered by an expert or learned through self-supervision, to extract a latent representation. Then, the rewards are relabeled in hindsight based on a thresholded distance in the latent space.", "summary_of_the_review": "The paper tackles an important and relevant problem: solving long-horizon tasks given a few demonstrations. The experiments are quite thorough, but there are some missing details that seem critical to evaluating the performance of the method compared to prior work. Also, it would be good to include a discussion of scenarios in which this goal relabeling method might fail, i.e., when reaching the demonstration states is difficult and so relabeling still doesn’t provide additional learning signal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Wwwckh_wAd2", "reviewer": "Reviewer_rSZQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strong points of this paper are:\n\nA more comprehensive ablation study of topics such as encoder quality, effect of number of demonstrations on method performance, and goal-distribution sampling, which are often important but unclear aspects of implementing these types of goal-conditioned methods.\n\nThe use of TCC to learn the latent observation representations seems quite promising as shown in Figure 4 and useful for the reward computation\n\nThe bimanual manipulation task which is evaluated on appears quite challenging and the method achieves significant gains on the shown tasks.\n\nThe weak points of this paper are:\n\nI’m not sure if the claim made in the Related Literature section “However, in all those works, hindsight goals were always chosen directly from the agent’s own trajectory” is accurate -- Nair (2018a) also selects goals directly from demonstration states. Using demonstration states as relabeled rewards in itself does not seem novel.\n\nThe engineered encoder representation which is used for the method evaluation seems to essentially provide a more dense reward for the agent, which is the $\\ell_2$ distance between the engineered features. In this case, I would argue that comparisons should be made to HER and other relabeling strategies which are given access to this engineered reward function (this may already be the case, but it was not clear to me when reading the text). When moving to use the TCC-based encoder and reward function, the performance on some tasks drops significantly, which leads me to believe that the performance improvements could largely be due to essentially giving more signal from the human expert. \n\nAdditional baselines would make the empirical results more convincing, for example, to Reinforcement Learning with Imagined Goals (RIG). \n\nAdditional questions:\n\nHow are the number of relabeled goals for each task selected?\n\nWhy is it that in Figure 9, the performance of the green line for full insertion is around 0.2, whereas it is much higher in Table 1? Are these the same method, or is the difference in number of training steps?\n\nA few recommendations for the manuscript:\n\nThe reference to Table 6 in the text (results) should be to Table 1 instead? There seems to be no Table 6\n\nSection 4.2 was rather hard to parse, for example I don’t quite understand how “In contrast, hindsight goal selection is the process of sampling candidate goal ....” Is different from “Instead, we sample goals for each time step from trajectory ζ using some candidate goal distribution p(zg)”. \n\n", "summary_of_the_paper": "This paper contributes a method called HinDRL, which tackles the important problem of sparse reward robotic RL tasks when supplied with demonstrations. HinDRL seeks to build off of hindsight relabeling to learn a goal-conditioned policy which is trained in a self-supervised manner. Specifically, by relabeling transitions with task-specific goal candidates from expert demonstrations, the agent will be given more task-relevant feedback. By combining the goal selection strategy with goal-conditioned rewards which are either engineered or learned through temporal time-consistency, HinDRL is demonstrated on simulated sparse-reward robotics tasks where it achieves improved final performance and demo efficiency compared to HER. \n", "summary_of_the_review": "My recommendation currently is to reject the paper, because I feel that the experimental results seem to indicate that the engineered features are contributing to the difference in performance and thus creating an unfair comparison, and the novelty of the goal selection method is not currently apparent to me. If the authors would be able to clarify either or both of these points, it would help immensely. If the methods which are compared to are not being currently given the engineered encodings, comparisons to versions which are given that information (as well as additional baselines) would be helpful. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method to improve the sample efficiency of the HER algorithm by sampling goals from a distribution that is learned from human demonstrations. Empirical results on a simulated robotic insertion task show that the proposed method enjoys a better sample efficiency compared to HER.

The reviewers find the paper well-written overall and the proposed idea reasonable. However, there are concerns regarding the limited novelty of the proposed method, which seems incremental. Also, the empirical evaluation suffers from a lack of diversity. The considered tasks are virtually all equivalent to an insertion task. The paper would benefit from further empirical evaluations that include tasks such as those considered in the original HER paper.
Decision: Accept (Poster)