Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors present a goal-conditioned RL method, which performs relabelling given a set of demonstrations. The main claim is that demonstrations will help guide exploration a lot more efficiently. This method is inspired from HER + LfD methods. The agent starts with a database of demonstrations. At training time, the agent samples a goal from this database, which it updates with successfully reached goal-states. In the hindsight relabelling process, the goals are sampled from the trajectory or from the demonstration database. This approach is evaluated on a set bimanual cable insertion tasks, where it outperforms prior approaches such as HER and is able to learn with fewer demonstrations than RL + LfD. 
main_review: Strengths: 

This paper presents a (to my knowledge), novel goal-conditioned RL method. This approach improves appon Hindsight Experience Replay by incoporating demonstrations and efficiently uses them to set goals. I think this way of using goals will be quite useful for goal-conditioned RL in general. The method is well presented and easy to understand. The results are also well analyzed and HindRL shows strong performance when compared to HER and other RL + LfD approaches. I think the ablations, especially looking at the number of demonstrations needed for each method, are very detailed and provide good intuition about HindRL.


Weaknesses: 

I think the main weakness of this paper is the lack of diversity in the empircal evaluations. It would be good to see results on more complex continuous control tasks in different settings than the bimanual cable insertion, for example the tasks from the HER paper (Andrychowicz et al., 2017). I am willing to increase my recommendation score if the authors can address this issue. 


summary_of_the_review: The method is novel, useful and well presented, but the paper lacks diversity in the domains evaluated.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies long-horizon manipulation tasks given sparse rewards and a few demonstrations from a hindsight relabeling-based approach. Specifically, it leverages the demonstration states as relevant goals in the goal relabeling process to guide task-specific exploration.

When evaluating the policy, the online goals are sampled from a goal dataset, composed of successful states and final states from the demonstrations. When relabeling the goals for training, the goals are sampled from the set of states visited by the demonstrations. Rather than operating over the raw state observations, this method utilizes an encoder, which can be either engineered by an expert or learned through self-supervision, to extract a latent representation. Then, the rewards are relabeled in hindsight based on a thresholded distance in the latent space.
main_review: Strengths:
- The paper studies a difficult problem in robotic manipulation, and shows results on tasks of varying difficulty. Leveraging demonstrations as task-relevant goals is intuitive and straightforward.

- The experiments are thorough. They study the importance of the goal selection strategy, sensitivity to different numbers of demonstrations, and how quickly different learned policies can solve the task.

Weaknesses:
- In the hindsight goal selection, relabeling the rewards for goals not part of the rollout trajectory necessitates a different reward function separate from the environment reward. The separate reward function seems to need to follow a sparse-reward structure to learn a Q-value function from both reward functions. This requires choosing a thresholding value \epsilon (although Appendix A.8 presents a seemingly general way to define what this should be, it ultimately still depends on task-specific knowledge of how large the window should be, from which \epsilon is computed). It would be nice to include a sensitivity analysis of this hyperparameter.

- Relabeling the goals with just demonstration states can fail to make the sparse-reward problem easier (as intended by HER) if the resulting goal distribution is still relatively narrow. For example, we could imagine tasks with a wide initial state distribution and a large state space that the demonstrations only sparsely cover, and even reaching demo states can be difficult. It seems like mixing the demo-driven goal distribution and distribution over reached states could lead to some benefits as shown in Appendix A.4.

- The choice of encoder seems to be quite important, with the TCC results being the closest to those with the hand-engineered features across the four tasks.

Finally, some of the implementation/experimental details are currently a bit unclear to me. My questions are below:

- Which encoder is used for the results in Table 1? Do the other methods in Table 1 operate over the same latent representation as HinDRL? Is the strong performance due to the proposed relabeling scheme or its combination with the extracted representation?

- What is p(z_g) in the definition of R_G? Is this derived from the demonstration data, e.g., a uniform distribution over the set of demonstration states?

- What does the “number of hindsight goal samples” hyper-parameter refer to? Is it the total number of hindsight goals, i.e., |R_G|?

- In Figure 7, why is there a dip in success rate for the different methods at around 60K environment steps? Is this because the BC term in the actor’s loss is annealed away?

Minor comments:
- The placement of the figures is a bit awkward. For example, Figure 8 corresponds to Sec. 5.4 but is placed in Sec. 5.5; Table 6 corresponds to Sec. 5.2 but is placed in Sec. 5.3. Some of the figures also appear in the paper out of order.

- It’d be nice to have a consistent legend across the 4 plots in Figure 12.
summary_of_the_review: The paper tackles an important and relevant problem: solving long-horizon tasks given a few demonstrations. The experiments are quite thorough, but there are some missing details that seem critical to evaluating the performance of the method compared to prior work. Also, it would be good to include a discussion of scenarios in which this goal relabeling method might fail, i.e., when reaching the demonstration states is difficult and so relabeling still doesn’t provide additional learning signal.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper contributes a method called HinDRL, which tackles the important problem of sparse reward robotic RL tasks when supplied with demonstrations. HinDRL seeks to build off of hindsight relabeling to learn a goal-conditioned policy which is trained in a self-supervised manner. Specifically, by relabeling transitions with task-specific goal candidates from expert demonstrations, the agent will be given more task-relevant feedback. By combining the goal selection strategy with goal-conditioned rewards which are either engineered or learned through temporal time-consistency, HinDRL is demonstrated on simulated sparse-reward robotics tasks where it achieves improved final performance and demo efficiency compared to HER. 

main_review: The strong points of this paper are:

A more comprehensive ablation study of topics such as encoder quality, effect of number of demonstrations on method performance, and goal-distribution sampling, which are often important but unclear aspects of implementing these types of goal-conditioned methods.

The use of TCC to learn the latent observation representations seems quite promising as shown in Figure 4 and useful for the reward computation

The bimanual manipulation task which is evaluated on appears quite challenging and the method achieves significant gains on the shown tasks.

The weak points of this paper are:

I’m not sure if the claim made in the Related Literature section “However, in all those works, hindsight goals were always chosen directly from the agent’s own trajectory” is accurate -- Nair (2018a) also selects goals directly from demonstration states. Using demonstration states as relabeled rewards in itself does not seem novel.

The engineered encoder representation which is used for the method evaluation seems to essentially provide a more dense reward for the agent, which is the $\ell_2$ distance between the engineered features. In this case, I would argue that comparisons should be made to HER and other relabeling strategies which are given access to this engineered reward function (this may already be the case, but it was not clear to me when reading the text). When moving to use the TCC-based encoder and reward function, the performance on some tasks drops significantly, which leads me to believe that the performance improvements could largely be due to essentially giving more signal from the human expert. 

Additional baselines would make the empirical results more convincing, for example, to Reinforcement Learning with Imagined Goals (RIG). 

Additional questions:

How are the number of relabeled goals for each task selected?

Why is it that in Figure 9, the performance of the green line for full insertion is around 0.2, whereas it is much higher in Table 1? Are these the same method, or is the difference in number of training steps?

A few recommendations for the manuscript:

The reference to Table 6 in the text (results) should be to Table 1 instead? There seems to be no Table 6

Section 4.2 was rather hard to parse, for example I don’t quite understand how “In contrast, hindsight goal selection is the process of sampling candidate goal ....” Is different from “Instead, we sample goals for each time step from trajectory ζ using some candidate goal distribution p(zg)”. 


summary_of_the_review: My recommendation currently is to reject the paper, because I feel that the experimental results seem to indicate that the engineered features are contributing to the difference in performance and thus creating an unfair comparison, and the novelty of the goal selection method is not currently apparent to me. If the authors would be able to clarify either or both of these points, it would help immensely. If the methods which are compared to are not being currently given the engineered encodings, comparisons to versions which are given that information (as well as additional baselines) would be helpful.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to improve the learning effectiveness and data efficiency for long-horizon control tasks under sparse rewards with demonstrations. Building upon prior works on DPGfD and HER, the authors proposed a new hindsight goal relabeling technique, which is referred to as “task-constrained goal-conditioned RL” to differentiate against “general goal-conditioned RL” as in prior works. Specifically, instead of selecting goals from the learned agent’s rollouts, the new technique only selects goals from the successful rollouts (and demonstrations). Rigorous experiments demonstrated that the proposed algorithm outperforms prior works on a very challenging bidextrous peg insertion task in simulation, in the aspects of the learned agent’s final performance (e.g. task completion rate) and data efficiency (e.g. number of demonstrations).
main_review: Strong Points

+ The final task used in the paper (“bimanual insertion”) is challenging enough to convince readers of the authors’ claim that the proposed algorithm is a meaningful improvement on learning long-horizon control tasks. Authors’ breaking it down to several tasks by complexity is very helpful.

+ The empirical results are comprehensive and convincing. Difference between the proposed algorithm and prior works is big.

+ The paper is well motivated. The overall “storyline” is clear and makes sense.

Weak Points

- The amount of novelty is mediocre. The main innovation seems to be the new technique for hindsight relabeling. 

- Technical clarity can be improved, especially in the way prior works are introduced in the Methodology section, which seems a bit high-level and requires readers to have substantial knowledge in the prior works to understand the full picture. 

Additional Questions

? I’m curious to learn from the authors why HER (both the “final” and “future” variants) underperforms DPGfD in the Bring Near and Bring Near + Orient tasks, as shown in Table 1?

summary_of_the_review: Overall I enjoy reading this paper. While the novelty is not steller, the experimental results are convincing and impressive. I recommend acceptance for broader dissemination.
review_title: null

Meta-review: This paper proposes a method to improve the sample efficiency of the HER algorithm by sampling goals from a distribution that is learned from human demonstrations. Empirical results on a simulated robotic insertion task show that the proposed method enjoys a better sample efficiency compared to HER.

The reviewers find the paper well-written overall and the proposed idea reasonable. However, there are concerns regarding the limited novelty of the proposed method, which seems incremental. Also, the empirical evaluation suffers from a lack of diversity. The considered tasks are virtually all equivalent to an insertion task. The paper would benefit from further empirical evaluations that include tasks such as those considered in the original HER paper.
Decision: Accept (Poster)