Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The manuscript presents CRISP, an approach for hierarchical reinforcement learning from expert trajectories. Their approach relabels states from expert demonstration as subgoals to train a higher level. States are labelled as subgoals when they are the first in a subsequence that can’t be reached by the lower level from the start of this subsequence. Thus, subgoals are chosen that are “just out of reach” for the lower level, developing a learning curriculum. The two levels are trained to optimize both offline reinforcement learning and imitation learning within one objective. The paper provides a bound for the suboptimality of the method compared to an optimal policy. The method is evaluated on two simulated tasks and outperforms two hierarchical reinforcement learning baselines.
strength_and_weaknesses: Strengths:
+ The relabelling of the expert demonstration states based on failures to reach them is a very interesting idea.  The emergence of a learning curriculum from this simple procedure seems very promising and is nicely visualized in Figure 2.
+ The results compared to the baselines on the two investigated tasks look impressive and highlight the strength of using a hierarchy combined with the subgoal learning curriculum.

Weaknesses:
- The method part is at different parts very hard to follow, and I found various parts confusing. Below is a list of points that I think need to be addressed:
    • p. 3, section 3.1: The lower primitive policy $\pi^j_L$ is introduced with superscript $j$, where $j$ denotes "an indicator of the current subgoal reaching capability of the lower primitive". In my opinion this sentence does not sufficiently explain $j$. The indicator $j$ is not further clarified, does not change in the pseudocode, and is dropped later (section 3.2). So why is the superscript $j$ needed?
    • p. 4, Definition 1: What is $D_{TV}$? I'm assuming total variation, but it is not introduced.
    • p. 4, The proof for the suboptimality of the lower primitive is missing. It can probably be derived from the proof for the upper policy, however, seeing that they differ with respect to lambda it is necessary to explain where this difference comes from.
    • p. 5, Eq. 10: What is $\epsilon$? This needs to be introduced.
    • p. 5, section 3.4: Why does Eq. 10 incentivize the higher policy to make "reasonable progress towards achieving the final goal"? Does it not just incentivize generating subgoals that are hard to distinguish from the subgoals of dataset $D_g$?
    • p. 5, section 3.5: It reads like the off-policy RL objectives can be referred to as $J^H_D$ and $J^L_D$. Shouldn't the off-policy objective be $J^H_{\theta_H}$ and J^L_{\theta_L} of Eq. 11 and Eq. 12? Because $J^H_D$ is defined above as the IRL objective (Eq. 10). $J_D^L$ is not defined before. The authors need to be more precise about the different objectives.
    • p. 5, Eq. 11 & 12: $\lambda$ is here introduced as a hyperparameter whereas in section 3.2, $\lambda$ was used as a factor for the upper bound of suboptimality. To avoid confusion different names should be used.
- From the experiments it is not clear how much the approach relies on the expert trajectories to learn to solve the tasks. How many expert trajectories were used for each of the two tasks? How were the expert trajectories generated? How does the number of expert demonstrations influence the system's performance? For example, how do fewer or more demonstrations affect the performance? Could the system deal with "bad expert demonstrations" that are not useful in solving the task? For example, what would happen when a couple of random rollouts are included in the demonstration dataset? I think these questions need clarification in the description of the experiments, further discussion, and maybe additional experiments.
- What is the effect of resetting the subgoal buffer, i.e., the hyperparameter $u$? How does $u$ affect the sample efficiency of the approach? An ablative study evaluating different values of $u$ would help understand how resetting the subgoal dataset affects the method.  
- How is CRISP-BC different to CRISP-IRL? Why does it perform better than CRISP-IRL in the Pick and Place task? Highlighting the difference between the two versions would also help understanding the approach.
- The method requires that the simulator can be reset to an arbitrary state. This is a major restriction for applying the method. The authors acknowledge this limitation but a more detailed discussion on how this limitation could be weakened would improve the paper. For example, it prohibits applying the system on real robots. Thus, the claim of the abstract that the method “[…] is suitable for most real world robotic control tasks” is too strong.
- Please fix the citation style. Use citet only of the citation is part sentence, grammatically (as a subject or object). Otherwise ,use \citep. Here an example: “Learning effective hierarchies of policies has garnered substantial research interest in RL Barto & Mahadevan (2003)” should be “Learning effective hierarchies of policies has garnered substantial research interest in RL (Barto & Mahadevan, 2003)”.
Minor corrections and suggestions:
- typo on p. 1: "balnced"
- bottom of p. 5: it should be Algorithm 2 instead of Algorithm 0
- typo on p. 9: "tn maze navigation"
- Figure 1 should be referenced in the text.
- introduce the abbreviation IRL on p. 3 when first mentioning inverse reinforcement learning
clarity,_quality,_novelty_and_reproducibility: While motivation and related work are written clearly, the method section needs clarifications and improvements. The experiments are mostly clear, but some information (e.g., on the expert trajectories) are missing to fully understand the experiments and be able to reproduce them. There are too many design details (optimizer, learning rate, …) missing, to guarantee reproducibility. However, this could be fixed by providing code to run the experiments. The idea seems novel.
summary_of_the_review: In sum, I think the idea is very interesting, but the clarity and details of the method and experiments still need improvement to be accepted. The idea seems interesting, but it’s potential and limitations could be evaluated much more thoroughly in experiments. 

-- post rebuttal update --
See the answers below. I have raised my score to 5 (and updated correctness to 3).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper investigates the problem setting of off-policy hierarchical reinforcement learning, focusing on automatic-curriculum generation (via the proposed PIP) and bootstrapping from  a handful of expert demonstrations using imitation-based regularization. The paper’s contributions include an adaptive parsing method that segments state-only expert demonstrations into skills that are at the fringe of the current capabilities of the lower-level policy. The proposed adaptive method is shown to outperform its fixed counterpart on two simulated tasks involving a robotic manipulator moving through a maze, and moving a block to a desired goal position. Furthermore, visualizations in the paper confirm that subgoals of increasing difficulties are set by the higher-level policy as training progresses (see Figure 2, Page 3 of manuscript).

strength_and_weaknesses: Strengths Of The Paper:

1) One way the proposed approach is appealing is due to the elimination of one form of expert annotation: using the lower-level policy to adaptively parse expert state-demonstrations into skills, rather than a fixed horizon-based parse.

2) Suboptimality bounds for the higher-level policy and lower-level policy are given, which is implied to become tighter as the lower-level skills improve as training progresses.

3) Implementation details for the algorithm on both tasks is discussed at length, and specific hyperparameter values are given which will help to facilitate reproducibility.

Weaknesses Of The Paper:

1) Experimental evaluation of the approach, while showing the proposed method in a positive light, is not thorough, and can be improved in several ways. Firstly, the method is only evaluated on two reinforcement learning tasks, both of which are based on the same position-based robotic manipulator. This environment functions similarly to a 3D pointmass, where actions are directly added to the current 3D position of the gripper. The environment is made more challenging by using a sparse reward, but the complexity of the underlying task is simple in comparison to other tasks that were studied in previous HRL papers, including locomotion-based maze navigation (from HIRO, Nachum et al. 2018), and torque-based manipulation (from HAC, Levy et al, 2018).

2) While the proposed adaptive parsing method is effective in practice, some details are unclear, which I will discuss next. First, the paper states that on the maze task the replay buffer is flushed and re-populated every 200 steps, and 100 steps for the block task. How many new environment steps are taken or required during this flushing and re-population operation, and are these included in the budget reported (2.93e6 timesteps for the maze task and 6.75e6 timesteps for the block task). For a fair comparison, baselines would ideally be compared to the proposed method under the same conditions (ie, the same number of environment timesteps, including those used by PIP). It remains possible the evaluation already takes this detail into account, but is unclear from the text.

3) On Page 6 of the manuscript, the authors state “our method deals with non-stationarity by regularizing the higher policy with imitation learning” and this is not validated. I agree with the author’s that it is likely the proposed approach does mitigate non-stationarity by design, but this claim is not experimentally verified. Such an experimental verification should be performed that compares to other HRL methods that mitigate non-stationarity. Building on Weakness (2), the authors should ensure all methods operate in the same conditions in regards to any additional samples that PIP requires to relabel the buffer.

4) Performance is reported in a somewhat uncommon format. It would be helpful to report success rate versus the cumulative number of environment samples collected (including any additional samples that may be required during the PIP parsing step, if any).

clarity,_quality,_novelty_and_reproducibility: Overall, this paper presents an original idea for an important problem in hierarchical reinforcement learning: learning effective lower-level skills. The idea is high-quality, and the inclusion of an analysis of the suboptimality of the learned skills helps to improve its quality. However, the paper is not experimentally thorough, and the quality of the paper is brought down by the overall simplicity of the tasks used for evaluation, see Weakness (1). The paper’s experimental limitations outweigh the high-quality idea, and the included suboptimality analysis.

The paper is generally clear, except for a minor issue detailed in Weakness (4) regarding how performance is reported, and in Weakness (2) regarding the data requirements of PIP. After these are resolved, I would consider the paper generally clear, with no major clarity flaws. Experiments appear to be algorithmically reproducible, as hyperparameters of the proposed method (including the introduced \lambda and u and specified), though some details from the environment setup are missing. For example, how are the doors of the maze randomized, and how is the train-test split of the evaluations tasks constructed for each task.

summary_of_the_review: Overall, the paper presents an original and promising idea, but falls short in terms of its experimental evaluation of the idea. In addition, certain flaws related to clarity hinder the interpretability of the results (see ‘Strength And Weaknesses’). I am willing to reconsider my evaluation if weaknesses are addressed, either with new results, or clarifications to questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Learn temporally abstracted skills from expert demonstrations. This work uses a two-level hierarchy, where the subgoals given to the lower level of the hierarchy are always achievable, and progressively increase in difficulty. This is achieved by choosing subgoals of increasing length for the agent until the agent is unable to achieve the subgoal.
strength_and_weaknesses: Strengths:

The paper is relatively easy to understand and the concept is an interesting application of hindsight in learning from demonstration. 

THis paper represents a fairly sizable combination of techniques to produce results.

Weaknesses:

This method is limited by the requirement of being able to reset the environment to a state mid-trajectory, which either requires a carefully designed environment or access to a simulator. This can impact the application of this method to real-world tasks.

The paper combines ideas from a wide range of areas which obfuscates where the improvements come from or if a single core idea is in particular more compelling. In particular, the primitive informed parsing could be performed without an expert and have a similar effect of curriculum, still requiring the agent to move to this information. Including the expert and thus IRL/behavior cloning, this makes it less clear why these components are inherently necessary to the core idea (or resetting hindsight labels). In addition, adding on a hierarchical meta-policy seems to only add an extra layer of complexity.

In the movement domains, there is only marginal improvement, and these domains are simple enough to be toy domains. The majority of the improvements are in the pick and place domain, where it appears that both the hierarchical and RPL methods completely fail. In this case it seems like this is because pick-and-place tasks require expert information, and a basic application of that data to policy learning would probably result in a significant improvement in performance since pick and place tasks have certainly been performed by hierarchical, RL and IRL methods in the past. 
clarity,_quality,_novelty_and_reproducibility: The work is high quality, providing a somewhat novel hindsight-based hierarchical algorithm in the context of supervised learning, and deriving some limited analysis on the divergence between the learned policy and the expert policy. It also provides insight in a sizable number of add-ons and hacks that allow this method to work in simulated robotics tasks, which can be useful for HRL researchers trying to use similar methods. It is formatted clearly and the main ideas are expressed well, although as is typical for HRL, where the benefit comes from can be somewhat obfuscated.
summary_of_the_review: I propose to accept this paper because it has sufficient novelty, is well written and provides interesting insights into getting HRL to work in some simulated robotics domains.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a hierarchical approach to learning from demonstration that uses a novel data relabeling technique to periodically relabel certain states as sub-goals of the task during learning. Combined with other methods, such as a discriminator and inverse RL, the approach aims to learn a curriculum of subgoal tasks to solve the primary task. The reviewers agree that the approach shows promise and has the potential to lead to better learning from demonstration. However, there is a need for more understanding of how the method behaves and why it works well. In particular, it would be helpful to validate the components necessary for the algorithm to work effectively and if it is truly mitigating nonstationarity as claimed. Additionally, the current results are based on only three trials, making it difficult to conclude that one method is superior to another.

Overall, I encourage the authors to conduct more experiments focusing on understanding how the algorithm works rather than just its performance. For example, Figure 2 provides some insight into the curriculum learning aspects of the method, and expanding on this type of experiment for the other components and hyperparameters of the algorithm would greatly increase the reader's understanding of the method and what is necessary for creating effective hierarchical learning from demonstration systems.
justification_for_why_not_higher_score: This paper should not receive of higher score because of the flaws in the experiments. There experiments do not properly justify better performance and there are no experiments that produce knowledge of how the new method actually works.
Decision: Reject