Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The main contribution of this paper is a sample-efficient RL approach for learning policies for long-horizon tasks, by making use of multiple levels of abstraction, each of which is associated with a set of option templates. An option template defines an option, in terms of initial and final states, but not how to implement it. All option templates are handcrafted at the start of training, so the agent can execute an option template despite not knowing how to implement it yet. During training, the agent learns how to implement each option template in terms of the option templates at the next lower level of abstraction.

The paper evaluates this approach in three domains, all in simulation: craft (a grid world based on Minecraft), stacking with a robot arm, and 11 vs. 11 football. This top-down approach of learning options reaches higher performance than learning the options first and then combining them. It is orders of magnitude more sample-efficient than learning without options.
strengths_and_weaknesses: ### Strengths:
* The proposed approach demonstrates orders of magnitude improvement in sample-efficiency, compared to state-of-the-art RL approaches that do not use options.
* The approach is evaluated in complex continuous control domains.
* Includes an interesting comparison that shows this approach of learning top-down with option templates works better than learning bottom-up (i.e., learning the options first and then connecting them together). The hypothesis is that this is because the learned options typically do not perform perfectly, whereas option templates do, which makes it easier to learn how to correctly connect options together when using the latter.
* Clear explanation of the background, in terms of options and semi-MDPs.

### Weaknesses:
* The approach requires defining the levels of abstraction and the set of option templates in each. The effectiveness of the approach presumably depends greatly on how well the problem is decomposed into these levels of abstraction and option templates.
* The implementation of option templates can be nontrivial, as is acknowledged in the Limitations section.
* The description of the method could be more clear. Including a more detailed Figure 2 would help, perhaps with more concrete examples of what the options could be (e.g., from the Minecraft task), and showing how over the course of training, each of the options is learned in terms of the next level of options. Also, in the implementation of the approach, how does one restrict the policy to pick from the set of options that are valid in a particular state?
* There is no comparison against option value iteration in the manipulation and football environments. Is it because option value iteration would take too long to learn for these tasks?
* No evaluation on real robots. This would be a stand-out paper for CoRL if the evaluation included learning a long-horizon task with a real robot.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes an exciting new approach of using options in a top-down manner, that could be highly effective at learning long-horizon tasks efficiently on real robots. This approach does require significantly more effort to apply than end-to-end RL, since the abstraction levels and sets of option templates need to be defined, as well as a handcrafted policy for each option template. But this could also be an advantage, in that it enables combining low-level controllers with higher-level learned policies. With a few improvements in clarity and evaluation, I think this would be a strong paper and very relevant for the CoRL community.
issues: ### Suggestions / questions:
* Points 3, 4, and 5 from Weaknesses section above
* I'm not sure what Figure 2 is trying to communicate. The graph structure is confusing to me, since each option is executed by a _sequence_ of options at the next level, so ordering matters. I think it would be useful to make Figure 2 more concrete, as suggested above.
* End of Section 3: What happens if the maximum number of episodes L is reached but the option has not been learned properly yet?
* It is worth making space for Related Work in the main paper, rather than leaving it to the Appendix. I recommend mentioning Task and Motion Planning (TAMP) in Related Work, as an alternate hierarchical approach for tackling long-horizon problems in robotics. e.g. Garrett et al. Integrated Task and Motion Planning. https://arxiv.org/abs/2010.01083

### Typos / wording:
* Page 1, line 20: "Designed to achieve intermediate subgoals." - not a complete sentence
* in Algorithm 2, in line 6 shouldn't the initial state be sampled from $\mathcal{I}$, rather than $I$? (i.e., the initial states of the semi-MDP rather than of the option)
* Page 5, line 173: Missing a space before "Once"
* Page 8, line 255: "planers" → "planners"
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper provides an interesting perspective to existing Option-based hierarchical RL methods by proposing the idea of Option Templates: It introduces a novel framework to learn option templates, which enables the agent to have expert intervention as high-level option of the task when they are unsure how to continue the action from the state. Specifically, it could transit from one state to another immediately, where the transition represent the execution of an unimplemented skill primitive. It lays out different levels of option templates, and eventually learns the lower level actions in the environment. This could improve the policy performance and sample efficiency of RL and the paper shows the effectiveness of this method on tasks including robot manipulation and soccer.
strengths_and_weaknesses: Strengths:
- The motivation for using Option Template from the idea of Sticky Mitten is interesting and strong. It provides a nice analogy between human and agent learning. 

- It is a very practical and interesting idea to have expert interventions as high-level instructions which largely reduces the exploration burden and provide learning signals to the agent. 

- The explanation on option template method and idea is very clear. There are clear comparison between the option template 

- The limitation section provides true and to-the-point discussion on the sources of teleportation, which should be considered in future work. 

- There are extensive discussions on how this approach compared with prior methods like option-value iteration and curriculum learning. 

Weaknesses:

- The main concern of the paper is how well it could be applied to more complex robotics tasks that are 1) have longer-horizon which motivates the use of hierarchical RL 2) where the skills are not clearly defined. The robot experiments in the paper involves simple moving of objects, and does not reflect how hierarchical RL could help with more complex tasks.

- There are no real robot experiments and no discussions of how well this could be scaled to real-world setting where there are more intricate manipulation of objects and more complex skills.

- For more complex, long-horizon robotics tasks, how could the skills be extracted and the options to be formed in the first place? Does it assume that it already have skill segmentation done? The learning levels here seems a bit contrived and manually designed for each individual task. 

- While comparing the Option Template method to previous hierarchical RL method is sound, the idea of Option Template have the use of expert intervention, which relates to the line of work on learning with expert interventions. While this is a different formation and a different framing, there should be references made to these methods, listed a few in below:
1. Efficient Learning of Safe Driving Policy via Human-AI Copilot Optimization [Li 2022]

    https://arxiv.org/abs/2202.10341
2. Expert Intervention Learning [Spencer 2020] 

    https://link.springer.com/article/10.1007/s10514-021-10006-9


robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think the Option Template is a very interesting idea that complements the existing option framework and addresses the limitations in previous hierarchical reinforcement learning methods. While it is in theory an interesting approach, the experiments in the paper haven't been able to show the effectiveness of the approach in complex robot manipulation tasks. The fact that the tasks are mainly in grid world and the robotics tasks are simply pushing and placing objects are a bit limited. The authors need to show more evidence on how this method could be used on more complex tasks like robot manipulation etc. Also, there should be more evidence of how this could be scaled to real-world setting with real robots.
issues: As discussed in the Weaknesses section above:

1. There should be more discussion on how the expert intervention can be obtained, and the cases where it is unable to obtain a high-performing policy as the expert. 

2. There should be experiments on more complex tasks like more challenging manipulation of objects (e.g. grasping) instead of pushing and moving objects. Also the skills should be more realistic than simply placing one block at a certain location. If there are limitations to doing such experiments, they should be addressed in the limitation section.

3. More reference and comparison need to be made to intervention-based / interactive / human-in-the-loop RL
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper "Exploring with Sticky Mittens: Reinforcement Learning with Expert Interventions via Option Templates" introduces a hierarchical reinforcement learning approach that divides long time-scales in multiple "option templates", i.e. a group of initial and terminal states without option-specific, low-level policies. The low-level control policies then only come into play at the very end, to successively implement the previously found option templates. The approach is evaluated on multiple reinforcement learning benchmarks in simulation.
strengths_and_weaknesses: Strengths:

	* In general, the research topic of the submitted manuscript is of high practical relevance and hence of interest to the community of CoRL.
	* It is well motivated and the background is well presented.

Weaknesses:

	Unfortunately, the manuscript in its current form comes with a couple of issues.

	* Clarity: The explanation of the approach is very confusing and not easy to grasp for external readers.
	* Technical Novely: The general idea seems very similar to previous work, e.g. [1]. In addition to the lack of clarity, the contributions to the field remain vague and seem minor.
	* Evaluation: Even though the shown results in the given benchmarks are somewhat convincing, they don't really highlight the special ingredients of the proposed method. 

[1] Learning Multi-Level Hierarchies with Hindsight, Levy et al., ICLR 2019
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The current manuscript neither explains the concept very well, nor does it really convince regarding real-life impact. Despite the general idea being very suitable for CoRL, this paper is no candidate for acceptance in its current form.
issues: * Clarity: Some of the mathematical definitions are very vague (definition of environment at learning level l, the definition of the reward functions, the satisfaction of the termination condition). In addition, some things remain undefined, such as "teleporting" or how exactly the policies are optimized. In addition, despite having a section "Learning option policies", from the experiments it seems that some option-policies are given (Tables 1, 3 and 4). Furthermore, the "option templates" shown in these tables correspond to policies rather than "option templates" (which should only be defined over sets of states as defined in the paper). Lastly, it remains unclear, how exactly these option templates are defined. In the paper, it says that "For l = 1, we assume there is a single option template A0", however, what precisely is this option template and how is the division over time for subsequent layers? To summarize, the general concept is clear, but the detailed explanations are really confusing and even seem contradictory to some degree. The respective sections really have to be improved upon clarity and precise definitions. The rather involved algorithms do not bring any additional benefit in this case. Maybe the space could be used to leave the respective sections more space.

* Technical Novely: The additional benefit of the proposed approach, especially compared to [1], has to be put more in focus.

* Evaluation: The proposed method claims to yield better sample complexity due to the limited horizon of each subtask by leveraging option templates in a hierarchy. However, the experiments do not really analyze the effect of this hierarchy. How does granularity (i.e. the number of layers) influence performance, which kind of division does bring the most benefit? As is, the experiments only show results for two levels (with the first being the normal task, it seems) which is not really convincing regarding the general idea.

[1] Learning Multi-Level Hierarchies with Hindsight, Levy et al., ICLR 2019

-------------------------------------

Post-rebuttal:
While I appreciate the rebuttal, the paragraph about the effect of the hierarchy is not really convincing. One key aspect of the approach, splitting a task into multiple subtasks to account for long horizons, is not well analyzed, so the experimental evaluation is really lacking some general insights beyond the chosen simulated benchmarks. On the flip side, the results themselves are impressive and hint to at least some real-life applicability -- so it really depends on the scope of CoRL whether this manuscript would be a candidate for acceptance. However, if the camera ready improves upon clarity and presentation, I rather tend to raise my score to WA.
review_title: null

Meta-review: metareview: The strength and weakness of the paper raised by reviewers are summarized as follows:
Strength:
- The motivation and ideas are clearly presented
- The approach is evaluated on complex continuous control domains such as fetch-and-stack and G Football tasks, and the benefits of the proposed approach is apprppriately demonstrated.
- Empirical comparison between the bottom-up and top-down approaches is interesting

Weakness:
- The proposed approach requires the option templates, which is not trivial to extract. It is not clear how to extract the option templates for tasks with longer horizons.
- More in-depth discussions on the related work are necessary

In addition to the weakness raised by reviewers, AE raises the following concerns: 
- the comparison with IMPALA, DQN and the method in A. Nair et al. [1] in fetch-and-stack and GFootball tasks does not look fair. The task is significantly simplified for the proposed method using the pre-defined option templates, but it seems that the option templates are not given to baseline methods. 
- It seems that there is a gap between the motivation in the introduction and what is done in the proposed method. From the introduction, AE thought that all the layers would be learned simultaneously in the proposed method. However, in the proposed method, the lowest level policy is manually designed and given to the agent as option templates. 

Overall, the paper contains interesting ideas and results. However, it is necessary to improve the presentation in some parts.

=== post-rebuttal comments ===

While some concerns were raised by reviewers, the authors addressed them by deepening the discussion and providing additional results.  Considering that reviewers agree that the paper presents interesting ideas and results, AE recommend the acceptance of the paper. AE also encourages the authors to check the comments from the reviewers once again and make some more efforts to improve the presentation.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)