Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: The paper develops a hierarchical reinforcement learning algorithm and analyzes its behaviour in four robotic manipulation and navigation tasks. The approach is based on a two-level hierarchy, *scheduler* at the top and *worker* at the bottom. This is similar to other approaches in the literature and the algorithm uses many ideas and elements from existing algorithms. However, these ideas and elements are combined in a novel and well-justified manner. The result is an algorithm that yields good results in a range of problems. The experiments are well done. The paper is generally organised well and written clearly. Relevant literature is reviewed well. 

The paper can be improved by a more comprehensive and detailed analysis of the behaviour of the algorithm, in particular, the options that are found. Section 4.4 is useful but very short. It describes only two options. Perhaps such an analysis can be added to the appendix.

In the proposed algorithm, the scheduler outputs an option every K steps in the environment. It would be reasonable to question whether more flexibility would be useful here, one that allows for varying option durations.

The authors state in Section 2 that they 'will use the terms “goal”, “option”, and “skill” interchangeably.' In the literature, these terms refer to related but different concepts. Using them interchangeably is not good scientific practice.

I did not find Figure 1 particularly useful. The simple and intuitive structure of the algorithm does not come through in the figure.  
review_title: The paper develops a new hierarchical reinforcement learning algorithm that yields good results in four robotic manipulation and navigation tasks. The analysis is well done. The paper is clear and well structured. 

final_score: rating: 7: Good paper, accept
review_content: This paper present HIDIO, a hierarchical RL method that leverages self-supervised losses to discover intrinsic options while learning a scheduler to leverage the learned options to optimize the accumulated reward. HIDIO differentiates from prior work through enabling the low-level network/worker to discover task-agnostic options that can be generalized to future tasks, thus requiring no pre-training of skills,  and at the same time makes minimal assumption about the task structure.

This paper is in general well-written with clarity. The proposed  method is technically sound and empirically outperforms existing methods. The experiments in four different tasks first demonstrate an ablation of different feature extractors used for the self-supervised loss and analyzed the each of their advantages/disadvantages; followed by experiments comparing HIDIO with existing hRL methods and show that HIDIO outperforms in all tasks. Although the experiments are conducted in two different domains and four different tasks, all four tasks seem to be similar in nature (i.e. all pushing and reaching tasks). It would be great to see how HIDIO compare with other methods in more complex task domains such as in the work of [1] and [2], where the skill discovery network need to work with high dimensional raw input data and computing state-similarity can be tricky.


[1] Lynch, C., Khansari, M., Xiao, T., Kumar, V., Tompson, J., Levine, S., & Sermanet, P. (2020, May). Learning latent plans from play. In Conference on Robot Learning (pp. 1113-1132).
[2]Chuck, C., Chockchowwat, S., & Niekum, S. (2020). Hypothesis-Driven Skill Discovery for Hierarchical Deep Reinforcement Learning. International Conference on Intelligent Robots and Systems, 2020.


review_title: a novel hRL method that learns options in a self-supervised way

final_score: rating: 4: Ok but not good enough - rejection
review_content: #######################################################################

Summary:

In this paper the authors present a new method for hierarchical reinforcement learning, demonstrated with a 2 layer architecture, in which the higher layer Scheduler policy choose lower level Worker policies at fixed intervals. The lower level Worker policies (options) are trained through an intrinsic entropy minimization.

#######################################################################

Reasons for score:

This paper seems to be based strongly on the work in Sharma et al., 2019b and Eysenbach et al., 2019, and while it makes a number of interesting modifications to those methods there is in sufficient decoupling / ablation of the issues. I think the paper would be notably stronger with additional experimental evidence 

#######################################################################Pros:

1. The paper makes some reasonably sensible extension to well-know methods.

2. The authors treatment of hyper parameters and implementation details looks to be thorough

#######################################################################

Cons:

1. I found parts of the paper difficult to parse. For example it was not immediately clear to me what exactly an option $\bf{u} \in [-1,1]^D$ is... What values can / does D take? Another free parameter? Is $\bf{u}$ a latent variable that yields an option through the worker policy? (this terminology would seem to better align with that in Eysenbach et al., 2019 and the broader literature around options). I'm also not sure $\bar{\bf{s}}_t$ (single subscript) is defined?

2. As the authors note the paper introduces a number of new elements: the reset of the worker policy every k-step, the utilization of full sub-trajectories, and the various instantiations of the discriminator... but these are never unwound. It would seem to be the case that there would be a fair bit of interplay between these elements. Additionally it appears that there is some need to modify the implementation to support these choices - most notably the discounting for the "shortsighted worker"... is this still necessary if full sub-trajectories are not used?

3. The experimentation did not address some fairly obvious enquires (what is the effect of different choices of K in your experiments? How would a practitioner approach choosing this value?)

4. There is remarkably little in this paper that talks about the actual options uncovered via this method. Does the method uncover different options for different values K? What do they look like? etc.

#######################################################################

Questions during rebuttal period:

Q1: a fair amount of the focus of the paper appears to be on the instantiation of the discriminator - do you consider that to be a central contribution?

Q2: you talk about the fact that a key divergence from priori work is that there is "no constraint on the value on what $\bf{u}$ could represent." Can you talk about why this is a good thing and have you explored when it might not be such a good thing?

#######################################################################
review_title: Review of Hierarchical Reinforcement Learning by Discovering Intrinsic Options: little to say about options

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper describes an interesting approach to self-supervised option learning for hierarchical reinforcement learning. In particular, the authors propose to learn options through an intrinsic entropy minimization objective conditioned on option trajectories. Empirical studies are performed to demonstrate the success rate and sample efficiency of this method over two state-of-the-art methods.

The key concern that I have with this paper is its organization and presentation. I am concerned that the paper takes too much space describing what other methods do not do (throughout the contributions sections), rather than concretely describing the approach. Furthermore, many sections seem to be an aggregation of contributions, related work, and results, which further makes the paper hard to follow. I believe that this does the paper a disservice by decreasing readability. Therefore, I am currently voting for rejecting the paper. 

Questions and Comments:
1) In the Introduction, the authors claim that their approach requires "little" manual design? Can this claim be better grounded? What design is required?
2) There is extensive work on option discovery [1,2]. as well as learning decompositions for reinforcement learning [3-6]. I would have liked to see a more extensive literature review comparing and contrasting with these works.
3) On page 2, the authors say, "for the purpose of this paper, we consider a hierarchy of two levels". However, it is not clear to me how this approach would extend to a hierarchy with more than two levels. Can the authors please clarify this?
4) What is k? h? (page 2)
5) On page 3, the authors claim that, unlike prior work, their formulation has no constraints on what the options can be. Can the authors comment on this claim in more detail?
6) In section 3.2, the authors mention the use of goals. Are the authors considering goal-conditioned MDPs [7]? If so, this should be made more explicit.
8) The MDP definition in section 3.2 is unclear. Could the authors please clarify this definition? 
9) What is meant by "resolution" (section 3.2)? 
10) Why does the objective defined by the authors enable skills that have more diverse semantics than previous approaches (claimed on page 4, section 3.2)?
11) Page 5, section 3.4: is the importance correction not used at all (for either the scheduler or discriminator)? 
12) What assumptions are needed for this method to work well?
13) In the experiment section, it says that in the pusher environment, success means that the goal is achieved in the final step of the episode. Does this mean that pushing the object to the position does not terminate the episode and, instead, the object must remain there until the episode terminates in a predetermined number of steps?
14) Can the authors please clarify the purpose of the additional suite of environments? What do these environments enable that existing ones do not?

[1] Brunskill, et al., PAC-inspired option discovery in lifelong reinforcement learning. ICML, 2014.
[2] Topin, et al., Portable option discovery for automated learning transfer in object-oriented Markov decision processes. IJCAI, 2015.
[3] Mehta et al., Automatic discovery and transfer of MAXQ hierarchies. ICML, 2008.
[4] Winder et al., Planning with abstract learned models while learning transferable subtasks. AAAI, 2020.
[5] Li, et al., An efficient approach to model-based hierarchical reinforcement learning. AAAI, 2017.
[6] Rafati, et al., Learning representations in model-free hierarchical reinforcement learning. AAAI, 2019.
[7] Nasiriany, et al., Planning with goal-conditioned policies. NeurIPS, 2019.
review_title: Official Review 4

Meta-review: This paper presents an approach to hierarchical RL which automatically learns intrinsic task-agnostic options. The approach involves a two-level hierarchy, with policies learned by lower-layer Workers and selected by a higher-layer Scheduler. The approach is evaluated on four complex tasks and is shown to outperform existing methods.
There were initial concerns with this paper around clarity of a number of points. These included the contributions of this work and questions around the experimental results, such as  discussing the learned options themselves. The authors provided extensive responses to these concerns, and updated the paper accordingly, including addition results and analysis. I believe the paper is now much clearer with interesting contributions.
Decision: Accept (Poster)