Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper is under the topic of hierarchical reinforcement learning. The motivation of this paper is "most methods still decouple the lower-level skill acquisition process and the training of a higher level that controls the skills in a new task." The paper proposes a method to learn higher-level skill selection and lower-level skill improvement jointly.

What I like in this paper:
    1. The paper, in general, is well-written so that I can understand it well.
    2. Experiments are question-driven and provide interesting results.
    3. Theories are closely related to the algorithm.

Key reasons for my rejection:
    1. My biggest concern is the motivation of this paper. 
    The joint learning of higher-level policy and lower-level skill discovery is not rare in modern literature. Some works are even cited in this paper, for example, option-critic, feudal network, etc. These methods fix their skills in the new task, not because they are inherently not able to do so, but because they want to demonstrate that the learned skills can be reused in new tasks, even if there is no further adaptation. I agree with the author that the agent needs to adapt its skills when faced with new tasks. But I don't think most works are limited in this aspect, as claimed by the paper in the abstract.

    2. I think the author didn't justify his key design choices well. 
    This paper is under the research area of "hierarchical reinforcement learning." However, just like temporal abstraction, the HRL is a general idea instead of an existing problem formulation or a particular algorithm. It seems that the author is not aware of this point as the paper claims a particular way of achieving HRL is the HRL itself (in section 4.1 "In the context of HRL, a hierarchical policy with a manager πθh(zt|st) selects every p time-steps one of n sub-policies to execute."). I would like to see the paper takes the responsibility to justify the reason it follows this particular way. There are two more key decisions the paper proposed but not fully justified and analyzed.
        1. Why is random length a valid choice? The paper doesn't tell readers the consequence of this design choice. For example, what about the optimality of the solution? Since bounding the random length needs prior knowledge, how difficult is it to come up with the prior knowledge. Is the algorithm sensitive to prior knowledge?
        2. Why is it fine to assume "for each action, there is just one sub-policy that gives it high probability"? What would be the consequence of this assumption? Well, the extreme case is each action is only being chosen by one sub-policy. Therefore, executing the sub-policy becomes executing a repeated sequence of the same action. Obviously, this is a kind of temporal abstraction but is a very limited one. 

Other small issues:
Section 2: "... maximize the discounted expected reward ..." should be "... maximize the discounted expected return ...".
Section 2: the horizon T in the definition in \eta should be H.
Section 4: the advantage function is not defined.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The main problem they try to tackle is to train agents for unseen tasks and environmental changes. They show that their method has a better performance and is more robust against sensor errors and physical parameter alterations.

The authors clearly position their work in the HRL paradigm and explain current limitations/challenges within that field. Alike other HRL agents, their method has two types of policies (manager and subpolicies), but different from other works they do not keep parameters fixed in post training for new tasks. In addition to the parameters, they do not fix the time length. 

The paper is very well written, clearly stated the contributions. 

Remarks:
- Fig 2 has no caption. How are the colors of balls obtained, since they only explain how the sensors (lidar) measure distances to balls (bombs/apples).
- Fig 4/5a, some agents (blue) seems to have undesired behaviour (until half of the iterations). This behaviour is not described anywhere. 
- The URL of the website with code and videos does not have any code. 

Questions to the authors:
- Closest work is Frans et al. (2018). The experiments do not show Frans et al. as a benchmark method. Why?
- HiPPO shows to have a higher robustness. Why are the results of different methods (p=10/random) in Table 1 for different environments (Snake/Ant) different?



review_title: Official Blind Review #564

Meta-review: This paper considers hierarchical reinforcement learning, and specifically the case where the learning and use of lower-level skills should not be decoupled. To this end the paper proposes Hierarchical Proximal Policy Optimization (HiPPO) to jointly learn the different layers of the hierarchy. This is compared against other hierarchical RL schemes on several Mujoco domains.

The reviewers raised three main issues with this paper. The first concerns an excluded baseline, which was included in the rebuttal. The other issues involve the motivation for the paper (in that there exist other methods that try and learn different levels of hierarchy together) and justification for some design choices. These were addressed to some extent in the rebuttal, but I believe this to still be an interesting contribution to the literature, and should be accepted.

Decision: Accept (Poster)