Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: While this paper has some interesting experiments. I am quite confused about what exactly the author are claiming is the core contribution of their work. To me the proposed approach does not seem particularly novel and the idea that hierarchy can be useful for multi-task learning is also not new. While it is possible that I am missing something, I have tried going through the paper a few times and the contribution is not immediately obvious. The two improvements in section 3.2 seem quite low level and are only applicable to this particular approach to hierarchical RL. Additionally, it is very much not clear why someone, for example, would select the approach of this paper in comparison to popular paradigms like Option-Critic and Feudal Networks. 

The authors mention that Feudal approaches "employ different rewards for different levels of the hierarchy rather than optimizing a single objective for the entire model as we do." Why reward decomposition at the lower levels is a problem instead of a feature isn't totally clear, but this criticism does not apply to Option-Critic models. For Option-Critic models the authors claim that "Rather than the additional inductive bias of temporal abstraction, we focus on the investigation of composition as type of hierarchy in the context of single and multitask learning while demonstrating
the strength of hierarchical composition to lie in domains with strong variation in the objectives such as in multitask domains." First of all, I should point out that [1] looked at applying Option-Critic in a many task setting and found both that there was an advantage to hierarchy and an advantage to added depth of hierarchy. Additionally, it is well known that Option-Critic approaches (when unregularized) tend to learn options that terminate every step [2].  So, if you generically apply Option-Critic, it would in fact be possible to disentangle the inductive bias of hierarchy from the inductive bias of temporal abstraction by using options that always terminate. 

In comparison to past frameworks, the approach of this paper seems less theoretically motivated. It certainly does not seem justified to me to just assume this framework and disregard past successful approaches even as a comparison. While the experiments show the value of hierarchy, they do not show the value of this particular method of creating hierarchy. The feeling I get is that the authors are trying to make their experiments less about what they are proposing in this paper and more about empirical insights about the nature of hierarchy overall. If this is the case, I feel like the empirical results are not novel enough to create value for the community and too tied to a particular approach to hierarchy which does not align with much of the past work on HRL. 

[1] "Learning Abstract Options". Matthew Riemer, Miao Liu, and Gerald Tesauro. NeurIPS-18. 
[2] "When Waiting is not an Option: Learning Options with a Deliberation Cost" Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. AAAI-18. 
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper introduces a hierarchical policy structure for use in both single task and multitask reinforcement learning. The authors then assess the usefulness of such a structure in both settings on complex robotic tasks. These tasks include the stacking and reaching of blocks using a robotic hand, as an example. In addition to carrying out these experiments on simulated robots, the authors have also carried out experiments on a real Sawyer robotic arm. 

The particular form of their hierarchical policy for the multitask case is as follows. The policy, which is conditioned on the current state and task index consists of a gaussian mixture, where the individual gaussian densities are conditioned on the state and a context variable. The weights of this mixture are then dependent on a density on this context variable, which is conditioned on the state and task index. The intuition behind this is that the weight portion, which is called the high level component identifies task specific information, while the low level policy learns general, shareable knowledge of the different problems. 

The authors adapt the Multitask Policy Optimisation algorithm for their use by introducing an intermediate non-parametric policy, which is derived by setting KL bounds on the policy w.r.t to a reference policy. Having derived a closed-form solution to this, they go on to learn the parametric policy of interest. 

The authors consider 3 settings of experiments. Firstly, they assess the benefits of the hierarchical structure for single task settings in a simulated environment. For the most part, they find that compared to a flat policy, the hierarchical structure shows benefits only if the initial means of the high-level components are sampled to be different. While the experimental results are shown to support this, further discussion of why this is the case would have been welcome. 

The main benefits of the hierarchical policy are shown in the multitask case, in both simulated and real situations. In fact, the authors have shown that the hierarchical case often shows major benefits in difficult, more complicated tasks (reach vs stacking for example).

I think that the paper was very well written. It is nicely structured, with easy to read language, and without unnecessary jargon or clutter. Where necessary, the relevant extra details were provided in the Appendices.

The following are some additional notes:
1) It would have been interesting to see how the hierarchal policy faired in new tasks that were not a part of the original training set, compared to a flat multitask policy.  
2) Further details about how each task is differentiated from each other in the experiments. That is, what are their different goals, which are reflected by the reward functions. 

As such I recommend this paper to be weak accepted.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper is rather interesting and is able to solve some difficult tasks. A combination of different learning techniques for acquiring structure and learning with asymmetric data are used. While the combination of methods is new I am not sure that this particular combination of methods to train an HRL policy is sufficiently novel. Is the authors can highlight the effects and contribution of how these methods are combined to indicate this better it would be good.

More detailed comments:
- In section 3.2 you mention a reference policy? Can you provide more details on this reference policy? 
- IN the paper it is mentioned that the method collects data, including the reward for each task on a single state, action transition. This assumption seems rather strong. Earlier in the paper, the authors discussed the motivation for learning transferable sub-policies. In the real world, it may not be possible to collect the reward for every kind of task simultaneously.
- The first evaluation in Section 4.1 uses two humanoid environments. While these environments can be considered difficult that does not seem like the multi-task type environment the method is motivated to work well on. There is little sub-task transfer in this task.
- In section 4.1 it is noted that the version that initialized with different policy means works best. How are these means initialized?
- Is the Pile1 collection of tasks really separate tasks? It would be good to have some more details on how these are organized. There may not be a clear definition in the community what is considered a specific task but I am not overly convinced that these "different" tasks are separate. Most of them look like a similar version of pick and place.
- In Figure 2 the hierarchical method is similar in performance on the stack and leave (Pile1) set of tasks and marginally better in the Pile2 set yet does far better on the Cleanup2 set. While these are all simulations with multiple tasks is there some reasoning to why each method looks similar on the Pile1 set of tasks?
- For the robotic tasks, it is noted that again the baseline methods do well on the "Reach" task. It is shown that the RHPO does much better on the Stack task. It would be great if the authors can describe the interesting differences between the tasks. It is not clear how difficult the Stack task is and why it is largely different from Reach + Grasping.
- For the images on the right of Figure 4, It shows a comparison between the tasks and some "components" Are these components the states? The "components' are not explained well in the papers.
- There is no algorithm in the main paper which makes it a little difficult to understand the operation of the learning method. For example how exactly is the learning of the two different levels compared? It seems like they are trained together. If they are they should be compared to HIRO or HAC. How is temporal abstraction handled between the two policy layers if they are trained together?

review_title: Official Blind Review #2

Meta-review: This paper is concerned with improving data-efficiency in multitask reinforcement learning problems. This is achieved by taking a hierarchical approach, and learning commonalities across tasks for reuse. The authors present an off-policy actor-critic algorithm to learn and reuse these hierarchical policies.

This is an interesting and promising paper, particularly with the ability to work with robots. The reviewers did however note issues with the novelty and making the contributions clear. Additionally, it was felt that the results proved the benefits of hierarchy rather than this approach, and that further comparisons to other approaches are required. As such, this paper is a weak reject at this point.
Decision: Reject