Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The submission proposes a novel method for explicit decomposition of hierarchical policies for long-horizon navigation tasks. The approach proposes to separate a policy into 3 modules, high-level planner, intermediate planner and low-level control. The evaluation shows that explicit decomposition is well suited for generalisation across a limited set of RL domains.

The proposed method integrates aspects from a variety of recent work including planning layers from value propagation networks, hindsight training paradigms from hierarchical actor critic and hindsight experience replay and related techniques. The variety of different techniques combined instead of a single main contribution renders it challenging to follow all aspects and in particular to trace relevant contributions to performance - which is rendered harder by a limited evaluation section.

While the approach shows good performance against a couple of start of the art methods, it is necessary to provide sufficient ablations to enable long-term insights for the community. The submission high level goal (explicit decomposition and information asymmetry) is clear, the execution involves the combination of many existing techniques plus variations such that it is hard to make solid statements about the relevance of any part.

It is commendable that the authors have introduced adaptations and improvements to their baselines for a stronger and fairer comparison but the evaluation remains very limited. 
I suggest to run different domains as given by other domains from OpenAI gym or DeepMind control suite. But more importantly I suggest to run further ablations without the intermediate planning layer & with absolute goal positions. Furthermore, since HAC seems to perform worse when combined with the proposed low and mid level policies (RelHAC), it would make sense to compare to the proposed high-level policy using low and mid level from HAC instead. 

The submission provides an overall interesting perspective but makes it hard to narrow down on contribution and important insights by being unclear in formulation and providing only very limited ablations. 

Minor issues include:
- Missing description of the mid level policy - what encourages the proposal of closer short-term goals.
- Missing literature on information asymmetry in RL (e.g. see Tirumala et al 2019 ‘Exploiting Hierarchy for Learning and Transfer in KL-regularized RL’)
- Unclear description of how models that get attached to existing planning layers have been trained (Sec 5.3).
- Additional unclear description in the description of the experiments and method sections (4.2, 4.3)
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper addresses hierarchical deep reinforcement learning (RL), an important problem in control learning and RL. Based on my understanding of this paper and recent prior work, the most important difference between the proposed approach (HiDe) and other recent approaches, such as HIRO and HAC, is that the top-level goal proposal policy uses a learned planner based on VIN and a learned attention mask to decide on a subgoal. There are also other differences, e.g., this policy outputs a goal position that is relative to the agent's position, rather than an absolute position. HiDe seems to demonstrate impressive transferability to both unseen mazes and new agent embodiments, which are important problems to address for hierarchical RL.

Introducing learned planning and attention into the top-level policy seems to also introduce additional assumptions into the method. For example, it is my understanding that HIRO and HAC use the same state representation, e.g., joint positions and velocities of the agent, as input to their top-level policies. In contrast, HiDe uses a top down view of the maze and the x y position of the agent, which certainly is more privileged information. If this is correct, first, this should be discussed more thoroughly and directly in the paper. Second, the experimental setup should be elaborated on: is HIRO or HAC modified to include the same information for the top-level policy? Or can HiDe somehow be extended to not require this information? I do not think that requiring this information is egregious, but currently the experimental comparison is not clear in this regard.

The experiments are arguably the strongest part of the paper, and the transfer results and videos are quite nice. But there is still room for improvement. Table 1 and Figure 5 seem disconnected. In particular, the numbers reported in Table 1 are clearly not achieved in Figure 5. Is the figure cut off early? Furthermore, an additional experiment on a more complicated domain would greatly strengthen the paper. A humanoid agent, for example, seems easy to test for the current method. Another option would be the movable blocks tested in HIRO, though it is unclear if this readily fits into the current method's assumptions. In my opinion, as the paper currently rests heavily on the results, this section should be further improved. Doing so would also improve my rating of the paper.

------
Edit after author response: I appreciate the authors' efforts in providing extensive responses to all of the reviewers' concerns as well as a significant general response detailing what seems to be a large amount of additional experimental work. I think that all of this warrants a change to my score, and seeing that the authors have more or less addressed my concerns, I am bumping up to a weak accept.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes a neat framework for creating HRL framework that will be able to generalize its application to slightly different environment layout. This is done via an image-based top-down from as input to the high level. An intermediate layer is used to help create more fine-grained goal specification for a final goal-based control layer. These layers are trained together using HAC. Overall, the method shows promise but there needs to be more analysis to understand which parts of this combination of ideas are the most important. The results are also only shown for a single environment. Last, the generalization analysis in the paper does not appear to be overly thorough. It would be good to perform this on more than one type of environment also the random environment is not very random.

More detailed comments:
- The results seem very similar to some of the work in "Universal Planning Networks" that did not need a more complex HRL design to achieve subgoal specification via images. This should be discussed more in the paper.
- The authors point out that the use of relative goal positions "ensures generalization to new environments", this is a rather strong statement. The use of relative goal specification my help improve generalization but that can only be shown empirically.
- The demonstration to show that the method generalizes to other configuration after being trained on a fixed environment should be evaluated over many randomly generated environments so that we have a non-biased estimate of the true generalization performance. In Table 2, it is rather surprising that the HiDe trained model does better on the "Random" environment vs the "FOrward" environment it is trained on. Can more details be provided on how the "Random" environment is created? Are the locations of the walls randomized? Is the initial position of the agent and goal randomize?
- The video seems to contradict the ordering of operations for training the planning network. The video suggests that first it is learned with the Ant then transferred to the ball which is less complex to control.
- At the beginning of the prior work section, it is noted that many other methods require prior knowledge of the environment I would say this method also requires certain kinds of prior knowledge about the task. For example, a top-down view of the environment is needed which is not often feasible.
- HIRO and HAC use a more proprioceptive state space but I don't think the sharing of global states is intentional. I am not convinced that this choice, in particular, is what makes the approaches prone to overfitting.
- You show a comparison to the "windows" created from your method vs a fixed neighbourhood. Do you perform any empirical evidence that your introduced methods provide an improvement over this fixed window?
- It is mentioned at the end of section 4.1 that MVProp is differential so it can be trained with the Bellman error objective. Because many policies are being trained concurrently does the MVProp attention model need to be recomputed after every sub-policy update? Does the frequency of updates have a large effect on performance?
- Section 4.2 introduces an interface layer that is not a very common practice. It would be good to include an ablation study of the effects of this introduced layer.
- In section 4.3 it says that the control layer is the only layer with access to the agent's proprioceptive state. Would it not be good to at least include the agent facing direction or current average velocity to higher layers to improve the attention mask estimation?
- In figure 5 it says HIRO converges the fastest because it has dense rewards. Can you be more specific? Also, If different agents are using different reward signals I am not sure this evaluation is a fair comparison.
- Are tables 2 and 3 just for the HiDe algorithm? Is it possible to include data for the other algorithms? 
- You perform an experiment to train HiDe with random initial and goal locations for comparison. I think running this comparison for HIRO and HAC would be a good additional point of comparison. This would help the reader know if the generalization is not biased to the particular initial environment configuration for Maze Forward.
- In the generalization analysis for the paper, how is the analysis performed? There are percentages for the success of the policy, where does the randomness come from is the agent state and goal are always fixed? Are these averaged because the agent has a stochastic policy during evaluation? If this is the case how many random trajectories are collected to compute these statistics?
- It would be very helpful to have a description of the algorithm in the paper. How the algorithm works is not very clear and some details about how the goal and states are passed to the different policies would be very helpful if anyone wanted to reimplement this work.

Updated comments:

- The addition of more results and added analysis helps show the improvement of this method over the most related baselines.
review_title: Official Blind Review #2

Meta-review: The submission proposes a complex, hierarchical architecture for continuous control RL that combines Hindsight Experience Replay, vision-based planning with privileged information, and low-level control policy learning. The authors demonstrate that the approach can achieve transfer of the different control levels between different bodies in a single environment.

The reviewers were initially all negative, but 2 were persuaded towards weak acceptance by the improvements to the paper and the authors' rebuttal. The discussion focused on remaining limitations: the use of a single maze environment for evaluation, as well as whether the baselines were fair (HAC in particular). After reading the paper, I believe that these limitations are substantial. In particular, this is not a general approach and its relevance is severely limited unless the authors demonstrate that it will work as well in a more general control setting, which is in their future work already. 

Thus I recommend rejection at this time.
Decision: Reject