Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents an approach in which new tasks can be solved by an attention model that can weigh the contribution of different base policies conditioned on the current state of the environment and task-specific goals. The authors demonstrate their method on a selection of RL tasks, such as an ant maze navigation task and a more complicated “ant fall” task, which requires the agent to first move a block to fill a gap in the environment before it is able to reach the goal. 

I found the paper interesting and well written. My primary concern is that the primitive policies are learned independently of the composite policies, which might limit the application of this approach to more complex problems. Additionally, it would be great to also see the concurrent and sequential form of skill combination for the more complex tasks, and not just the point navigation task shown in Figure 7. 

Standard errors on Figures 5 and 6 seem to be missing. Additionally, I was curious that in Figure 4a and Figure 6a, the composite’s performance is already a lot better than the other methods after 0 training steps. Maybe the authors can elaborate on that. Maybe the performance at step 0 is just hard to make out in the graphs?

I would suggest accepting the paper but there could be a more detailed analysis of how the pre-trained sub-modules are used and learning both composite and sub-policies together would make the paper stronger. 

Additional comments:
- Where is the training graph for the Composite-SAV applied to the “ant fall” task? Maybe I missed it?
- Algorithm 1 should probably be moved to the main text. 

####After rebuttal####
The authors' response and the revised paper address my already minor concerns. 
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: What is the specific question/problem tackled by the paper?
This paper addresses the hierarchical RL problem of combining multiple primitive policies (pi_1, …, pi_K) into policies for more complex tasks. Given a number of primitive skills and a new task within an environment, the paper aims to learn to pick and combine the primitives as needed to solve the new task. This problem statement is interesting and the method performs well on difficult tasks. 

However, I argue for rejecting this paper because it lacks meaningful contributions to the field. I do not see how the method presented in the paper is more than RL over  hand engineered action spaces that are better for the tasks. While this improves results, we already know that for any task, there is some best action space for performing that task. This is why most HRL work aims to also find the primitive policies in additional to composing them. Is this any better than option-critic if the options are hardcoded to be the primitives? The experiment state that the option-critic method did not work, but did you give it access to the same primitives?

Summary:
The method presented in the paper is as follows: at each state s_t, the K primitives are queried for their action a_k ~ pi_k(s_t). Then, a biRNN reads in the actions in order from 1 to k. In parallel, the state and a goal are encoded by a network named “Decoder”. The encoded state and the hidden states of the RNN are used to output an attention weight over each primitive. Finally, the output action is the weighted combination of all the actions. The encoders and attention weights are trained with RL. 

This method is evaluated on several mujoco tasks, such as making a cheetah jump hurdles by combining “forward” and “jump” primitives. Each environment has predefined primitives such as “forward” “left” “right” etc. 

This method is compared against HIRO, which does not have access to the primitive policies. It is not surprising that hand engineering primitives helps performance. 

Is the approach well motivated?
The general idea behind the approach is well motivated: using primitive skills to learn complex skills is a useful goal. The details of the method are strange.
I would like to see a better motivation and empirical justification for the biRNN. Why should the primitive’s action be encode in order? The ordering of the primitives is arbitrary and constant: a fully connected network could be used, or the attentions could be output entirely independently per primitive.
In fact, I do see not why the primitives’ actions need to be encoded at all. It would be much simpler for the encoder to look at (s_t, g_t) and output a discrete probability over the K primitives.
The ablations in 5.2 are for outputting actions directly rather than mixture weights. The paper would benefit from ablations where mixture weights are output but without the biRNN or without passing in the primitive’s actions.
 

Is the method well placed in the literature? 
The main idea of predicting weights over multiple experts is not novel (see "Adaptive mixtures of local experts” from 1991). In the context of RL literature, we can interpret the primitive skills as actions directly, and then the method is performing basic RL over a better action space (the better actions being “go left”, “jump” etc. We can also interpret these as options, but unlike options a single primitive is not followed for multiple time steps with a termination condition. Functionally, this is equivalent to regular RL using domain knowledge to engineer the action space. 

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Overall, I think the method has some great merit but I am not overly confident in the reproducibility of the method. Some of the comparisons (HIRO) do not agree with the results in the HIRO paper. Also, more description is needed to describe how the baselines were used in the analysis. Were they also given the pre-trained sub-policies? A more fair comparison might be to give those baseline methods no sub-policies but let them pre-train for an equivalent amount of time as the sub-policies are trained.

 Here are some more detailed comments: 
- Figure 1 is not very clear and does not appear to add much to the explanation of the method. More detail should be included in the caption.
 - In the paper, it is noted that HRL has high sample complexity and needs lots of training data. I find the comment about how HRL requires many more training steps than regular RL very odd. The purpose of HRL is to have better sample efficiency and learn strong polices faster. Has the author observed different? The purpose of HRL is to reduce sample complexity and search in a well suited and structured way.
- The assumption that the sub-policies solve the underlying MDP is rather strong. How are these policies going to be trained to guarantee this? 
- I like the idea of using an attention model to help pick learn a weighting for the combination of a number of sub-policies. I am not sure if using a bi-directional LSTM is the best or simplest method to accomplish this. The authors can look at "MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policies" NeurIPS 2019 for a recent work that is similar to theirs. 
- For Figure 4: Do these methods also get to use the sub-policies that have pre-trained some version of tasks? Also, how are these component policies trained? Over what tasks are they trained? This information is very important to make sure the method is not overly biased to the composition of them. 
- I find the results in Figure 5 very odd. The baseline shows that HIRO does not learn to perform well on these tasks even though these are the tasks from the HIRO paper that it learned to solve rather well. Can this contradiction be explained? 
- For the HIRO comparison was the system also using the composite policies there were pretrained? HIRO is designed to learn the sub-policies concurrently but it seems in this case the authors are using the outputs of the composition policies as input to the HIRO low policy. 
- I do not understand the visualization in Figure 7. How to the colored paths for the agent represent the weights for the compose policy?
review_title: Official Blind Review #2

Meta-review: This paper considers deep reinforcement learning skill transfer and composition, through an attention model that weighs the contributions of several base policies conditioned on the task and state, and uses this to output an action. The method is evaluated on several Mujoco tasks.

There were two main areas of concern. The first was around issues with using equivalent primitives and training times for comparison methods. The second was around the general motivation of the paper, and also the motivation for using a BiRNN. These issues were resolved in a comprehensive discussion, leaving this as an interesting paper that should be accepted.
Decision: Accept (Poster)