Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: 	This paper takes a different approach for tackling the hierarchical RL problem. Their approach is to decompose the policy into a bunch of primitives. Each primitive acts according to its own interpretation of the state. All the primitives are competing with each other on a given state to take an action. It turns out that these primitive policies can be transferred to other tasks as they represent subtasks of a bigger task. The paper performs extensive experiments to show that this scheme improves over both flat and hierarchical policies in terms of generalization.
	
	This paper is well-written. I enjoyed reading it. Almost all my questions and doubts are explained when I read through the paper. The framework about using primitive policy to solve a big task is novel and original. The experiments are nice and convincing.
	
	Minor comments:
	• I am understanding the methods as a decomposing the policy into components. Different components are combined together using a probability distribution. To balance the competition and overlap of different primitives, different regularization objective are used.  Did you think about other simple methods, e.g., decompose the policy using linear combination work? It may worthwhile to compare your method with this baseline?
	• About the prior p(z), how to choose it in a meaningful way? I do not see why a unit gaussian is a good prior. It seems you want p(z|s) to be as close to a unit Gaussian as possible?
	• Figure 4 does not show a clear cluster structure. Explain more?

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper draws upon the idea of information bottleneck to do task decomposition so as to learn policy primitives similar to hierarchical reinforcement learning that combine together in a competitive manner to specialize in different parts of the task's domain. These policy primitives don't need a higher-level meta-policy to stitch them together. Instead the decision is made in a decentralized manner balancing the cost of information acquisition with maximizing rewards.
The paper seems to build on the idea of decomposing the task primitives that specialize in different parts of the state space. A related recent paper [1] which used similar ideas, albeit with a central coordinator to do task composition is missing from related works.

There are other issues with the paper as well. 
- Decision making is not exactly "decentralized"? Computing $Z$ in $\alpha_k$, still requires the values from other primitives?
- Sec 5.1 seems tacked on. Motion Imitation is not RL. Experiment details about motion imitation unclear from description. 
- In Figure 5: Zero-shot generalization, it's unclear from the plot whether it actually generalized to solve the tasks A and B. Relative frequency of activation are meaningless without reporting the actual performance on these environments. Where do the 4 indices come from? From the plot above there were only 2 indices but plot below has 4.
- Continuous control tasks required pretraining.

Overall the experiments seem a little underwhelming. More details on transfer performance without pretraining would be quite helpful. I like the idea of a competitive ensemble figuring out a useful task decomposition and using an information bottleneck like approach makes sense.

[1] https://dl.acm.org/citation.cfm?id=3331671

*Edit*: Updated the score after the reading the revision and the authors' responses.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper is about a policy design, where the policy is expressed as a mixture of policies called primitives.  Each primitive is made of an encoder and a decoder, mapping state to actions, rather than temporally extended actions (or options in RL).  The primitives compete with each other to be selected in each state and thus do away with the need for a meta-policy to select the primitives.  The selected primitive in each state trades between reward maximization and information content.  

The paper is well written and is enjoyable to read.  It is helpful for me to have equation (3) in mind before reading about the explanation on the tradeoff between the reward and information, but this is a minor point.  My concern is that by scaling the reward in proportion to L_k redistributes the rewards in a way that is not reflective of the underlying reward structure of the MDP.  If so, the constructed policy \pi could place a high probability on the suboptimal actions.  How do we know if the action selected according to policy \pi will indeed lead to high rewards?
review_title: Official Blind Review #3

Meta-review: In contrast to many current hierarchical reinforcement learning approaches, the authors present a decentralized method that learns low level policies that decide for themselves whether to act in the current state, rather than having a centralized higher level meta policy that chooses between low level policies.  The reviewers primarily had minor concerns about clarity, reward scaling, and several other issues that were clarified by the authors.  The only outstanding concern is that of whether transfer/pretraining is required for the experiments to work or not.  While this is an interesting question that I would encourage authors to address as much as possible, it does not seem like a dealbreaker in light of the reviewers' agreement on the core contribution.  Thus, I recommend this paper for acceptance.
Decision: Accept (Poster)