Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The setting in this paper is very interesting but I would have liked the authors to discuss and report failure models. Especially as the richness of behaviors gets more complex, I expect this approach to have diversity issues with the skills learnt. Did the authors try the approach with an agent more complicated than the 3-link swimmer? I guess part of what I am wondering is whether the idea in this paper has been pushed to its limits. If it has, it would be good to know where it fails. 

Also looking at Sec 5.3 -- " let X be a random variable denoting the grid in which the agent is currently situated" -- is the space discretized? And if so why and what happens if it isn't. 
review_title: diversity and failure modes

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: I like the setting presented in this paper but I have several criticism/questions:

(1) What are the failure model of this work? As richness of behaviors get complex, I expected this approach to have issues with the diversity of skills that could be discovered.

(2) Looking at Sec 5.3 -- " let X be a random variable denoting the grid in which the agent is currently situated" -- is the space discretized? And if so why and what happens if it isn't. 

(3) Expanding on the first point, does the approach work with more complicated embodiment? Say a 5-link swimmer instead of 2? I think this is important to assess the generality of this approach

(4) Authors claim that "Recently, Heess et al. (2016) have independently proposed to learn a range of skills in a pre-training environment that will be useful for the downstream tasks, which is similar to our framework. However, their pre-training setup requires a set of goals to be specified. In comparison, we use intrinsic rewards as the only signal to the agent during the pre-training phase, the construction of which only requires very minimal domain knowledge."

I don't entirely agree with this. The rewards that this paper proposes are also quite hand-crafted and specific to a seemingly limited set of control tasks. 
review_title: null

final_score: rating: null
review_content: In figure 1, is it correct that (1) concatenation means based on categorical value, you have different bias to the first hidden layer, and (2) bilinear integration means you have different weight matrix connecting observation to the first hidden layer?
review_title: Snn architecture 

final_score: rating: null
review_content: Sorry for some last minute questions. What is the intuition for why the strong baseline in section 7.3 perform very poorly and can't solve the task? Assuming "CoM maze reward"is this baseline. Does the baseline also take into account the extra experience pre-trained policies have got? What is multi-policy, as compared to snn?
review_title: Clarification on experimental results

final_score: rating: 7: Good paper, accept
review_content: Interesting work on hierarchical control, similar to the work of Heess et al. 
Experiments are strong and manage to complete benchmarks that previous work could not. Analysis of the experiments is a bit on the weaker side.

(1) Like other reviewers, I find the use of the term ‘intrinsic’ motivation somewhat inappropriate (mostly because of its current meaning in RL).  Pre-training robots with locomotion by rewarding speed (or rewarding grasping for a manipulating arm) is very geared towards the tasks they will later accomplish. The pre-training tasks from Heess et al., while not identical, are similar. 

(2) The Mutual Information regularization is elegant and works generally well, but does not seem to help in the more complex mazes 1,2 and 3. The authors note this - is there any interpretation or analysis for this result?

(3) The factorization between S_agent and S_rest should be clearly detailed in the paper. Duan et al specify S_agent, but for replicability, S_rest should be clearly specified as well - did I miss it?

(4) It would be interesting to provide some analysis of the switching behavior of the agent. More generally, some further analysis of the policies (failure modes, effects of switching time on performance) would have been welcome.
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: *Edited the score 6->7.

The paper presents a method for hierarchical RL using stochastic neural networks. The paper has introduced using information-theoretic measure of option identifiability as an additional reward for learning a diverse mixture of sub-policies. One nice result in the paper is the comparison with strong baseline which directly combines the intrinsic rewards with sparse rewards and shows that this supposedly smooth reward can’t solve tasks. Besides the argument made from the authors on difficulty on long-term credit assignment/benefits from hierarchical abstraction, one possible explanation for this might be the diversity requirement imposed in sub-policy training, which is assumed to be off in the baseline case. Wonder if this can shed insights into improving the baseline and proposing new end-to-end hierarchical policy learning as hierarchical REPS/option-critic etc. papers do. Nice visualizations.

The paper presents a promising direction, and it may be strengthened further by possibly addressing some of the following points. 

1) Limited diversification of sub-policies: Both concatenation and bilinear integration allow only minimal differentiations in sub-policies through first hidden weight, which is not a problem in the tested tasks because they essentially require same locomotion policies with minimal diversification, but such limitation can be more obvious in other tasks where ideal sub-policies are more diverse. Thus it is interesting to see it apply on harder, non-locomotion domains, where ideal sub-policies are not that similar, e.g. for manipulation, solving some task from one state can be very different from solving it from another state. 

2) Limitation on hierarchical policies: Manager network is trained while the sub-policies are fixed. Furthermore, the time steps for sub-policies are fixed. This requires “intrinsic” rewards and their learned sub-policies to be very good for solving down-stream tasks. It would be nice to see some more discussions/results on handling such cases, ideally connecting to end-to-end hierarchical policy learning.

3) Intrinsic/unsupervised rewards seem domain-specific/supervised rewards: Because of (2), this seems unavoidable. 

review_title: Final Review

final_score: rating: null
review_content: It is an interesting step toward hierarchical RL. Comments/questions:
- Hierarchical REPS [C Daniel, G Neumann, J Peters, 2012] imposes a hard constraint on the entropy of conditional of option given state and action to encourage identifiability of skill/”option”, which is similar to your information-theoretic regularization. It is reasonable to discuss the differences. 
- The rewards for pre-training the SNN policy network are called intrinsic rewards/unsupervised rewards, but they seem hand-specified rewards specific to domains and expected downstream tasks. It is helpful if you cite the paper which defines intrinsics reward as you do when you give examples.
- It would be good to see other examples of hierarchical RL besides locomotion + navigation. In locomotion + navigation, reward functions are easy to define for each stage, and most of the benefits seem to rely on the knowledge that if the agent has learned basic locomotion, any future navigation-related tasks will benefit. 

review_title: preview question

Meta-review: The authors provide an approach to policy learning skills via a stochastic neural network representation. The overall idea seems novel, and the reviewers are in agreement that the method provides an interesting take on skill learning. The additional visualization presented in response to the reviewer comments were also quite useful.
 
 Pros:
 + Conceptually nice approach to skill learning in RL
 + Strong results on benchmark tasks
 
 Cons:
 - The overall clarity of the paper could still be improved: the actual algorithmic section seems to present the methods still at a very high level, though code release may also help with this (it would not be at all trivial to re-implement the method just given the paper).
We hope the authors can address these outstanding issues in the camera ready version of their paper.
Decision: Accept (Poster)