Title: Stochastic Neural Networks for Hierarchical Reinforcement Learning
Authors: Carlos Florensa
Yan Duan
Pieter Abbeel
Venue: ICLR 2017
Keywords: Deep learning
Unsupervised Learning
Reinforcement Learning

Abstract: Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks.
Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments show that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can significantly boost the learning performance uniformly across a wide range of downstream tasks.

TL;DR: We propose a framework for learning a diverse set of skills using stochastic neural networks with minimum supervision, and utilize these skills in a hierarchical architecture to solve challenging tasks with sparse rewards

Review discussion:
The setting in this paper is very interesting but I would have liked the authors to discuss and report failure models. Especially as the richness of behaviors gets more complex, I expect this approach to have diversity issues with the skills learnt. Did the authors try the approach with an agent more complicated than the 3-link swimmer? I guess part of what I am wondering is whether the idea in this paper has been pushed to its limits. If it has, it would be good to know where it fails.
Also looking at Sec 5.3 -- " let X be a random variable denoting the grid in which the agent is currently situated" -- is the space discretized? And if so why and what happens if it isn't.
We have been able to scale the setup to more complex agents.
For the sake of including figures, please refer to this short report:
https://docs.google.com/document/d/1Eif0KTelnk1CR4sAEhxHw4bg3OqUhflkkAkmkpXVaZs/edit?usp=sharing
These results will be integrated as an appendix to the current paper.
It is an interesting step toward hierarchical RL. Comments/questions:
- Hierarchical REPS [C Daniel, G Neumann, J Peters, 2012] imposes a hard constraint on the entropy of conditional of option given state and action to encourage identifiability of skill/”option”, which is similar to your information-theoretic regularization. It is reasonable to discuss the differences.
- The rewards for pre-training the SNN policy network are called intrinsic rewards/unsupervised rewards, but they seem hand-specified rewards specific to domains and expected downstream tasks. It is helpful if you cite the paper which defines intrinsics reward as you do when you give examples.
- It would be good to see other examples of hierarchical RL besides locomotion + navigation. In locomotion + navigation, reward functions are easy to define for each stage, and most of the benefits seem to rely on the knowledge that if the agent has learned basic locomotion, any future navigation-related tasks will benefit.
- Hierarchical REPS: Their setup is very interesting but they only try with options that last one time-step and with environments considerably simpler than the ones considered in our paper. It will be interesting to see the scalability of their method. We now discuss the relationship in the paper, Related work Section 2).
- Indeed we use the word “intrinsic” in a fairly general sense as we do use a specific subset of the state-space to give the motivation.
Nevertheless, we disagree about this being “hand-specified rewards specific to domains and expected downstream tasks”. The pre-training rewards used  are very mild modifications of the standard rewards given to these agents for encouraging locomotion, as can be seen in OpenAI Gym[https://gym.openai.com/envs#mujoco]. No tailored modifications for navigation in mazes, use of the sensors (adding 20 dimensions to the observations) or knowing the value of different parts of the downstream environments is used. As mentioned in our work, having access to the robot (the common sub-MDP of all downstream tasks) to pre-train locomotion is an assumption, and we believe our pre-train reward to be the least hand-specified among current/past work for such high-dimensional tasks.
Nevertheless, we agree that it is less “intrinsic” than other intrinsic rewards seen in the literature (prediction-error based, full-state count, … ). If the reviewer opposes to the use of the term “intrinsic”, we can use the term “unsupervised” reward, given that this is the desired feature: be able to perform the pre-training and training steps without any expert demonstrations or highly instrumented setups.
- The same setup could be extended to manipulation tasks. The same kind of simple reward based on motion (speed or displacement) could be used, but related to the objects instead. This is a promising future work.
But indeed, our approach does rely on the “prior knowledge” that the robot has to move itself or something to somewhere.
In figure 1, is it correct that (1) concatenation means based on categorical value, you have different bias to the first hidden layer, and (2) bilinear integration means you have different weight matrix connecting observation to the first hidden layer?
Indeed, both observations are correct. This is now mentioned when describing the architectures and referring to Figure 1. Thanks for the insight!
Sorry for some last minute questions. What is the intuition for why the strong baseline in section 7.3 perform very poorly and can't solve the task? Assuming "CoM maze reward"is this baseline. Does the baseline also take into account the extra experience pre-trained policies have got? What is multi-policy, as compared to snn?
Question (adding index to questions): Clarification on experimental results
Comment: Sorry for some last minute questions. (a) What is the intuition for why the strong baseline in section 7.3 perform very poorly and can't solve the task? (b) Assuming "CoM maze reward"is this baseline. (c) Does the baseline also take into account the extra experience pre-trained policies have got? (d) What is multi-policy, as compared to snn?
Answer:
(b) “CoM maze reward” is the baseline of having the same intrinsic motivation that the agent had in the pre-train task, but directly  in the downstream tasks (without any previous pre-training). It is in the spirit of the state-of-the-art methods that try to solve sparse tasks by guiding exploration with intrinsic rewards (Schmidhuber, 1991; 2010; Houthooft et al., 2016; Bellemare et al., 2016).
(c) Therefore this baseline does not have any pre-training experience, as all other curves have.
Nevertheless, the pre-training sampling cost (10 million time-steps for our swimmer SNNs) is only a fraction of the sampling cost of any of the downstream tasks (200 millions for Gather, 14 to 25 millions for Maze).
Furthermore, the pre-training sampling cost is amortized among all downstream tasks as it is only done once.
(a) The intuition of why the “CoM reward” performs quite poorly in Mazes is simply because of the lack of hierarchy. Only with the intrinsic motivation it is hard to reach the goal and long time-horizons are needed. This furthermore increases the problem of credit assignment. Note also that the reward signal is still weak as it encourages to move, but not necessarily towards the goal. For the Gather task, some extra discussion can be found in Appendix B.
(d) The multi-policy is the hierarchical architecture that uses the one-hot vector -sampled from the categorical distribution parametrized by the manager neural network- to select which of the K policies to use during the following aggregated time steps (500 for the mazes, 10 for Gather). Note that it used 6 times more training data and has more parameters than our SNN architecture. Its description has been improved in Section 7.2.
I like the setting presented in this paper but I have several criticism/questions:
(1) What are the failure model of this work? As richness of behaviors get complex, I expected this approach to have issues with the diversity of skills that could be discovered.
(2) Looking at Sec 5.3 -- " let X be a random variable denoting the grid in which the agent is currently situated" -- is the space discretized? And if so why and what happens if it isn't.
(3) Expanding on the first point, does the approach work with more complicated embodiment? Say a 5-link swimmer instead of 2? I think this is important to assess the generality of this approach
(4) Authors claim that "Recently, Heess et al. (2016) have independently proposed to learn a range of skills in a pre-training environment that will be useful for the downstream tasks, which is similar to our framework.
However, their pre-training setup requires a set of goals to be specified.
In comparison, we use intrinsic rewards as the only signal to the agent during the pre-training phase, the construction of which only requires very minimal domain knowledge."
I don't entirely agree with this. The rewards that this paper proposes are also quite hand-crafted and specific to a seemingly limited set of control tasks.
We are grateful to the reviewer as these questions and criticisms have helped us improve the clarity of our paper. We detail here the modifications:
(1) We have added Appendix D section on failure modes. This includes new experiments with the “Ant” robot (Duan et al. 2016), which can very explicitly fail by falling over.
→ New Appendix D studies limits of our approach for unstable robots
(2) We updated our explanations on space discretization in Section 5.3: "We further partition the x-y coordinate space into discrete grids, and map the continuous-valued coordinate into the closest grid. This way, calculation of empirical mutual information only requires maintaining visitation counts of the latent variables in each grid.”
We hope this clarifies that the discretization of the space is solely done to make the computation of the estimated MI more efficient.
In particular, the discretization does not affect the MDP, which is fully continuous in actions and states. It is possible to avoid discretizing the space, in which case the MI estimation would require fitting a regressor at every iteration.
→ Section 5.3 updated
(3) We have tried more complicated embodiments.
In particular, we have followed the suggestion of the reviewer and tested our approach on a 5-link swimmer (“Snake”).
→ We report extensive experiments in Appendix C: results are as strong as for 3-link swimmer (“Swimmer”).
(4) We agree that the reward proposed in this paper is particularly suited for locomotion tasks. This still covers a large variety of control tasks, the hardest ones proposed in the benchmark paper by Duan et al. (2016).
Furthermore, although designed to help on future navigation problems, our pre-train task is very easy to set-up and satisfies our problems statement of not having direct access to any specific information of the downstream tasks (sensor readings, wall physics, goals,…).
*Edited the score 6->7.
The paper presents a method for hierarchical RL using stochastic neural networks. The paper has introduced using information-theoretic measure of option identifiability as an additional reward for learning a diverse mixture of sub-policies. One nice result in the paper is the comparison with strong baseline which directly combines the intrinsic rewards with sparse rewards and shows that this supposedly smooth reward can’t solve tasks.
Besides the argument made from the authors on difficulty on long-term credit assignment/benefits from hierarchical abstraction, one possible explanation for this might be the diversity requirement imposed in sub-policy training, which is assumed to be off in the baseline case. Wonder if this can shed insights into improving the baseline and proposing new end-to-end hierarchical policy learning as hierarchical REPS/option-critic etc. papers do. Nice visualizations.
The paper presents a promising direction, and it may be strengthened further by possibly addressing some of the following points.
1) Limited diversification of sub-policies: Both concatenation and bilinear integration allow only minimal differentiations in sub-policies through first hidden weight, which is not a problem in the tested tasks because they essentially require same locomotion policies with minimal diversification, but such limitation can be more obvious in other tasks where ideal sub-policies are more diverse.
Thus it is interesting to see it apply on harder, non-locomotion domains, where ideal sub-policies are not that similar, e.g. for manipulation, solving some task from one state can be very different from solving it from another state.
2) Limitation on hierarchical policies: Manager network is trained while the sub-policies are fixed.
Furthermore, the time steps for sub-policies are fixed. This requires “intrinsic” rewards and their learned sub-policies to be very good for solving down-stream tasks. It would be nice to see some more discussions/results on handling such cases, ideally connecting to end-to-end hierarchical policy learning.
3) Intrinsic/unsupervised rewards seem domain-specific/supervised rewards: Because of (2), this seems unavoidable.
We thank the reviewer for the very valuable feedback and we discuss in the following our point of view on the insights provided, as well as the improvements performed on our paper.
Indeed we think that our baseline of having the intrinsic reward in the sparse environments (Maze and Gather) might be hindered by not having a diversity encouragement, hence reducing the exploration (this insight has been added to Section 7.3).
It is a challenging open direction how to use diversity encouragement to improve performance in tasks with sparse rewards. Both the Hierarchical REPS framework (already cited in our related work) and the Option-critic one (now added) are very interesting on the option discovery side, but none shows a clear hierarchical use of the learned skills to improve learning across multiple tasks.
On the other hand our two-phases algorithm manages to both unsupervisedly discover a variety of skills, and use them efficiently to improve learning in a collection of downstream sparse environments.
Also, as noted in point (2) below, our method could be made end-to-end thanks to recent work on straight-through gradient estimations of stochastic computation graphs with discrete latents.
→ Related work updated with Option-critic
As for the specific points that were brought up:
(1) Apart from going in different directions, the gaits observed under different sub-policies do look quite diverse, as can be observed in the new experiments and animations in Appendix D with a more complex robot (“Ant”, described in Duan et al. 2016). We have not yet reached the limit of how diverse the sub-policies can be by just changing the first hidden weights of the architecture, and we are excited to try our approach on manipulation tasks given that the concept extends straightforwardly.
Nevertheless, there is no established benchmark on such tasks, so it will require a better study of what are the hard problems that would benefit from skills hierarchy.
→ Appendix D.1 and videos added on skill diversity of “Ant”
(2) We agree that our current work is limited by having fixed sub-policies during the training of the downstream tasks.
Nevertheless, we have added a discussion in  the Future work Section about how to use the very recent work (also ICLR submissions) on the Gumbel soft-max trick to obtain gradient estimations of stochastic computation graphs with discrete latents, like our Manager-NN + SNN.
Therefore, end-to-end training of our architecture is possible, getting rid of the fixed sub-policy limitation.
As for the fixed time steps for sub-policies, we show in the new Appendix C that, for the kind of task we consider, it is not a critical issue and not a hard hyper-parameter to tune. To go towards more dynamic tasks that require fast and precise reactions to changes in the environment, this might become a stronger limitation, but it should not be too complicated to learn a stopping policy concurrently with the Manager policy that dictates when to switch actions. We leave these very promising lines of work for future investigation.
→ Future work Section updated with end-to-end directions.
→ Switch time analysis added in Appendix C (effect of the fixed number of time steps per sub-policy on learning performance)
(3) As discussed in more depth in the reply to Reviewer3, we agree that the intrinsic/unsupervised/proxy reward is indeed domain-specific.
But the reward proposed is very easy to set up and is solely based on the robot and not the specifications of the hard downstream tasks. If what seems inappropriate is the terminology, as we wrote in response to Reviewer3, we believe switching our terminology to Proxy Reward could be the solution: in the initial training phase the agent does not have access to the more complex final environments, hence can’t train directly against its final goals.
Instead the agent acts in a simpler environment and is provided with a “proxy reward function” catered towards letting it prepare for the more complex final environments.  What do you think about the “proxy” terminology?
Thank you for updating with additional experiments. "Proxy" does seem more appropriate.
Interesting work on hierarchical control, similar to the work of Heess et al.
Experiments are strong and manage to complete benchmarks that previous work could not. Analysis of the experiments is a bit on the weaker side.
(1) Like other reviewers, I find the use of the term ‘intrinsic’ motivation somewhat inappropriate (mostly because of its current meaning in RL).  Pre-training robots with locomotion by rewarding speed (or rewarding grasping for a manipulating arm) is very geared towards the tasks they will later accomplish. The pre-training tasks from Heess et al., while not identical, are similar.
(2) The Mutual Information regularization is elegant and works generally well, but does not seem to help in the more complex mazes 1,2 and 3. The authors note this - is there any interpretation or analysis for this result?
(3) The factorization between S_agent and S_rest should be clearly detailed in the paper. Duan et al specify S_agent, but for replicability, S_rest should be clearly specified as well - did I miss it?
(4) It would be interesting to provide some analysis of the switching behavior of the agent. More generally, some further analysis of the policies (failure modes, effects of switching time on performance) would have been welcome.
We thank the reviewer for the positive feedback. Based on the comments, we have improved the experiment description and analysis, as well as included in the new Appendix C a further study on the switching behavior of the agent.
Next we treat point by point the specific requests, starting with a terminology discussion we would appreciate feedback on:
(1) We agree that the use of “intrinsic” in our paper stretches the usual use of the term in RL, so we would love to discuss which is the most adequate terminology: In our work we demonstrate that, with a very simple speed reward and the use of the SNN architecture, we can learn useful skills for downstream tasks. We called this simple reward intrinsic as, although inspired by the navigation tasks that might come, it has no notion of the actual goal in there, and it can be understood as encouraging the robot to just “move for the sake of moving”.
Another possible term could be “proxy reward” -- as, in absence of the final environments during the initial training phase, the agent cannot directly train against the ultimate rewards in those final environments, hence is given “proxy rewards” in the meantime.
→ What do you think about using the “proxy” terminology instead?
(2) The MI bonus is a powerful tool to consistently obtain a wide range of skills while training the SNN.
Nevertheless, the Maze environment offers a way of turning that does not really require a different skill than going forward: the robot can use the wall to push itself and re-orient towards the goal. This explains why in some cases (like mazes 1, 2 and 3) the hierarchy given by the methods without MI bonus might be good enough.
→ Further analysis on the observed solutions has been added to section 7.3.
(3) Both the Maze (the one denoted by 0) and the Gather task are fully described in Duan et al. (2016), but they describe them as a monolithic task. In our work we just propose a natural break down of the state-space of these hard hierarchical tasks into S_agent (the robot) and S_rest (the walls, the sensor readings, the goals,…). Actually, the swimmer locomotion task described in Duan et al. (2016) corresponds exactly to our pretrain task, as we also solely reward speed in a plain environment.
→ The Section 6 has been updated clarifying this point.
(4.1) The switching behavior of the agent is an interesting topic to which we have devoted more attention in the Appendix C:
The switching time (“skill commitment length” in the old version of the paper, now “switch time T” has been adopted) effects on the learning performance are studied for a new robot, Snake, a higher dimensional version of Swimmer.  We show that for the tasks considered, the switching time barely affects performance! This is due to the tasks being basically static: none requires fast and precise reactions to changes in the environment.
Therefore, the robot will just use the pre-learned skills in a different way.
→ Appendix C.3 has been added
(4.2) Failure modes are studied with  Ant, a more challenging robot with 27-dimensional observations and 8-dimensional actions. This robot is unstable in the sense that it might fall over and cannot recover. We show that despite being able to learn well differentiated skills with SNNs, switching between skills is a hard task because of the instability. This reveals a potential limitation of our method when applied without considering this issue. We also point at several future directions that should alleviate the problem.
→ Appendix D (with Ant failure modes) has been added
→ The Discussion and future work section now includes these points
Here we summarize the updates performed on the paper (uploaded):
→ Use term “Proxy reward” to refer to the reward in our pre-train environment.
→ Videos added on: skill diversity of “Ant”, and “Snake” solving Maze 0 and Gather. (see https://goo.gl/5wp4VP )
→ Sec. 2 (Related work): include Option-critic comparison.
→ Sec. 5.3 (Information-theoretic regularization): clarify space discretization to compute the MI bonus.
→ Sec. 6 (Experiments): clarifying agent/rest MDP factorization in the benchmark tasks for replicability.
→ Sec. 7.3. (Results on Maze and Gather): further analysis on the observed solutions. See also new Appendix C.4.
→ Sec. 8 (Discussion and Future work): include end-to-end directions as well as limitations for unstable robots.
→ Appendix C.1-C.2: report extensive experiments for more complex robot (5-link swimmer “Snake”): results are as strong as for 3-link swimmer (“Swimmer”).
→ Appendix C.3: Switch time analysis shows the mild effect of the fixed number of time steps per sub-policy on learning performance.
→ Appendix D studies limits of our approach for unstable robots ("Ant").
We thank all our reviewers for their great feedback that has made these improvements possible. We hope to have clarified and resolved most of the critical points.
Code will be released upon publication.

Meta-review:
role: PC
data: comment: The authors provide an approach to policy learning skills via a stochastic neural network representation. The overall idea seems novel, and the reviewers are in agreement that the method provides an interesting take on skill learning. The additional visualization presented in response to the reviewer comments were also quite useful.
 
 Pros:
 + Conceptually nice approach to skill learning in RL
 + Strong results on benchmark tasks
 
 Cons:
 - The overall clarity of the paper could still be improved: the actual algorithmic section seems to present the methods still at a very high level, though code release may also help with this (it would not be at all trivial to re-implement the method just given the paper).
We hope the authors can address these outstanding issues in the camera ready version of their paper.
role: Author
data: value: comment: The code has been release at https://github.com/florensacc/snn4hrl

Furthermore, the method section has been revised, and now includes more details to easily re-implement the algorithm. In particular, we have added an algorithm pseudo-code and further detailed the MI computation.

Decision: Accept (Poster)