Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The submission tackles an important problem of learning and transferring multiple motor skills. The approach relies on using an embedding space defined by latent variables and entropy-regularized policy gradient / variational inference formulation that encourages diversity and identifiability in latent space.

The exposition is clear and the method is well-motivated. I see no issues with the mathematical correctness of the claims made in the paper. The experimental results are both instructive of how the algorithm operates (in the particle example), and contain impressive robotic results. I appreciated the experiments that investigated cases where true number of tasks and the parameter T differ, showing that the approach is robust to choice of T.

The submission focuses particularly on discrete tasks and learning to sequence discrete tasks (as training requires a one-hot task ID input). I would like a bit of discussion on whether parameterized skills (that have continuous space of target location, or environment parameters, for example) can be supported in the current formulation, and what would be necessary if not.

Overall, I believe this is in interesting piece of work at a fruitful intersection of reinforcement learning and variational inference, and I believe would be of interest to ICLR community.
review_title: Interesting work at the intersection of reinforcement learning and variational inference

final_score: rating: 7: Good paper, accept
review_content: In this paper, (previous states, action) pairs and task ids are embedded into the same latent space with the goal of generalizing and sharing across skill variations. Once the embedding space is learned, policies can be modified by passing in sampled or learned embeddings.

Novelty and Significance: To my knowledge, using a variational approach to embedding robot skills is novel. Significantly, the embedding is learned from off-policy trajectories, indicating feasibility on a real-world setting. The manipulation experiments show nice results on non-trivial tasks. However, no comparisons are shown against prior work in multitask or transfer learning. Additionally, the tasks used to train the embedding space were tailored exactly to the target task, making it unclear that this method will work generally.

Questions:
- I am not sure how to interpret Figure 3. Do you use Bernoulli in the experiments?
- How many task IDs are used for each experiment? 2?
- Are the manipulation experiments learned with the off-policy variant?
- Figure 4b needs the colors to be labeled. Video clips of the samples would be a plus.
- (Major) For the experiments, only exactly the useful set of tasks is used to train the embedding. What happens if a single latent space is learned from all the tasks, and Spring-wall, L-wall, and Rail-push are each learned from the same embedding. 

I find the method to be theoretically interesting and valuable to the learning community. However, the experiments are not entirely convincing.

review_title:  I find the method to be theoretically interesting and valuable to the learning community. However, the experiments are not entirely convincing.

final_score: rating: 7: Good paper, accept
review_content: The paper presents a new approach for hierarchical reinforcement learning which aims at learning a versatile set of skills. The paper uses a variational bound for entropy regularized RL to learn a versatile latent space which represents the skill to execute. The variational bound is used to diversify the learned skills as well as to make the skills identifyable from their state trajectories. The algorithm is tested on a simple point mass task and on simulated robot manipulation tasks.

This is a very intersting paper which is also very well written. I like the presented approach of learning the skill embeddings using the variational lower bound. It represents one of the most principled approches for hierarchical RL. 

Pros: 
- Interesting new approach for hiearchical reinforcement learning that focuses on skill versatility
- The variational lower bound is one of the most principled formulations for hierarchical RL that I have seen so far
- The results are convincing

Cons:
- More comparisons against other DRL algorithms such as TRPO and PPO would be useful

Summary: This is an interesting deep reinforcement learning paper that introduces a new principled framework for learning versatile skills. This is a good paper.

More comments:
- There are several papers that focus on learning versatile skills in the context of movement primitive libraries, see [1],[2],[3]. These papers should be discussed.

[1] Daniel, C.; Neumann, G.; Kroemer, O.; Peters, J. (2016). Hierarchical Relative Entropy Policy Search, Journal of Machine Learning Research (JMLR),
[2] End, F.; Akrour, R.; Peters, J.; Neumann, G. (2017). Layered Direct Policy Search for Learning Hierarchical Skills, Proceedings of the International Conference on Robotics and Automation (ICRA).
[3] Gabriel, A.; Akrour, R.; Peters, J.; Neumann, G. (2017). Empowered Skills, Proceedings of the International Conference on Robotics and Automation (ICRA).

review_title: This is an interesting deep reinforcement learning paper that introduces a new principled framework for learning versatile skills. This is a good paper.

Meta-review: This is a paper introducing a hierarchical RL method which incorporates the learning of a latent space, which enables the sharing of learned skills.

The reviewers unanimously rate this as a good paper. They suggest that it can be further improved by demonstrating the effectiveness through more experiments, especially since this is a rather generic framework. To some extent, the authors have addressed this concern in the rebuttal.

Decision: Accept (Poster)