Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a latent variable model to perform imitation learning. The authors propose the model in the control-as-inference framework and introduce two additional latent variables: one that represents a latent state (z) and another that represents a latent action (h). For the generative model, the authors use a sequence latent variable model. For inferring the latent action, the authors use a particle filter. For inferring the states, the authors use an "Adaptive path-integral autoencoder," though it was unclear where the controls "u" come from. (I assume u is the same as the actions, at which point inferring the states amounts to rollout the policy in the sequence latent variable model). The authors compare to not having the latent states and/or not having the latent actions, and demonstrate that they get better imitation learning scores.

Overall, I found the paper difficult to follow and some of the reasoning a bit unclear. The experiments seemed limited in scope, given that the authors discuss reinforcement learning in general, but only provide results on the reconstruction error when doing imitation learning. It is also unclear to me whether the gains from the experiments are from their model, or from the fact that their model probably has more parameters since it has more components. It would be good for the authors to compare to existing work that uses sequential latent variables models for deep RL, such as [1,2,3]

More detailed comments:

It would be good for the authors to substantiate statements like, "Since training sas? is just a simple supervised learning problem, it had the lowest reconstruction error but the computed action from such the internal model couldn’t make the humanoid walk." with plots.

The statement, "zaz' also failed to let the robot walk, because reasoning of the high-dimensional action can’t be accurate enough." seems similarly unjustified. If the authors wanted to test this, they could train a zaz' model with some low-dimensional action (e.g. left, straight right) and verify that this works.

While the authors state that "h can be interpreted as high-level commands," but if it is inferred at every time step, why is this a "high-level" command?

Nit-picks:
- "The procedure consists of outer internal" --> "The procedure consists of *an* outer internal"
- "via via"
- "Such the sophisticated separation was"

[1] Danijar Hafner et al. Learning Latent Dynamics for Planning from Pixels.
[2] Maximilian Igl et al. Deep Variational Reinforcement Learning for POMDPs.
[3] Alex Lee at al. Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper presents a framework for learning hierarchical policies using a latent variable conditioned policy operating at the low level, with model based planning at the high level. Unlike prior work which does hierarchical reinforcement learning, the key technical contribution of this work is that they use planning with a latent dynamics model as their high level policy. They demonstrate the method on a humanoid walking task in the DeepMimic [1] environment.

While the idea is well motivated, this paper should be rejected, primarily due to a lack of experimental results. In particular, the experiments (1) are missing several critical details about the experimental setup, (2) the experimental setup differs significantly from the claims of self-supervision and multi-task RL made in the introduction/method, and (3) there is no comparison to any prior work. Without these, it is impossible to determine if any of the claims made about the proposed method are empirically true.

First, no information is provided about the reward function used, the horizon of the tasks, or about the planning parameters or policy learning parameters. As currently stated, I don't think any of the results in the paper could be reproduced. Furthermore, the reward function and task horizon used are necessary to determine the difficulty of the proposed tasks. 

Second, the title and method section would imply that the method is self-supervised, specifically in how the latent dynamics model is learned. While the samples used to train the latent dynamics model are taken from the agent's experience, the latent conditioned policy is trained (1) with ground truth task reward, and (2) is actually pre-trained on demonstrations of the tasks with ground truth skill labels. This suggests that much of the actual skill learning is done offline in this pre-training stage - with full supervision. As a result, this would make the learning of the latent dynamics model much easier, since the sub-policies have already converged to different behaviors for different values of h. Without this pre-training, learning the low level policies and the LVM jointly would be much more challenging. Hence it seems that the "self-supervised" learning of the LVM is actually heavily dependent on the full supervision used in the pre-training stage. Additionally, the demonstrations used for the pre-training correspond to the same 3 tasks that the agent is later evaluated on (moving forward, left, right). So the method receives full supervision on the test tasks, so the experiments do not actually reflect generalization in multi-task RL as claimed.

Lastly, and most importantly, there are no comparisons to prior work. The only result shown is the trajectory error against 3 ablations of the proposed method. The reported numbers are error between the reference trajectory and ground truth, predicted plan and reference, and predicted plan and ground truth. First, it seems like the most important number here is the difference between the predicted plan and ground truth, for which numbers are missing for 2/3 ablations. Why is task success or reward not the reported number, and why is performance for 2/3 ablations missing? Additionally, there should be comparisons to existing work both in terms of hierarchical model free RL (for example Nachum et al [2]) and model based RL with latent dynamics models (Hafner et al [3]). The results as presented do not actually support that the proposed method performs better than existing work. The final result of the video of the agent doing a long horizon task also has no quantitative numbers, so again does not support that the proposed method is better.

Some other less significant points:
- there are typos throughout (for example "Figure 3: (a) Leanred latent model.").
- the tables and figures have very limited or no captions.
- The method section is difficult to follow and could use some figures which demonstrate the key technical contribution.
- Also from the method section it seems like the novelty is combining the latent conditioned policy learning from Haarnoja et al [4] and the latent dynamics learning/planning from Ha et al [5]. Is there an additional technical contribution beyond combining these two existing works? If so the method section should more clearly show it.
- There is a recent work (Sharma et al [6]), which also learns skill conditioned low level policies, and does model based planning in the space of skills to reach previously unseen goals. In this work the skill discovery is also totally unsupervised. The authors should add citation to this paper and clarify the differences between their work and this work.

[1] Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel van de Panne. DeepMimic: Example guided deep reinforcement learning of physics-based character skills.
[2] Ofir Nachum, Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning
[3] Hafner, D., Lillicrap, T. P., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels
[4] Tuomas Haarnoja, Kristian Hartikainen, Pieter Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning.
[5] Jung-Su Ha, Young-Jin Park, Hyeok-Joo Chae, Soon-Seo Park, and Han-Lim Choi. Adaptive path integral autoencoders: Representation learning and planning for dynamical systems.
[6] Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamicsaware unsupervised discovery of skills.


review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: *Edit: original score was a weak reject (3), updating to a weak accept (6) in light of revisions.*

This work implements a hierarchical control scheme for a high-dimensional control problem (locomotion using a humanoid body).  The hierarchy consists of a high-level module that plans in an abstract space of "intention", and the intention variables serve as inputs, along with state, to a low-level controller that actually executes the movements.  The premise is that a lower-level controller should be usable for multiple tasks, and should be able to be commanded by a lower-dimensional intention input.  I find the basic ideas presented clear, the literature reviewed reasonably well, and the motivation and setting to be very interesting.  The video summary is valuable.

My main concerns have to do with presentation, but I think they are relatively significant concerns.  As the draft currently stands, I would, somewhat regrettably, be inclined to reject the submission (marginally).  I think revisions could seriously improve this paper and incline me towards acceptance.

Algorithm 1 indicates that the learning of the low-level controller will be done jointly with the learning of the latent model and planning using the high-level, learned intention space.  In the experiment, it is indicated that the low-level controller is pretrained.  This points to a couple issues that are not clear in the draft:
(1) Presumably this pretraining is necessary and things do not work without it.  Indeed, it is hard to imagine that the movements will be well grounded to human motion capture movements without this pretraining.  Does the algorithm work as written or is pretraining a fundamentally essential step?  There are no settings, even toy settings, where the algorithm as written is shown to be effective.
(2) The authors should be clearer how they conduct the pretraining which involves learning the low-level controller.  
(3) I'm not clear how updating the low-level controller is effective in the algorithm.  While I understand why it makes sense to plan in the intention space of the pre-trained controller, and I understand why learning a model is a core part of planning, it would seem like fine-tuning the low-level controller could make the movements deviate considerably from the initial movement space and maybe even eliminate the ability of the low-level policy to express movements that are not used early in training. So essentially, while the planning in the low-D space makes sense and the learning of the model makes sense, the low-level controller update seems possibly to not make sense, and there aren't experiments showing that step helps. 

Is it just a coincidence that the intention space (h) is one-of-three and the low-d state space (z) is 3-dimensional as well?  Or are these both selected with sort of going straight vs turning left or right in mind?

The experiment section is generally very unclear, though details are made a little clearer from the video.  In the paper, there are a few points that need to be clearer: 
(1) "ref", "plan", and  "true" are not well defined and it is unclear what these distances in Table 1 refer to precisely.  Clearly introduce what each of these refers to.  The authors simply say that there are imitation tasks but do not walk through what these terms refer to. 
(2) In 4.1, the different structures are not adequately introduced.  The pointers to the figure 2 diagrams are essential, but there is no pointer for zaz', the pointers are only in the table (not in the text), there are grammar issues in the text and the text could be verbally clearer about the variants.
(3) shs' setting is a bit unclear. Basically, clarify briefly how planning is performed in this case.   Is a forward model still trained, but the model opperates with the full state space?  If so, presumably the forward model is much worse and then the planning approach is correspondingly bad, hence the poor rollouts? 
(4) 4.2 is I think obviously inadequately described in the text and I can only assume was the result of rushing for the deadline?  The second experiment is essentially not presented in the text all aside from a still image.  
(5) And for all of the experiments that rely on planning with a particle filter, details such as how reliable the filter is in generating useful control, how many samples are required, and possibly elements of compute speed would make much clearer how well the approach actually works.  Does the choice of planner matter at all?  A common, albeit relatively weak, baseline planning approach is CEM...would CEM work here?  I'd like to understand if the choice of particle filter is the author's default choice, which is fine if so, or if there is a positive assertion being made that the particle filter is particularly valuable.

I hope the authors will generally improve the exposition in the experiment section (4) during the revisions.

Overall, I find the paper well motivated in framing the problem (i.e. using model-based approaches to control the latent space of a low-level controller).  I also appreciate the scale of the problem (humanoid control is challenging, so this is not a toy problem).  I find the results a bit unclear, perhaps due to hurriedness in writing, so I find them a bit difficult to fully appreciate.  Nevertheless, the core contribution that I take away from this work is that there is a value to learning the low-dimensional state representation (z, via the LVM), relative to planning using a forward model on the full state (?...I'm still unclear on the presentation of this result, due to unclear exposition). Slightly more broadly, this is a good demonstration of using a planner jointly with a learned high-level command/intention representation, for a high-dimensional problem. 

If I've understood this correctly, I'd be reasonably interested in this result. If the authors can both clarify the core results and communicate that the choices made in the algorithm are well thought through, I would be happy to adjust my score.


Relatively minor:

Abstract says 90-dimensional humanoid system, but later it is stated "34 degrees of freedom,
197 state features, and 36 action parameters". Where the 90 dimensions comes from is unclear.  Often people refer to number of actuators or DoFs.  Please adjust this or be more explicit.

In equation 9, f() is not very clearly specified. Is f() a nonlinear function (e.g. a neural network) or is it a linear function? It seems like it might as well be a linear function, since the authors propose to learn a latent dynamics model that is nonlinearly related to the state.

Typos in Fig 3 caption.

review_title: Official Blind Review #2

Meta-review: The authors present a self-supervised framework for learning a hierarchical policy in reinforcement learning tasks that combines a high-level planner over learned latent goals with a shared low-level goal-completing control policy.  The reviewers had significant concerns about both problem positioning (w.r.t. existing work) and writing clarity, as well as the fact that all comparative experiments were ablations, rather than comparisons to prior work.  While the reviewers agreed that the authors reasonably resolved issues of clarity, there was not agreement that concerns about positioning w.r.t. prior work and experimental comparisons were sufficiently resolved.  Thus, I recommend to reject this paper at this time.
Decision: Reject