Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Paper Summary:
The paper proposes a method for learning a set of primitives for robotic movements from a dataset of demonstrations, showing a diverse set of tasks, in an unsupervised fashion. The central underlying idea is that robotic tasks can be solved by combining fundamental building blocks, the so-called "motor programs", in the right way. The described algorithm takes a demonstration and uses a transformer network to embed the trajectory into a sequence of latent variables. Then each individual latent is transformed to a 10 step trajectory for the joint space of the robot via an LSTM network. Finally the individual trajectories are concatenated and the reconstruction is compared to the original demonstration through dynamic time warping. In this structure the latent variables represent a query to a specific learned primitive, which can be accessed using the LSTM. 
In the experimental section, the paper gives mostly qualitative insight into the learned representation. The paper visualizes different movements and their corresponding latent variables projected onto two dimensions. Further, it is shown that the segmentation of demonstrations roughly compares to how a human expert would manually segment the given tasks. Finally, the paper shows that a hierarchical RL algorithm trained in the learned latent space outperforms one which works directly in the low-level control space of the robot in the sense that it learns to solve the given task much faster. 

Evaluation:
The problem of discovering primitives is approached by the authors in a novel and interesting way, however in my opinion the paper should be rejected because:
    (a) the experimental section is not convincing enough to support the claim that the method captures the shared motions across different skills. Especially, the paper misses to adequately show how these motions can be recombined and used to solve robotic tasks. 
    (b) the paper is imprecise and missing important details in both the description of the method and the experimental verification 
    (c) the paper misses important related work, which tackles the same problem.


The two main claims in the paper are:
1. The presented method learns a latent space which represents common shared motions among diverse tasks encountered in robotics
2. Robotic tasks can be solved by recombining primitives from the aformentioned space 

Although it is impossible to verify the first claim based on the paper alone, the provided webpage, which shows an animated version of Figure 3 nicely visualizes the learned latent space and shows that the representation is somewhat smooth with similar movements clustered together.

a1) Figure 5 is supposed to show that the method manages to segment given demonstrations in a meaningful way, but even after zooming into the pdf, it is impossible to see what is actually going on. A visualization in a video would be preferable.

a2) 4.1.1 and Figure 4 show that individual primitives can be executed on a real robot, however, the paper fails to show the execution of a combination of primitives. 

a3) Given that the method seems to loose the connection of movements to time it would be interesting to see whether a combination actually results in smooth, natural movement of the robot.

a4) The main quantitative assessment of the usefulness of the learned motor program network is given by the RL experiments in section 4.3. However, the baseline method seems to output one single velocity control action per evaluation of the policy (?), whereas the presented method essentially outputs an action sequence of 50 actions per policy evaluation. State-of-the-art methods commonly use frame-skipping and repeat the same action for multiple timesteps, because it makes the resulting optimization problem easier and speeds up the learning. See for example (Mnih 2013) (Mnih 2015) (Lillycrap 2015) (Hafner 2018). It would be interesting to compare against a baseline which also incorporates some form of frame-skipping and validate the speedup is not simply due to chunking of action sequences.

a5) Finally, the method outperforms the plain PPO baseline when it comes to speeding up the learning process, but the solutions found do not look like natural robot movements. You can clearly spot different primitives and transitions between individual segments look unnatural and jerky. How does the baseline solution compare in this regard?

a6) Given that the presented movements look unnatural and the fact that the paper only shows the execution of individual primitives in the rest of the paper, I simply cannot support the second claim.


Details I am missing from the paper:
b1) How is the "continuation probability" computed with the transformer network?
b2) Are the biases in section 3.2 necessary to make the method work at all?
b3) In section 4.2, how does the sequence alignment work?
b4) In section 4.2, what is the training set for the labelling task?
b5) What are the exact task parameters given to the policy in the RL task?
b6) What do you mean with, "these motor programs are executed without environment feedback"? Does the policy determine the complete sequence of programs in one step?


c1) Finally, to my knowledge the problem of learning meaningful primitives and showing that they can be combined in a different way to solve novel, unseen tasks has already been investigated in (Lioutikov 2017). Although this paper approaches the problem very differently and the dimensionality of the primitives is lower, I still consider this paper very much related to what is presented by the authors. I would suggest adding it to the related work. The paper shows that previous methods for discovering primitives from a set of different tasks exist.


In my opinion the paper in its current is probably not yet ready for publication. However, I strongly encourage the authors to address the above mentioned problems.

References:
Mnih, Volodymyr, et al. "Playing atari with deep reinforcement learning." arXiv preprint arXiv:1312.5602 (2013).
Mnih, Volodymyr, et al. "Human-level control through deep reinforcement learning." Nature 518.7540 (2015): 529.
Lillicrap, Timothy P., et al. "Continuous control with deep reinforcement learning." arXiv preprint arXiv:1509.02971 (2015).
Hafner, Danijar, et al. "Learning latent dynamics for planning from pixels." arXiv preprint arXiv:1811.04551 (2018).
Lioutikov, Rudolf, et al. "Learning movement primitive libraries through probabilistic segmentation." The International Journal of Robotics Research 36.8 (2017): 879-894.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The paper aims to learn middle-level motor task primitives from unlabeled actions. The main insight is that the decomposition of motor tasks can be learned using a set of LSTMs with a loss function that minimizes the differences between the original task and the recomposed task. They evaluate their approach on MIME dataset that includes 20 different tasks.

+ The idea of recomposition based loss function seems very useful for learning from unlabeled data. 
+ The evaluation results seem to be strong. It outperforms a supervised LSTM baseline by 4 percentage points.

- The related work is somewhat narrowly focused on the controlled program. It will be nice if the authors can describe whether such ideas have explored in other domains before. 

- It is not clear to me how much the accuracy gain in the latent representation transfer to the accuracy of the actual recomposed task. The authors presented no quantitative results to show how much the 4pp gain improves the accuracy of new tasks.  


review_title: Official Blind Review #1246

final_score: rating: 8: Accept
review_content: This work presents a novel approach to extracting reusable motor primitives from task demonstrations.  The approach taken in this work involves learning a deep encoder network which translates an arbitrary length trajectory in a robot's configuration space (is this right?) into a sequence of vectors describing different motor primitives.  A second decoder network translates these vectors into a sequence of trajectory segments.  These networks are trained to minimize the distance between the original trajectory, and the trajectory generated by encoding and reconstructing the original as a sequence of primitives and reconstructing.  An additional regularization term discourages the network from learning trivial, one step primitives.  The decoder network is also initialized by training on a set of simple trajectories generated by a robotic planning algorithm.

Experiments involved extracting motor programs from the MIME data set consisting of demonstrated trajectories for the Baxter robot.  In addition to qualitative visualizations of the learned primitives, quantitative results using a limited set of human-segmented trajectories demonstrate that the learned primitives roughly correspond to the segmentations that humans identify.  Further experiments show that reinforcement learning in the space if learned primitives is more sample efficient than RL in the low-level control space.

The work presents a novel and effective solution to the difficult task of learning reusable motor skills.  While the work focuses on robotic control, it is likely that similar approaches could be developed for more general reinforcement learning problems.  There is room for improvement.  In particular, ablations on the regularization and initialization mechanisms could help us better understand the importance of these elements in learning useful motor programs, and illustrate the robustness of this method to different methods of initialization.
review_title: Official Blind Review #1

Meta-review: The work presents a novel and effective solution to learning reusable motor skills.  The urgency of this problem and the considerable rebuttal of the authors merits publication of this paper, which is not perfect but needs community attention.
Decision: Accept (Poster)