Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper presents a weakly supervised method for segmentation of trajectories into sub-skills inspired by multi-instance learning (MIL) in image classification by Andrews et al. (2002). This is done via training a classifier to label each observation per time-step with the probability of skills corresponding to that observation. These predictions are then accumulated throughout the trajectory to compute the probability of the skill in that trajectory. There is only a trajectory level supervision provided which specifies which skills are present with no specification of the order in which they appear. They empirically show that their model can achieve decent skill level classification scores on multiple environments provided that there is a large variety of demonstrations provided.

In its current form, I would recommend this paper to be rejected because 1) the framing and motivation of the paper does not correspond to the results and experiments reported and hence seems misleading 2) the paper is limited in scope 3) further experiments and comparisons to relevant baselines are needed to support the claims made in the paper.

The problem that they are proposing is interesting and is of clear value, however, the paper falls short in addressing this problem. In particular, the paper is framed as a way to learn re-usable useful sub-skills that can help generalize to new situations in control. However, the method presented provides a per time-step labelling of each observation with the associated most likely sub-skill. Having the per time-step labelling of the trajectory provides no indication that the data could be useful for learning reusable skills for downstream tasks later. One very basic experiment could be to train a behaviour cloning (BC) agent on the observation-action pairs conditioned on the sub-skill (or a separate network per sub-skill) and show that the learned policies can be leveraged in solving the tasks presented. For instance, one can train a meta-controller that can switch between these learned sub-skills to successfully perform the task. Training such sub-skills from weakly supervised skill annotations has been successfully done by Shiarlis et al. (2018). It should be noted that in their setting, the annotations are ordered which simplifies the problem to some extent.

Ignoring the motivation and focusing only on the problem setting addressed which is annotation of trajectories with skill labels, the experiments seem very restrictive to me. To my understanding there are at most 4-6 primitive skills present in each environment investigated. Looking at the videos linked, it feels like there is some overfitting to the trajectories provided as for example in the case of the video with the red bowl (Reach and Stir inside Cup), the classifier is predicting 'Stirring' from the first frames where there is not much information present in the scene and 'reach to object' could also be plausible for example. It would have been nice to see the logits of predictions per classes for these examples to understand the confidence of the model particularly when some of the skills could be equally likely (e.g. first few frames).

I found the experimental setup unclear. Particularly, details regarding the task setup, how many demonstrations are needed per task/skill, architectural choices and hyper-parameters are lacking and makes experiments hard to follow and understand. I would have also liked to see more analysis regarding the segmented trajectories, particularly how consistent these predictions are through time. To my understanding there is nothing that keeps a skill annotation consistent over some period of time (e.g. the model could keep switching its prediction every time-step). This would be quite unsatisfactory if one would like to use this to actually segment sub-trajectories associated with a given skill that could be useful for training policies. They report applying Gaussian smoothing to filter out noisy predictions but there are no details provided on how this is tuned and how much this affects the quality of segmentations.

Overall, the paper seems to me very limited in its scope and experimental results. The claims made throughout the paper are not supported empirically or theoretically. There is not enough evidence for me to assess the significance of the proposed method and know whether this is indeed useful in practice. 

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper tackles the problem of learning to label individual timesteps of sequential data, when given labels only for the sequence as a whole. The authors take an approach derived from the multiple-instance learning (MIL) literature that involves pooling the per-timestep predictions into a sequence-level prediction and to learn the per-timestep predictions without having explicit labels. They evaluate several pooling techniques and conclude that the log-sum-exp pooling approach is superior. The learned segmentations are used to train policies for multiple control skills, and these are used to solve hierarchical control tasks where the correct skill sequence is known.

This is a good application of the MIL approach. However, I have settled on a weak reject because in my view, the novelty and results are minor.

The main point of comparison is the log-sum-exp() pooling as compared to max() and neighborhood-max() pooling. However, if I understand correctly, the log-sum-exp() approach has been used successfully in several other domains including its original domain of semantic image segmentation. So I view the novelty of the approach to be fairly low.

In addition, although the superior pooling method (which already exists in the literature) does outperform the alternatives evaluated here, the results are somewhat underwhelming, at only ~35-60% validation accuracy. How does this compare to a fully-supervised oracle method trained with per-timestep labels?

The behavioral cloning results are also fairly underwhelming, and the experiments are not very clearly described. Am I correct in my understanding that the learned skills are composed to solve a task where the correct sequence of skills is known, but is longer than the training sequences? A success rate of 50% on this task seems rather low. How does this compare, as above, to a fully-supervised oracle baseline? Why is there no success rate reported for the CCNN baseline?

I think this is a good application of weakly-supervised MIL, but I find the specific contributions to be lacking in novelty and impressiveness of results. There are several directions that I think could improve the work:
- oracle fully-supervised results, to indicate the gap between the fully- and weakly-supervised case
- more thorough baselines on the behavior task, such as Policy Sketches [1]
- perhaps the temporal aspect of the problem could be incorporated into the pooling approach more directly to produce a more novel algorithmic contribution

[1] Andreas, Jacob, Dan Klein, and Sergey Levine. "Modular multitask reinforcement learning with policy sketches." Proceedings of the 34th International Conference on Machine Learning-Volume 70. JMLR. org, 2017.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper introduces a weakly supervised learning approach for trajectory segmentation, which relies on coarse labelling about the occurrence of a skill in a demonstration to segment trajectories. This is accomplished through a recurrent model predicting skill categories for each step in a trajectory, and a trajectory level loss function that penalises the probability of seeing a given skill in a trajectory.

Overall, I like the idea of identifying skills in this manner, and think that this is an important problem to address. However, I have concerns about it's feasibility when a large number of skills are present. It seems that there are certain requirements of skill occurrences in datasets that need to be met if this approach is to be feasible. For example, consider the dataset of skill sequences:

[111 222 333]
[444 222 333]
[222 333 555]

It seems that it will never be possible to learn to identify skills 2 and 3 from this dataset. This paper would be greatly strengthened if the minimum dataset requirements to learn all skills were enumerated, or some theoretical bounds provided around when this loose labelling could possibly be successful provided.  The paper glosses over this point by suggesting that "data" makes this a non-issue, but the paper would be much stronger if these limitations were confronted and some bounds on the chances of meeting the requirements needed for learning provided. 

The classification results seem extremely poor, and it is hard to assess these on the basis of accuracy alone. For example, in the Robosuit example, the test results could potentially have been obtained by simply making the same prediction throughout the test, and there is no indication that anything sensible was actually learned. Confusion matrices, or precision and recall metrics, are required to avoid this. At present it is impossible to know if these errors are due to noisy predictions, the dataset limitations described above, or simply a poor classifier.

Along these lines, qualitative results in the form of trajectory classification (say following the presentation conventions in 

Bolanos, 15, https://arxiv.org/pdf/1505.01130.pdf 
Ranchod, 15, https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7353414&tag=1

would also help to address these concerns. 

A question regarding the dial jaco videos experiments, why does the classifier not predict a 5 when moving from 4 to 6? I would expect a very jumpy prediction here, but the prediction looks very smooth - is this a filtered result?

Finally, baseline experiments are limited to other weakly supervised learning segmentation approaches,  but I think comparisons with unsupervised clustering methods would also be useful.

Unfortunately, due to the lack of evidence that the proposed approach is able to learn effectively, I am inclined to reject this paper. Clarifying the bounds, and presenting stronger evidence (beyond accuracy) would make this paper stronger.

review_title: Official Blind Review #1

Meta-review: The authors present a multiple instance learning-based approach that uses weak supervison (of which skills appear in any given trajectory)  to automatically segment a set of skills from demonstrations.  The reviewers had significant concerns about the significance and performance of the method, as well as the metrics used for analysis.  Most notably, neither the original paper nor the rebuttal provided a sufficient justification or fix for the lack of analysis beyond accuracy scores (as opposed to confusion matrices, precision/recall, etc), which leaves the contribution and claims of the paper unclear.  Thus, I recommend rejection at this time.
Decision: Reject