Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel method to learn vision feature as intermediate rewards to guide the robot training in the real world. Since there are only a few sequences of human demonstrations, the paper first segments the sequences into fragments so that the features are roughly invariant on the corresponding fragments across sequences, then clusters and finds most discriminative features on those fragments, and uses them as the reward function. The features are from pre-trained deep models.

The idea is simple and seems quite effective in picking the right reward functions. Fig. 6 is a good comparison (although it could be better with error bars). However, some baselines are not strong, in particular vision related baselines. For example, the random reward ("simply outputs true or false") in Tbl. 2 seems quite arbitrary and may not serve as a good baseline (but its performance is still not that bad, surprisingly.). A better baseline would be to use random/simpler feature extraction on the image, e.g., binning features and simply picking the most frequent ones, which might not be as discriminative as the proposed feature. I wonder whether a simpler vision-based approach would lead to a similarly performed reward function. If so, then these delicate steps (segment, etc) altogether.  
review_title: Nice idea but more baselines are needed.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper explores a simple approach to learning reward functions for reinforcement learning from visual observations of expert trajectories for cases were only little training data is available. To obtain descriptive rewards even under such challenging conditions the method re-uses a pre-trained neural network as feature extractor (this is similar to a large body of work on task transfer with neural nets in the area of computer vision) and represents the reward function as a weighted distance to features for automatically extracted "key-frames" of the provided expert trajectories.

The paper is well written and explains all involved concepts clearly while also embedding the presented approach in the literature on inverse reinforcement learning (IRL). The resulting algorithm is appealing due to its simplicity and could prove useful for many real world robotic applications. I have three main issues with the paper in its current form, if these can be addressed I believe the paper would be significantly strengthened:
1) Although the recursive splitting approach for extracting the "key-frames" seems reasonable and the feature selection is well motivated I am missing two baselines in the experiments:
   - what happens if the feature selection is disabled and the distance between all features is used ? will this immediately break the procedure ? If not, what is the trade-off here ? 
   - an even simpler baseline than what is proposed in the paper would be the following procedure: simply use all frames of the recorded trajectories, calculate the distance to them in feature space and weights them according to their time as in the approach proposed in the paper. How well would that work ?
2) I understand the desire to combine the extracted reward function with a simple RL method but believe the used simple controller could potentially introduce a significant bias in the experiments since it requires initialization from an expert trajectory. As a direct consequence of this initialization the RL procedure is already started close to a good solution and the extracted reward function is potentially only queried in a small region around what was observed in the initial set of images (perhaps with the exception of the human demonstrations). Without an additional experiment it is thus unclear how well the presented approach will work in combination with other RL methods for training the controller.
3) I understand that the low number of available images excludes training a deep neural net directly for the task at hand but one has to wonder how other baselines would do. What happens if one uses a random projection of the images to form a feature vector? How well would a distance measure using raw images (e.g. L2 norm of image differences) or a distance measure based on the first principal components work? It seems that occlusions etc. would exclude them from working well but without empirical evidence it is hard to confirm this.

Minor issues:
- Page 1: "make use of ideas about imitation" reads a bit awkwardly
- Page 3: "We use the Inception network pre-trained ImageNet" -> pre-trained for ImageNet classification
- Page 4: the definition of the transition function for the stochastic case seems broken
- Page 6: "efficient enough to evaluate" a bit strangely written sentence

Additional comments rather than real issues:
- The paper is mainly of empirical nature, little actual learning is performed to obtain the reward function and no theoretical advances are needed. This is not necessarily bad but makes the empirical evaluation all the more important. 
- While I liked the clear exposition the approach -- in the end -- boils down to computing quadratic distances to features of pre-extracted "key-frames", it is nice that you make a connection to standard IRL approaches in Section 2.1 but one could argue that this derivation is not strictly necessary.

review_title: Well writen paper on a simple idea for extracting reward functions in real-world scenarios. Is missing some baselines.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper tries to present a first step towards solving the difficult problem of "learning from limited number of demonstrations". The paper tries to present 3 contributions towards this effort:
1. unsupervised segmentation of videos to identify intermediate steps in a process
2. define reward function based on feature selection for each sub-task

Pros:
+ The paper is a first attempt to solve a very challenging problem, where a robot is taught real-world tasks with very few visual demonstrations and without further retraining.
+ The method is well motivated and tries to transfer the priors learned from object classification task (through deep network features) to address the problem of limited training examples.
+ As demonstrated in Fig. 3, the reward functions could be more interpretable and correlate with transitions between subtasks.
+ Breaking a video into subtasks helps a video demonstration-based method achieve comparable performance with a method which requires full instrumentation for complex real-world tasks like door opening.

Cons:
1. Unsupervised video segmentation can serve as a good starting point to identify subtasks. However, there are multiple prior works in this domain which need to be referenced and compared with. Particularly, video shot detection and shot segmentation works try to identify abrupt change in video to break it into visually diverse shots. These methods could be easily augmented with CNN-features.
(Note that there are multiple papers in this domain, eg. refer to survey in Yuan et al. Trans. on Circuits and Systems for video tech. 2007)

2. The authors claim that they did not find it necessary to identify commonalities across demonstrations. This limits the scope of the problem drastically and requires the demonstrations to follow very specific set of constraints. Again, it is to be noted that there is past literature (video co-segmentation, eg. Tang et al. ECCV'14) which uses these commonalities to perform unsupervised video segmentation.

3. The unsupervised temporal video segmentation approach in the paper is only compared to a very simple random baseline for a few sample videos. However, given the large amount of literature in this domain, it is difficult to judge the novelty and significance of the proposed approach from these experiments.

4. The authors hypothesize that "sparse independent features exists which can discriminate a wide range of unseen inputs" and encode this intuition through the feature selection strategy. Again, the validity of the hypothesis is not experimentally well demonstrated. For instance, comparison to a simple linear classifier for subtasks would have been useful.

Overall, the paper presents a simple approach based on the idea that recognizing sub-goals in an unsupervised fashion would help in learning from few visual demonstrations. This is well motivated as a first-step towards a difficult task. However, the methods and claims presented in the paper need to be analyzed and compared with better baselines.
review_title: Simple well motivated approach, but requires better references and comparisons to existing methods

final_score: rating: null
review_content: See the review for main questions, no minor questions came to mind.
review_title: See review

final_score: rating: null
review_content: 1. Some clarifications of Algo. 1: 
- Should Algo. 1 be returning both min_std and min_split at the end?
- In the line avg_std <- .. , isn't the model double counting the std from previous runs as well? Also is prev_std + std1 + std2 set addition or value addition?
- Is there a dynamic prog. variant of optimally implementing the same method? What is the current complexity of this method in terms of video frames?

2. What is the advantage of the feature selection method in Sec. 2.2. Why not train a simple linear classifier for each subgoal and use the score of the classifier to define reward functions?
review_title: Clarifications of unsupervised video segmentation algorithm

final_score: rating: null
review_content: It would be great if the authors could provide more thorough ablation analysis on the parameter chosen (e.g, number of splits, number of discriminative features selected, mixing coefficients alpha in Eqn. 1). It would be great if the authors could provide with the reasoning process how they are chosen. 

In Section 2.2, it is not clear where exactly the features come from, though "mid to high-level features" are mentioned. Ideally, these features, if visualized at the original images, should be concentrated at the key locations (e.g., door handle), which will make the approach more convincing.  

Finally, could the authors show failure cases with learned features, e.g., when the image is misclassified, leading to over-pessimistic/over-optimistic rewards. 
review_title: More explanation and ablation analysis.

Meta-review: Quality, Clarity:
 
 The work is well motivated and clearly written -- no issues there.
 
 Originality, Significance:
 
 The idea is simple and well motivated, i.e., the learning of reward functions based on feature selection from identified subtasks in videos.
 
 pros:
 - the problem is difficult and relevant: good solutions would have impact
 
 cons:
 - the benefit with respect to other baselines for various choices, although the latest version does contain updated baselines
 - the influence of the initial controller on the results
 - the work may gain better appreciation at a robotics conference
 
 I am very much on the fence for this paper.
 It straddles a number of recent advances in video segmentation, robotics, and RL, which makes the specific technical contributions harder to identify. I do think that a robotics conference would be appreciative of the work, but better learning of reward functions is surely a bottleneck and therefore of interest to ICLR.
 Given the lukewarm support for this paper by reviewers, the PCs decided not to accept the paper, but invite the authors to present it in the workshop track.
Decision: Invite to Workshop Track