Title: Unsupervised Perceptual Rewards for Imitation Learning
Authors: Pierre Sermanet
Kelvin Xu
Sergey Levine
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Unsupervised Learning
Reinforcement Learning
Transfer Learning

Abstract: Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a suitable reward function takes considerable manual engineering and often requires additional and potentially visible sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide useful feedback on these implicit intermediate steps or sub-goals.
To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify the key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit sub-goal supervision. The resulting reward functions, which are dense and smooth, can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward functions, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also demonstrate that our method can be used to learn a complex real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task.


TL;DR: Real robots learn new tasks from observing a few human demonstrations.

Review discussion:
It would be great if the authors could provide more thorough ablation analysis on the parameter chosen (e.g, number of splits, number of discriminative features selected, mixing coefficients alpha in Eqn. 1). It would be great if the authors could provide with the reasoning process how they are chosen.
In Section 2.2, it is not clear where exactly the features come from, though "mid to high-level features" are mentioned. Ideally, these features, if visualized at the original images, should be concentrated at the key locations (e.g., door handle), which will make the approach more convincing.
Finally, could the authors show failure cases with learned features, e.g., when the image is misclassified, leading to over-pessimistic/over-optimistic rewards.
Thank you for your questions.
Q: It would be great if the authors could provide more thorough ablation analysis on the parameter chosen (e.g, number of splits, number of discriminative features selected, mixing coefficients alpha in Eqn. 1). It would be great if the authors could provide with the reasoning process how they are chosen.
A: We are working on providing more analysis and will update the paper in the coming days.
Q: In Section 2.2, it is not clear where exactly the features come from, though "mid to high-level features" are mentioned. Ideally, these features, if visualized at the original images, should be concentrated at the key locations (e.g., door handle), which will make the approach more convincing.
A: We are investigating what analysis of the features we can provide and will update the paper in the coming days.
Q: Finally, could the authors show failure cases with learned features, e.g., when the image is misclassified, leading to over-pessimistic/over-optimistic rewards.
A: Failure cases were already available in the text of the original paper but we have added a dedicated section in 3.1.1 for failure cases to make it more clear.
1. Some clarifications of Algo. 1:
- Should Algo. 1 be returning both min_std and min_split at the end?
- In the line avg_std <- .. , isn't the model double counting the std from previous runs as well?
Also is prev_std + std1 + std2 set addition or value addition?
- Is there a dynamic prog. variant of optimally implementing the same method? What is the current complexity of this method in terms of video frames?
2. What is the advantage of the feature selection method in Sec. 2.2. Why not train a simple linear classifier for each subgoal and use the score of the classifier to define reward functions?
Thank you for the clarification questions, we have updated the paper to answer them.
Q: Should Algo. 1 be returning both min_std and min_split at the end?
A: The algorithm hadn't been translated into pseudo-code correctly, it is fixed now.
Q: In the line avg_std <- .. , isn't the model double counting the std from previous runs as well?
Also is prev_std + std1 + std2 set addition or value addition?
A: This line does not recompute the average standard deviation, it just averages the list of standard deviations already computed. prev_std + std1 + std2 was actually a list merging, we changed the code to reflect this using a Join() function.
Q: Is there a dynamic prog. variant of optimally implementing the same method? What is the current complexity of this method in terms of video frames?
A: Dynamic programming is not applicable because sub-problems depend on the preceding decisions. More details can be found in Section 2.2. The complexity is O(n^m) with n the number of frames and m the number of splits. We added a non-exact but efficient version of the algorithm as a greedy binary optimization, which runs in O(n^2 log m).
Q: What is the advantage of the feature selection method in Sec. 2.2. Why not train a simple linear classifier for each subgoal and use the score of the classifier to define reward functions?
A: Our initial intuition is that training a classifier with such little data would badly overfit, so we first tried to select individual features directly. We were planning to provide a comparison to a trained classifier, it is under way and we will update the paper with it in the coming days.
The paper explores a simple approach to learning reward functions for reinforcement learning from visual observations of expert trajectories for cases were only little training data is available. To obtain descriptive rewards even under such challenging conditions the method re-uses a pre-trained neural network as feature extractor (this is similar to a large body of work on task transfer with neural nets in the area of computer vision) and represents the reward function as a weighted distance to features for automatically extracted "key-frames" of the provided expert trajectories.
The paper is well written and explains all involved concepts clearly while also embedding the presented approach in the literature on inverse reinforcement learning (IRL). The resulting algorithm is appealing due to its simplicity and could prove useful for many real world robotic applications. I have three main issues with the paper in its current form, if these can be addressed I believe the paper would be significantly strengthened:
1) Although the recursive splitting approach for extracting the "key-frames" seems reasonable and the feature selection is well motivated I am missing two baselines in the experiments:
- what happens if the feature selection is disabled and the distance between all features is used ? will this immediately break the procedure ? If not, what is the trade-off here ?
- an even simpler baseline than what is proposed in the paper would be the following procedure: simply use all frames of the recorded trajectories, calculate the distance to them in feature space and weights them according to their time as in the approach proposed in the paper. How well would that work ?
2) I understand the desire to combine the extracted reward function with a simple RL method but believe the used simple controller could potentially introduce a significant bias in the experiments since it requires initialization from an expert trajectory. As a direct consequence of this initialization the RL procedure is already started close to a good solution and the extracted reward function is potentially only queried in a small region around what was observed in the initial set of images (perhaps with the exception of the human demonstrations). Without an additional experiment it is thus unclear how well the presented approach will work in combination with other RL methods for training the controller.
3) I understand that the low number of available images excludes training a deep neural net directly for the task at hand but one has to wonder how other baselines would do. What happens if one uses a random projection of the images to form a feature vector? How well would a distance measure using raw images (e.g. L2 norm of image differences) or a distance measure based on the first principal components work? It seems that occlusions etc. would exclude them from working well but without empirical evidence it is hard to confirm this.
Minor issues:
- Page 1: "make use of ideas about imitation" reads a bit awkwardly
- Page 3: "We use the Inception network pre-trained ImageNet" -> pre-trained for ImageNet classification
- Page 4: the definition of the transition function for the stochastic case seems broken
- Page 6: "efficient enough to evaluate" a bit strangely written sentence
Additional comments rather than real issues:
- The paper is mainly of empirical nature, little actual learning is performed to obtain the reward function and no theoretical advances are needed. This is not necessarily bad but makes the empirical evaluation all the more important.
- While I liked the clear exposition the approach -- in the end -- boils down to computing quadratic distances to features of pre-extracted "key-frames", it is nice that you make a connection to standard IRL approaches in Section 2.1 but one could argue that this derivation is not strictly necessary.
Thank you for your review and suggestions! We incorporated your comments in the paper and additionally address them here:
1) We added an analysis of the accuracy of feature selection by varying the number of features (n) in figure 4. The rightmost part of the graph is the equivalent of the suggested experiment to use all features, this collapses to 0% accuracy when n > 8192, and is best in the range [32, 64].
2) For fairness, the kinesthetic demonstration was necessary for comparing to the baseline system developed by Chebotar et al (2016). We would note that the real world robotics tasks that we evaluated our method on are very difficult and no known prior RL method has demonstrated door opening of this sort from scratch without such a demonstration.
3) We have added a learned linear classifier on top of the fixed pre-trained features and believe this should help address your concerns by providing a model for comparison (details of this analysis can be found in sections 2.3 and 3.1.2).
Thank you for carrying out additional experiments to test alternative baselines, they add to the overall quality of the paper.
Unfortunately, overall I still feel that the contribution of the paper at hand leaves it at the border of acceptance.
On the one hand it serves as a well written account demonstrating that reward functions can be learned even from few demonstrations.
On the other hand, the "representation learning" part of the paper is rather small and it "only" provides a two data-points (e.g. two instances in which the method worked) that are fairly similar to each other.
Ultimately I think a decision by the area chair is needed. In my opinion the paper could well be accepted to the conference but it might be better suited for a robotics conference. I personally would also welcome an open discussion about this but it seems the schedule for acceptance is rather restricted.
Regarding the rebuttal:
1) Thanks this was useful for providing an intuition and justifies the taken approach. I am still wondering how well the simple distance based weighting would do / whether one cannot design an even simpler method.
2) I of course understand this and my comment was not meant as a criticism of your choice of methods.
Rather I worry that your choice of methods and the restricted setting (two experiments with the same robot arm) might induce a bias in the evaluation.
3) Thanks, as I wrote above I appreciate this and it does address my concern.
See the review for main questions, no minor questions came to mind.
This paper proposes a novel method to learn vision feature as intermediate rewards to guide the robot training in the real world. Since there are only a few sequences of human demonstrations, the paper first segments the sequences into fragments so that the features are roughly invariant on the corresponding fragments across sequences, then clusters and finds most discriminative features on those fragments, and uses them as the reward function. The features are from pre-trained deep models.
The idea is simple and seems quite effective in picking the right reward functions. Fig. 6 is a good comparison (although it could be better with error bars).
However, some baselines are not strong, in particular vision related baselines.
For example, the random reward ("simply outputs true or false") in Tbl. 2 seems quite arbitrary and may not serve as a good baseline (but its performance is still not that bad, surprisingly.). A better baseline would be to use random/simpler feature extraction on the image, e.g., binning features and simply picking the most frequent ones, which might not be as discriminative as the proposed feature. I wonder whether a simpler vision-based approach would lead to a similarly performed reward function. If so, then these delicate steps (segment, etc) altogether.
Thank you your review and suggestions! We agree that adding more baselines for doing feature extraction would expand the experimental contribution of the paper. We have added a learned classifier baseline and a more thorough analysis comparing these alternatives in sections 2.3 and 3.1.2 which we hope helps address your concerns.
The paper tries to present a first step towards solving the difficult problem of "learning from limited number of demonstrations". The paper tries to present 3 contributions towards this effort:
1. unsupervised segmentation of videos to identify intermediate steps in a process
2. define reward function based on feature selection for each sub-task
Pros:
+ The paper is a first attempt to solve a very challenging problem, where a robot is taught real-world tasks with very few visual demonstrations and without further retraining.
+ The method is well motivated and tries to transfer the priors learned from object classification task (through deep network features) to address the problem of limited training examples.
+ As demonstrated in Fig. 3, the reward functions could be more interpretable and correlate with transitions between subtasks.
+ Breaking a video into subtasks helps a video demonstration-based method achieve comparable performance with a method which requires full instrumentation for complex real-world tasks like door opening.
Cons:
1. Unsupervised video segmentation can serve as a good starting point to identify subtasks.
However, there are multiple prior works in this domain which need to be referenced and compared with. Particularly, video shot detection and shot segmentation works try to identify abrupt change in video to break it into visually diverse shots. These methods could be easily augmented with CNN-features.
(Note that there are multiple papers in this domain, eg. refer to survey in Yuan et al. Trans. on Circuits and Systems for video tech. 2007)
2. The authors claim that they did not find it necessary to identify commonalities across demonstrations. This limits the scope of the problem drastically and requires the demonstrations to follow very specific set of constraints. Again, it is to be noted that there is past literature (video co-segmentation, eg. Tang et al. ECCV'14) which uses these commonalities to perform unsupervised video segmentation.
3. The unsupervised temporal video segmentation approach in the paper is only compared to a very simple random baseline for a few sample videos.
However, given the large amount of literature in this domain, it is difficult to judge the novelty and significance of the proposed approach from these experiments.
4. The authors hypothesize that "sparse independent features exists which can discriminate a wide range of unseen inputs" and encode this intuition through the feature selection strategy. Again, the validity of the hypothesis is not experimentally well demonstrated.
For instance, comparison to a simple linear classifier for subtasks would have been useful.
Overall, the paper presents a simple approach based on the idea that recognizing sub-goals in an unsupervised fashion would help in learning from few visual demonstrations. This is well motivated as a first-step towards a difficult task.
However, the methods and claims presented in the paper need to be analyzed and compared with better baselines.
First of all, thank you for your constructive comments and your review. We have made an effort to incorporate your remarks in the paper and also address them here:
1. and 3. These references are useful and we have added them to the paper. We would emphasize that the main contribution of the paper is a step towards learning reward functions from videos that allow a real-world robot to perform a difficult manipulation task. We started with very simple components and aim to use more sophisticated ones like the ones you suggested in future work.
But because the main contribution of our paper was demonstrating that we could learn reward functions from videos, and because performing experiments on real robots is generally quite labor intensive, we feel implementing comparisons to multiple videos segmentation baselines would be out of scope here.
2.
While our current video segmentation is simple and does not make use of commonalities across demonstrations, we absolutely agree that using a method which leverages this would yield much more robust step-discovery in less constrained setups.
In fact, the feature selection experiments were in part designed to test the existence of small but discriminative subsets of features in our pre-trained features, in order to reduce the complexity of the problem. Based on the suggestions of R5 and your own, we added a graph which shows empirically how classification performance rapidly deteriorate when all the features are used (see fig.4, Section 3.1.2).
In general, with millions of features to compare, the unsupervised search for common patterns across videos leads to a combinatorial explosion.
But if we find a small set of good candidates, then the problem becomes tractable and we can use an optimization method for video alignment such as the one described in the video co-localization work of Tang et al. ECCV14. Our experiments are a first step towards a more sophisticated and tractable unsupervised alignment. We discuss this at length in Sections 2.3 and 3.1.2.
4. Thanks for the suggestion! We have added a linear classifier for step classification and compared it with feature selection. Our initial intuition was that using all the pre-trained features for classification would not be effective, which we confirmed empirically in Section 3.1.2 (see fig.4). It turns out that our assumption that a linear classifier would simply overfit given the large number of trainable parameters and the low-data regime was untrue (thanks!). The classifier performed surprisingly well.
However the hypothesis that a small subset of very discriminative features could be also be used (our experiments show it performs comparably and has lower variance) is true and could potentially be used in future video segmentation methods which scale poorly to large dimensions as discussed above.
We would point out however, that under the modeling assumptions which we discuss in the inverse RL interpretation (see Section 2.1), our naive Bayes classifier should use all of the features. Since this is highly ineffective as we show, this motivates using a feature selection criteria (or as you suggest a learned classifier) as a way to rectify the issue. Both methods perform similarly, with the classifier being slightly better.
Lastly, we would like to emphasize that both of the feature selection/classifier build upon the unsupervised video segmentation and as such is one part of our method.
Therefore, using a classifier should be viewed overall as a variant of our method, rather than an pre-existing baseline approach to learning rewards from videos.

Meta-review:
role: PC
data: comment: Quality, Clarity:
 
 The work is well motivated and clearly written -- no issues there.
 
 Originality, Significance:
 
 The idea is simple and well motivated, i.e., the learning of reward functions based on feature selection from identified subtasks in videos.
 
 pros:
 - the problem is difficult and relevant: good solutions would have impact
 
 cons:
 - the benefit with respect to other baselines for various choices, although the latest version does contain updated baselines
 - the influence of the initial controller on the results
 - the work may gain better appreciation at a robotics conference
 
 I am very much on the fence for this paper.
 It straddles a number of recent advances in video segmentation, robotics, and RL, which makes the specific technical contributions harder to identify. I do think that a robotics conference would be appreciative of the work, but better learning of reward functions is surely a bottleneck and therefore of interest to ICLR.
 Given the lukewarm support for this paper by reviewers, the PCs decided not to accept the paper, but invite the authors to present it in the workshop track.

Decision: Invite to Workshop Track