Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a framework for detecting incorrect visual demonstrations for imitation learning. The framework first performs temporal segmentation of sub-tasks. Then it uses a maximum entropy model to find the most consistent group of sub-tasks as the correct demonstrations. The experiments are performed on the 50-salad dataset and a customized dataset showing tea making by human.
strengths_and_weaknesses: Strength:
- The method is well-motivated and points out an important problem for imitation learning.
- The method is backed up by theoretical results and analysis.
- The presentation of the paper is overall clear.

Weakness:
- The whole framework is based on video temporal segmentation. What if the segmentation model has really bad performance? How will that affect the final performance of incorrect demonstration detection? For example, since the proposed framework uses a maximum entropy model, if most (> 50% or > 70%) of the demonstrations are wrong or imperfect, can the framework still give desired results by finding the most consistent group of sub-tasks? Or the video segmentation model has a small but the same bias of error on all demonstrations?
- The experimented data are both small in scale as well as very limited in the variations of their scenarios, i.e. 36 videos of human making tea with fixed background and scene. How about demonstration videos in the wild, e.g. making tea in different kitchens by different people?
- Though the overall goal of this paper is to deal with learning from demonstration, the core contribution of the proposed approach seems like an optimization approach for matching or finding consensus that uses max entropy and has no learning components in it. It raises the alert about one of the desk-reject criteria of "No learning" on the CoRL 2022 website: https://corl2022.org/instructions-for-authors/.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Though the problem is important, it's not clear how well the proposed method can perform if the video segmentation model is wrong or unreliable. The experiments are also a little weak for supporting the claim made in the paper.

----

I thank the authors for their response. I will still keep my original "weak reject" score.

My main concerns still remain after the rebuttal. I'm not convinced that the video model presented in this paper is able to generalize to different scenes and scenarios, due to the small scale and limited diversity of the experimented data. This is irrelevant to whether the model takes raw pixels as input or uses a pre-trained feature extraction module. Based on my experience of working on video temporal segmentation and my understanding of the state-of-the-art performance on this problem, I don't believe the proposed entropy method can scale to video data with a larger scale and more diversity in the wild.

By definition of "weak reject", I "will not argue for my recommendation if the majority of other reviewers have a different opinion".
issues: The author can respond to the "Weakness" part of the review.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper addresses an important problem in learning from video demonstrations, which is the presence of incorrect or suboptimal demonstrations in the dataset. Given a raw video input, it first performs sub-task segmentation on the video, and then use a feature generator to extract task-relevant features. Then it is put into an optimization problem solver to learn the demonstration weights, and eventually weight the demonstrations differently according to their demonstration quality. It is able to surpass the performance of IL models in the setting of imperfect demonstrations.
strengths_and_weaknesses: Strengths:

- Important problem setting, intuitive motivation of learning different weights for different quality samples

- Interesting discussion on "Leveraging Vs Discarding Incorrect Demonstrations" - this mirrors the discussion between offline RL and BC on their performance on different datasets. It makes sense that this methods performs better than BC / BC-RNN for datasets with incorrect demonstrations, since it performs the filtering function that filters out suboptimal trajectories. 

- clear presentation of the overview figure about model pipeline and model architecture. nice visualization of segmentation results and how they correspond to the video frames

- convincing results demonstrated on real, multi-step sequential tasks

Weaknesses: 

- For Figure 6: For "BC-RNN without Incorrect Demo", are the number of *correct* demos the same with those in Demo-Dice? If so, I would expect Demo-Dice have similar performance with BC / BC-RNN, because Demo-Dice should be able to eliminate the effect of the incorrect demos?

- For related work section on learning from suboptimal dataset, there should be discussions made to offline RL methods that learns from large dataset from different data quality. In particular, the weighted-BC line of work featuring advantage-weighted BC also learn weights for samples, and the weights from from advantage estimation. Such works include:

1. Critic Regularized Regression

    by Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, Nando de Freitas

2. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets

    by Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine

3. Offline Reinforcement Learning with Implicit Q-Learning

    by Ilya Kostrikov, Ashvin Nair, Sergey Levine
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think the paper addresses an important problem in imitation learning and have explained the motivations well. It gives a novel approach of learning demonstration weights that is applicable in real-world robotics setting. More clarifications and references are needed but in general the paper have convincing results.
issues: As clarifications: 

Regarding the dataset labeling: 
- Does the dataset always come with ground truth labels of incorrect demonstrations? What happens when such labels are not available?

Regarding the feature generator:
- The feature generator extracts relevant task features from the videos. I'm curious whether this requires any assumption on the video demonstration type. The 50 salads dataset seems to have clear visual distinctions between different tasks. What if we are dealing with some robot manipulation tasks that does not involve large difference in visual scenes? How is the performance of this feature generator on common real robot tasks?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper attempts to automatically detect incorrect video demonstrations for multi-step sequential tasks. The method learns a stochastic model of subtask sequence and generate weight for demonstrations using feature expectation matching with principle of maximum entropy. The approaches are verified with two datasets.
strengths_and_weaknesses: Strengths:

•	The paper is clearly written in detail.

•	The authors provide a comprehensive survey over related work. The author also pinpoints the difference between the existing work and their new approach.

•	The experiment analysis of no false negative detection is convincing.

Weaknesses:

•	The motivation emphasizes on detect visual incorrect demonstrations instead of suboptimal demonstrations. And in the statement the incorrectness often come from obvious missing steps or wrong step orders. One concern is that for such obvious mistake, it is very easy to have the demonstrator label it as incorrect or failure or directly delete the demonstration if they conduct a dangerous/incorrect behavior. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes to solve the problem of detecting incorrect visual demonstrations for multi-step sequential tasks. The paper is well-written and overall technical sound. The literature review is thorough and experiment results are convincing. The only problem is the necessity of leveraging the algorithm if human labelling is very easy.
issues: 1.	As mentioned in the “weaknesses” part, what are some situations where human cannot directly distinguish their incorrect behaviors or must spend lot of human efforts for doing that, so that detecting using the algorithm is necessary?

2.	For sequential tasks, what if most parts of the demonstration are correct and only a part of it is incorrect? Could the work be adapted to address this by labelling incorrect for subtasks so that this helps keeping important information?

3.	 Line 153 is a little bit unclear. Is it for each video segment or each video demonstration, that a set of ranked features is extracted? 

Minor issues: 

1.	Line 97 The ‘proposed’ framework

2.	Line 235 Seq_2 is not fully bolded.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The quality of demonstration data is crucial for imitation learning. This paper proposes a method to detect and filter out erroneous data from visual demonstrations (i.e., video streams) of sequential tasks, such as tea making or salad preparation. The proposed method first uses off the shelf methods to segment out subtasks from videos and then uses prior work to extract subtask-relevant features. Then, using the set of task demonstrations, it seeks to learn a predictive model of the next subtask given the current state and, in this process, identify which video demonstrations are erroneous (and therefore should be downweighted). The paper modifies the classic maxEntIRL optimization by posing a mini-max entropy optimization: find a weight for each demonstration that *minimizes* entropy of the sub-task predictive model (i.e., choose demonstrations that enables confident subtask prediction) while finding the distribution that *maximizes* entropy over the predictive model. 
strengths_and_weaknesses: Strengths:
*************
–Interesting problem of identifying erroneous / low-quality demonstration data.

–Valuable direction investigating erroneous high-dimensional data, like images.

–Interesting approach of posing this mini-max optimization to determine which demonstrations are helpful vs. erroneous for predicting subtasks. 

--Interesting comparison of leveraging vs. discarding incorrect demonstrations when it comes to policy learning. 

Weaknesses:
*************
–The paper often makes claims without substantiating them properly with citations or evidence. The specifics of these instances are detailed below on a section-per-section basis. 

–The intuition behind why this mini-max optimization procedure should yield “good” weights that detect erroneous visual data could be strengthened. 

–The technical correctness of equation (3) – the dual problem to the mini-max game – seems questionable. For example, the maximization over the distribution P disappears in the dual problem. Also, the explanation for how this optimization is solved is unclear; solving mini-max games is not trivial in general, and it seems like the proposed method is solved in a way that will likely be an approximate (and not exact) solution to the non-convex game. Providing a more detailed derivation of the dual problem, and discussing the solution quality would strengthen the paper. For example, I believe that the dual problem is:
$$
(w^*, p^*(t|s), \lambda^*, \nu^*) = \arg \min_{\lambda, \nu} \arg \min_{w} \max_{p(t|s)} \mathcal{L}(w, p(t|s), \lambda, \nu) 
$$
where $\mathcal{L}$ is the Lagrangian which includes the entropy objective plus $\lambda^\top \text{(feature expectation matching constraint)}$ plus $\nu \text{(valid probability distribution constraint)}$. It would then be nice to state what are the first order necessary conditions for optimality, for $w^*$, $p^*(t|s)$, $\lambda^*$, and $\nu^*$. 

Intro:

–Nitpick: “Visual imitation learning …. Is probably the most feasible…” This is quite a strong claim. I’d suggest soften it by saying that “Visual imitation learning is a promising way for …”

–“The state of the art…-almost all of which involve non-visual demonstrations–[is learning] weights…” Instead of [is learning] it should be [learn]. Also, it could be helpful to include a few citations of which state of the art approaches are being referred to here. 

–”...2) individual steps of a multi-step task - hereafter referred [to] as sub-tasks…” missing the [to] in the sentence.

–”We bridge this gap…. Which does not have any restrictive assumptions.” Instead of making the broad (and somewhat unsubstantiated claim) that there are no restrictive assumptions in the proposed method, it would be good to focus on the explicit differences between the prior work and the proposed work in this context – for example, mentioning explicitly that the proposed method does not require expert labeling or a simulator for data augmentation.   

Section 2:

–”...where a single incorrect action may or may not cause the algorithm to eventually learn a completely incorrect policy….however, a single incorrect action can lead to an incorrect policy” This statement is not well-explained or well-substantiated. It seems as though regardless of if the task is multi-step or not, a single incorrect action can lead to either catastrophic failure or simply a marginally suboptimal behavior. 

Section 3.1:

–”Here, q is a hyperparameter which does not require fine-tuning as we do not need very accurate sub-task segmentation.” Accurately identifying subtasks (and, related, the number of sub-tasks) seems like an essential prerequisite for the proposed method to work properly. This claim should either be removed or substantiated through some sensitivity analysis. 

–”The proposed framework is tolerant to typical errors in the existing video segmentation algorithms.” This claim should either be removed, or substantiated through sensitivity analysis / quantitative results. 

Section 3.3:

–It would be helpful to provide intuition behind what the value of the weights means. If the weight is = 0, does this mean that the demonstration is not good and therefore is not used, and if the weight = 1, then it is an “optimal” demonstration and should be used to the fullest extent? 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although the problem addressed in this work is interesting, I recommended weak reject on the basis of the mathematical formulation of the core method seeming incorrect/incomplete (i.e., equation 3) thereby bringing into question the technical correctness of the proposed method and results, and the need to further empirically / theoretically substantiate some of the strong claims made in this paper (e.g., the "triviality" of picking thresholds of correct and incorrect demonstrations, sensitivity having accurate sub-task segmentation). 

--------------------------------------

After discussion, I have changed my recommendation to "weak accept"
issues: Section 3:

–Please clarify the technical correctness of equation (3) – the dual problem to the mini-max game. As stated above in the weaknesses, I believe that the dual problem should be:
$$
(w^*, p^*(t|s), \lambda^*, \nu^*) = \arg \min_{\lambda, \nu} \arg \min_{w} \max_{p(t|s)} \mathcal{L}(w, p(t|s), \lambda, \nu) 
$$
where $\mathcal{L}$ is the Lagrangian which includes the entropy objective plus $\lambda^\top \text{(feature expectation matching constraint)}$ plus $\nu \text{(valid probability distribution constraint)}$. It would then be helpful to state what are the first order necessary conditions for optimality, for $w^*$, $p^*(t|s)$, $\lambda^*$, and $\nu^*$, and comment on how the solver works and if there is any concern about obtaining an approximate solution.  

Section 4:

–Table 1 is unclear on its own. For example, what do the rows of the table correspond to? Please clarify this. 

Section 4.2.3:

–”The weights are distributed in a way that it is trivial to pick up two thresholds to differentiate optimal…” This is a strong claim. For example, the thresholds chosen in Figure 4 do not achieve a 100% accuracy in filtering out the optimal demonstrations vs. incorrect demonstrations vs. suboptimal demonstrations. Discussing a principled way to go about choosing a threshold, and what happens when the threshold incorrectly characterizes an optimal demonstration as incorrect, etc. would strengthen this claim. This same concern exists for Section 4.3.3 where the same statement is reiterated. 

Section 4.4:

–”Fig 6 shows policy accuracy …When incorrect demonstrations are present in the training set….poor weights to incorrect videos.” The definition of policy accuracy relies on the number of correct actions chosen – where does the notion of a correct action come from?
review_title: null

Meta-review: metareview: Paper proposes a method to identify sub-optimal demonstrations. The main idea is following: 
- Use an off-the-shelf video segmentation algorithm.
- Identify key spatio-temporal features within each segment. 
- From what I understand, next task prediction $p(t | s)$ is used to calculate if the sub-tasks are in the right order, which is a proxy for how good of a demonstration a particular video is. I don't quite understand why this is the right thing to do, any clarification over here would be great. Authors also use terms like "segments" / "tasks" / "subtasks" interchangeably -- which is confusing. Further, $s$ seems to be overloaded to mean the current state and sub-task (Appendix D). I would highly recommend authors to use a single name for each term for clarity. 

I agree with the reviewers that this is an important problem, and the results are good on the Yumi Robot video demonstrations. Two reviewers vote to accept the paper, and the other two to reject. The primary concerns of the reviewers are: 

- Reviewer key5: Technical correctness, unsubstantiated claims, imprecise language. 
- Reviewer K64Q: analysis of how performance changes with segmentation errors, would the method work in complex scenes with realistic background. 
- Reviewer av9b: Further discussion of related works

I recommend authors to address all the reviewer concerns (both above and others in the reviews) and improve clarity of their method. It will also be good to see an explicit list of limitations -- where would the ranking method fail? 

**Post Rebuttal Update**
Authors provided detailed responses to reviewer questions and consequently reviewer key5 switched from weak reject to accept. In my opinion the authors has addressed the majority of concerns and the paper can be accepted. I encourage the authors to make all the edits promised in the rebuttal and be more explicit about limitations in the camera ready version.  


confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)