Detecting Incorrect Visual Demonstrations for Improved Policy Learning
Authors: Mostafa Hussein
Momotaz Begum
Venue: CoRL 2022 Poster (2022)
Keywords: Imitation Learning
Visual Demonstrations
Incorrect Demonstrations

Abstract: Learning tasks only from raw video demonstrations is the current state of the art in robotics visual imitation learning research. The implicit assumption here is that all video demonstrations show an optimal/sub-optimal way of performing the task. What if that is not true? What if one or more videos show a wrong way of executing the task? A task policy learned from such incorrect demonstrations can be potentially unsafe for robots and humans. It is therefore important to analyze the video demonstrations for correctness before handing them over to the policy learning algorithm. This is a challenging task, especially due to the very large state space. This paper proposes a framework to autonomously detect incorrect video demonstrations of sequential tasks consisting of several sub-tasks. We analyze the demonstration pool to identify video(s) for which task-features follow a ‘disruptive’ sequence. We analyze entropy to measure this disruption and – through solving a minmax problem – assign poor weights to incorrect videos. We evaluated the framework with two real-world video datasets: our custom-designed Tea-Making with a YuMi robot and the publicly available 50-Salads. Experimental results show the effectiveness of the proposed framework in detecting incorrect video demonstrations even when they make up 40% of the demonstration set. We also show that various state-of-the-art imitation learning algorithms learn a better policy when incorrect demonstrations are discarded from the training pool.

Detecting Incorrect Visual Demonstrations for Improved Policy Learning
Mostafa Hussein
Momotaz Begum
Department of Computer Sciences
Department of Computer Sciences University of New Hampshire
University of New Hampshire
mostafa. hwsseinuuniii.. edu
momotaz www.mm unh. edu
Abstract: Learning tasks only from raw video demonstrations is the current state of the art in robotics visual imitation learning research. The implicit assumption here is that all video demonstrations show an optimal/sub-optimal way of performing the task. What if that executing the task? A task policy learned from such incorrect demonstrations can be potentially unsafe for robots and humans. is therefore important analyze the video demonstrations for correctness before handing them over to the policy learning algorithm. This is challenging task, especially due to the very large state space. This paper proposes a framework to autonomously detect incorrect video demonstrations of sequential tasks consisting of several sub-tasks. We analyze the demonstration pool to identify video(s) for which task- features follow 'dirruptvve' sequence. We analyze entropy to measure this disruption and through solving a minmax problem assign poor weights to incorrect videos. We evaluated the framework with two real-world video datasets: our custom-designed Tea-Making with YuMi robol and the publicly available 50-Salads. Experimental results show the effectiveness of the proposed framework demonstrations even when they make up 40% of the demonstration set. We also show that various when incorrect demonstrations discarded from the training pool.
true? What if one more videos show a wrong way of
detecting incorrect video
imitation learning algorithms Iearn a better policy
Keywords: Imitation Learning, Visual Demonstrations, Incorrect Demonstrations
1 Introduction
Visual imitation learning (VIL) [1, - tasks are learned from raw video demonstrations- is a promising way for lay users to teach robots new skills in natural settings. VIL however challenging. especially due very high dimensional state space. VIL algorithms are generally evaluated in controlled laboratory settings [4, 5, 6, 7, 8, 9] or simulated environments [10, 11, 12, 13, 14], where the assumption that all video demonstrations reality, lay users will make inadvertent errors while demonstrating a task due to reasons including fatigue, presence of distractions, and lack of knowledge on robot learning mechanism. For example. it is not unusual to forget to turn-off the oven once or twice- c.g., because the phone rang- out of the ten times user was showing the robot how to make a cup of tea. Unless a robotics expert carefully curates the videos, a policy learning algorithm can learn a potentially unsafe tea- tea-making policy - that it is ok to keep the oven on from these video demonstrations. It is therefore important to automatically analyze video demonstrations for correctness before policy learning takes place. This paper proposes framework for detecting incorrect video demonstrations 1, Many weal-world tasks that service robot is expected to learn from humans sequential tasks consisting of several sub-tasks, e.g. following a recipe for cooking, tea- mea-making, preparing a dinner table, etc. The proposed framework therefore deals only with videos of sequential tasks.
optimal/ sub-optimal is possible to maintain. In
The offteoof-hhe-rd approaches for detecting incorrect demonstrations [15, 16, 17] almost all of which involve non-visual demonstrations - learn weights to assign importance to different demon- strations. Existing algorithms however come with restrictive assumptions such as, expert-labeling of incorrect demonstrations [15, 18, 19], using simulator to generate more data for training
We define incorrect demonstrations as ones that do not follow the standard task definition and are structurally wrong. For example, in the context of a tea-making task, an incorrect demonstration would be the one where the demonstrator did not add water or tea ba-baa to the cup.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

[16. 20, 17. 21] which nullify the core appeal of VIL that a lay user's only responsibility is to show how to do the task, preferably a handful of times; the onus is on the algorithms to learn an accurate policy without demanding more data. We bridge this gap by proposing a framework for autonomous detection of incorrect video demonstration without the need for prior labeling of the videos or a simulation for data augmentation.
The proposed framework leverages two simple facts: 1) spatio-temporal features are better repre- sentatives of dynamic human activities than spatial-only visual features and 2) individual tasks of a multi-step task hereafter referred to as sub-tasks typically maintain some internal structures among themselves, violation of which is an indication of incorrect demonstration Accordingly, the intuition behind the proposed framework is to identify groups of representatives of different sub-tasks (Sections 3.1 and 3.2) and analyze the internal consistency of these features overall videos in the demonstration pool (Section 3.3). To achieve that we propose to solve minmax optimization problem where we choose the model with the maximum entropy (the most uniform) and at the same time minimizing the demonstrations weight so we can achieve a saddle point with the minimum number of demonstrations. Evaluation of the proposed framework on two real-world datasets- publicly available 50-salad [22] and our custom-designe tea-making with YuMi robor[23] - shows promising results on detecting incorrect demonstrations even when they constitute 40% of the entire demonstration pool. We also show that various state-of-the -f---h--tth-rttt IL algorithms learn better policies when incorrect demonstrations are automatically removed using the proposed framework than IL algorithms that learns from uub-optimal/incorrett demonstrations.
features as the
2 Related Work
The current imitation learning literature does not explicitly discuss the issue of incorrect demonstra- tions and how they may affect the learned policy. However, there exists a handful of work that offers mechanisms to differentiate optimal, sub-optimal, and noisy demonstrations during the learning process. This section discusses these methods.
The 21WIL framework [15] requires an expert to label fraction of the demonstrations as optimal/non- optimal using confidence score. A semi-supervised classifier is then trained to predict the confidence score of any unlabeled demonstration. Finally, these scores are used by weighted GAIL framework while learning the task policy. To avoid error accumulation in such a two-step learning process, IC-GAIL [15] trains a single framework for generating the confidence score and learning the policy in an end-to-end fashion. The WGAIL [16] introduces a method of assigning weights to imperfect demonstrations in GAIL [24] without imposing extensive prior information. The success of WGAIL [16] however is limited to simulated environments.
The works that are conceptually closest to our proposed work are R-MaxEnt [25, 26), BCND [27], and DEMO-DICE [28]. M-MaxEnt [25, 26] jointly learns weights for different demonstrations and a task policy. The method however requires carefully-crafted task features, which are only achievable through hand-picking, and therefore has been evaluated for tasks with small state space. BCND [27] learns ensemble policies with weighted BC objective function where the expert demonstrations sampled from noisy expert policy. The weights are generated from the policy Iearned in a previous internal iteration. DEMO-DICE [28] learns from different level of optimality in the demonstrations but through a much robust convex optimization problem. None of these three approaches however discuss how they can be scaled up for large state space, i.e. visual imitation learning.
Several IL algorithms are capable of detecting demonstrations corrupted by a limited amount of random noise [18, 20, 28]. Demonstrations with artificially injected random noise are far more trivial to process than incorrect demonstrations provided by humans in real world! 1 Additionally, some of these works [18, 20] require simulators to generate additional data.
number of inverse reinforcement learning- based IL algorithms [19, 29] use "failed" demonstration, along with optimal ones, for policy learning. However, they assume that these failed demonstrations are labeled a priori by an expert. Although some recent works on offline reinforcement learning achieve high accuracy in policy learning, they can't handle incorrect demonstrations [30, 31, 32).
All algorithms discussed above are designed to work with non- vosuvl demonstrations and can not be scaled-up to deal with extremely large state space (of video demonstrations). None of these methods can detect incorrect demonstrations without prior labeling of training data or access to more data (through simulator). Furthermore, all existing work sub-optimal demonstrations deal with continuous control tasks- - e.g. Manipulation, Ant, HalfCheetah, etc. Where a single incorrect action (c.g. wrong joint-angle ) may not cause the algorithm to eventually learn a completely

---

Raw Input
Optimictiion problem salver
fraac
Mbp
A
MEg M21
A
Nes Mey1
Demonstration atioon
Weights
Step A: Sub task segmentation
Feature extraction
Step ghts generation
Figure I: The proposed framework has steps: the first step (green shaded box, Sections 3.1 and 3.2) takes as an input n raw videos and does sub-task segmentation, generating q segments each with frames. features generator extracts f spatio-temporal visual features from each segment, creating x f matrix for each video. The second step (red shaded box. Section 3.3) concatenates all these features and uses them to solve a minmax problem which generates weights, for each input video.
incorrect policy. In case of stet sequential tasks, however, a single incorrect action can lead to an incorrect policy e.g. not turning off the oven or not adding a tea-bag while making a cup of tea. Our proposed work fills up this void in visual imitation learning research.
3 Proposed Framework
The proposed framework identifies incorrect demonstrations of a multi-step sequential task through analyzing the uub-task-representatvv spatio-temporal features. We leverage the concept of feature expectation matching [33, 34, 35] along with the principle of maximum entropy [36, 37] to learn a stochastic model of the sub-task sequence and generate weights for different demonstrations. Entropy analysis enables us to identify the most consistent group of demonstrations in terms of sub-task features sequences and assign higher weights this group. Note that, multi-step sequential tasks can often be executed in multiple ways such as making tea with without milk/sugar all of which are considered as correct. The proposed model is capable of distinguishing such cases from the case of incorrect demonstrations through generating distinct weight distribution.
Fig. shows an overview of the proposed framework. The input is a set of raw video demonstrations and the output is n-dimensional vector representing the weight of each demonstration. The framework calculates these weights in a two-step process. At the first step (green shaded box in Fig. 1) each video is segmented into sub-tasks using any off-the-sheli video segmentation algorithm, e.g. [38] (Section 3.1). Our custom- designed algorithm then extracts spatio-temporal visual features from each sub-task video segments (Section 3.2). These features are representatives of various sub-tasks and are passed the second step of the framework. During the second step (red shaded box in Fig. 1) these representative features are used to learn a stochastic model of the sub-task sequence which in-turn generates weights for different video demonstrations (Section 3.3).
3.1 Sub-task Segmentation
video segmentation algorithm used to segment each video the demonstration D into q sub-tasks and generate the start- and end-frame for each sub- task. Here, q is hyper-parameter for our approach. We employed the semi-supervised MS-TCN+ [38] for this purpose. MS-TCN++ leverages temporal convolution for temporal action segmentation and re- quires some labeled data for training. MS-TCN++ uses multi-stage architecture for the temporal ac- tion segmentation task. Each stage features set dilated temporal convolutions to generate an initial prediction that refined by the next one. MS-TCN+ requires some training data with the ground truth frame labels. Fig. shows the segmentation sub-tasks (each sub-task coded with different colors). Note that the only purpose of this segmentation assigning a consistent label to the set of task-relevant features that will be extracted from the sub-task videos (Section 3.2). The proposed framework is tolerant typical errors in the existing video segmentation algorithms up to 83% accuracy.
MS-TON
Figure 2: Top row: Sample frames from 50-salad dataset representing the start of different sub-tasks. Middle row: The ground truth (GT) time-line for each sub-task this video (colors representing different sub-tasks). Bottom row: Segmentation generated by MS-TCN++
video from the 50-salad dataset into q = 17
3

---

3.2 Learning Task-relevant Features
The hallmark of most multi-step sequential tasks is their high-level temporal structure. For example, in the context of tea-making task, stirring the cup happens after milk or sugar added etc. We employ our own feature generator [39] that can identify and rank subset of visual features from video based on their temporal significance. We briefly discuss this feature identification process in this section.
The feature generation happens in three steps: 1) a set of spatial features are extracted from the training videos through any pre-trained convolutional neural networks (CNN) such as VGG-16 [40]. These features, by construction, are not sensitive to temporal content of the video. 2) Extracted spatial features alongside with their automatically extracted time stamp (i.e. when feature started and stopped appearing in the video) are used to create a graph of temporal relationships among spatial features. This graphical structure that we name interval temporal relationship (ITR) graph is used to train a graph convolution network [GC which learns the discriminatory temporal features present in the ITR graph. These temporally sensitive features ranked, based on their contribution in classifying the activity in the video, using erasure ranking [42]. Only a set of high-ranked features are used for further analysis. More details are in appendix B.
In summary, each video in the demonstration set D is automatically segmented into g sub-task segments. For each video segment, a set of ranked features {sy}ā-1 is extracted, s = {fn >N N: the number of features from each segment. These sets, along with the sub-task labels {t, } are passed to the next step.
3.3 Incorrect Demonstration Detection
We aim to learn stochastic model p(t]s) that can infer the next sub-task t E T given the current state E S and leverage this model to determine a weight w E [0, 1]2 for each video in the demonstration set D.
To derive the model p(([s) and the weights w we leverage the principle of maximum entropy (ME) [36, 37] which is proven to choose the most uniform model among a set of constrained models. We use the concept of feature expectation matching (FEM) [33, 34, 35] widely popular approach in II. [33, 34, 35] - to define a set of constraints to observed by p(||s). the context of incorrect demonstration detection, the goal is to match the expected value of each feature Epffi] that identify from the demonstration videos through the feature generator (discussed in Section 3.2) with the calculated expected value [fi] generated by the learned model p(t s). This model p(t]s) will represent the sub-task sequences in the demonstration videos in the most unbiased manner [37].
Along with the model p(t]s), we want to generate a weight w that indicates how well specific video conform with the learned model. We achieve this by adding to the model-objective the requirement of calculating the minimum weights that give us the model with the maximum entropy. Using this objective we are forcing the system to generate weights that indicate how consistent a video is with the majority of the videos in the training pool. Poor consistency (lower values of w) is an indication of incorrect demonstration. We start with the expected feature definition to formally derive s))ss):
EEPL = ΣΣ (())(()))
EplA) = =Σ ((s)
(I)
SESIET
SESIET
Here ((s) is the empirical distribution of the states in D. D.f(()s) is the empirical conditional distribution of predicting sub-task given the state s. f(, t) is a feature function whose values are state features corresponding to predicted sub-tasks (Section 3.2). The optimization constraint, according to FEM, [f E = O. This formalism considers the training set D one monolithic whole. Our goal however is to consider each video-demonstration separately and generate a weight w E [0, 1] for each of them. We therefore redefine the distributions terms and parameterized them by demonstration d and weight w as follows:
D
Pn (t|s)
Σ
Wa p(t]s, d)
(s) =
1 M
Σ Wa p(s d) d=1
M
d=1
Here D is the total number of demonstrations and M, which should be Σb wa M, is the minimum number of demonstrations that we can trust a given set. Through the experimental results, we can see that the algorithm is not sensitive to the value of M. Usually, we can set this number to be M = D/2 since a dataset where more than half of the videos have incorrect demonstrations is not suitable for learning.

---

D
min p(()SERSXX
max
CEp(lls log p(()s)
Σ
wa p(s,d)
WER"
SESIET
d=1
H(p(els))
(2)
Σ Wa ΣE SESIET
Si (0 - =(((ss,)) d))
6.
D
E
E -1118)-1100 Vs LET
Σ wa - M, wa 2 0, AA D.
wa S
We define the optimization problem as shown in equation 2. Here the objective function maximizes the model entropy H (p(t[s)) while minimizing the demonstrations weights w. The first constraint is the FEM between the empirical expected feature values and the generated feature value after the modification with the wight parameter. The second constraint is to make sure we generate a correct probability distribution of the sub-task prediction p(t(s) s). The last constraint is to ensure that there at least M optimal demonstration in the dataset. We use Lagrange multiplier approach to derive the dual problem shown in (3) that we solve to find the weights w. The weights assigned to incorrect demonstrations are significantly lower than the correct ones. A complete derivation and implementation details can be found in the appendix A.
min AAERN
A(A, w)
1 M
Σ Wa
Σ
((s, ZA(5
Σ i=1
A
ΣΣ
5(1ss, d).ss.)
#ES
#ES LET
D
8. t.
Σ d=1
Wi M, we 2 0 E wa 1
(3)
Here (s) = EIET exp (Σ> (s,((s.t) is normalization constant, and À is the Lagrange multiplier. A full pseudo code for the approach and how to solve the optimization problem can be found in appendix D.
4 Experiment and Results
We conducted experiments to investigate the following: (1) can a video segmentation algorithm be used to detect incorrect video demonstrations? (Section 4.1) (2) how well the weights generated by the proposed framework can differential among optimal, sub-optimal and incorrect demonstrations (Sections 4.2 and 4.3), and (3) what generates better policy?: learning while leveraging incorrect demonstrations vs discarding incorrect demonstrations from the training pool (Section 4.4). We used two video datasets for experiments: our custom-designed imitation learning dataset Tea-Making with a YuMi Robot and publicly available activity recognition datasct 50-salads [22].
4.1
Video Segmentation Vs Incorrect Demonstration Detection
Since the proposed framework used video segmentation algorithm which performed sub-task segmentation and thereby generated a sequence of sub-task for each video, natural confusion may arise: can we use a clustering algorithm
Accuracy Tolerance
0%
5%
30%
10%
to analyze the sequence of sub-tasks from a demonstration pool and identify the
50%
20%
incorrect demonstrations? We performed series of experiments to resolve this. 70%
30%
All video segmentation algorithms make non-zero segmentation errors and the consequence of inaccurate segmentation is detrimental identifying incorrect Table 1: Sub-task ac- demonstrations. For example, the highest reported per-frame accuracy of curacy iidee-segmentation algorithm is 83% on the publicly available 50-salads dataset [38]. The sub-task segmentation accuracy which is relevant for detecting incorrect demonstration reduces to 70% with 30% tolerance in per-frame accuracy (i.e. we allow 30% of the frames from sub-task to be incorrectly classified). This accuracy decreases significantly with reduced tolerance in per-frame accuracy. Table summarizes the results. As an example, Fig. shows the segmentation of a video from the 50- 10-salad dataset into 17 sub-tasks (each sub-task coded with different colors) using the best-performing segmentation algorithm reported in four sub tub-tasks with false labels. There is known way to directly use such output sequence to distinguish between correct vs incorrect demonstration.
[38]. The algorithm has generated more than
5

---

4.2 Analysis of Demonstration Weights: Tea-Making with YuMi Robot
4.2.1
The dataset
This custom- -estomddesigned dataset has 36 videos showing three different human participants making a cup of tea each participant performing 12 trials. The dataset was created through an IRB-approved user study. There are seven ground-truth sub-tasks in tea-making: Turn on the oven, Add water, Add sugar, Add milk, Add tea bag, Stir; Turn off the oven. Participants were asked to make tea while following any sequence of their choice but they must use both ingredients (i.e. milk and sugar). In majority of the resultant videos participants observed the following sequence: Turn the oven, Add tea bag, Add sugar, Add milk, Turn off the oven, Add water, Stir. We consider these videos as optimal demonstrations and use the term Seq_1 to denote them. Only subset of videos show participants using the following sequence: Turn on the oven, Add sugar. Add milk, Add tea bag. Turn off oven, Add water, Stir. We consider these videos as sub-optimal demonstrations and use the term Seq_2 to denote them. In a small subset of videos, the participants based on our instructions skipped one two of the three critical sub-tasks namely, Add tea-bag, Add hot water, Turn off the oven. These videos are considered as incorrect demonstrations since they show a wrong or unsafe way of performing the tea-making task. We use the term Seq to denote them. Finally, to facilitate IL policy learning by real robots which requires the joint angles of the robot for training another 12 demonstrations were added the dataset where the YuMi robot, instead of human demonstrator, is remotely controlled to do the same tea-making task while following the same sub-task sequence as human demonstrators. All 48 videos used for training and evaluations. A set of representative frames from three different types of video demonstrations are shown in appendix C 4.2.2 Sub-task segmentation and feature generation
Using MS-TCN++ [38] we segmented the videos into q 7 sub-task segments with a per-frame accuracy of 80%. Fig. 3 shows the segmentation of a video from the lea-making dataset From each segment, 64 frames were uniformly sampled. Frames were re- duced to 224 x 224 pixels and went through back- ground subtraction before being fed to the feature generator (Section 3.2). The dimension of the gen-
Figure 3: Top row: Sample frames from tea- making task representing the different sub-tasks. Middle row: The ground truth sub-task segmen- tation (each color represents sub-task). Bottom row: sub-task segmentation by MS MS-TCN+ [38]. Incorrectly added sub-tasks are in purple, green, 64 frames. Appendix B shows a visual representa- and black).
crated features was 64 x 32 which was further re-
duced to x 32 through a max function over the
tion of some of the features generated by the model.
4.2.3 Demonstration weights
The proposed framework, through equation (3), gen- erates weights for different videos. This section discusses how these weights are used to identify optimal, sub-optimal, and incorrect demonstrations, For this, we create two demonstration sets; the first one has 6 optimal videos (seq 1: dl to d6), sub- optimal (seq_2: d7 to d10) and 2 incorrect ones (d11 to d 12) while the second set has optimal (seq 1: d1 to d8) and 4 incorrect videos (d9 to d12).. Fig. shows the weights for different demonstra- tions for these two sets after normalizing them to have values between [0, 1]. The weights are dis- tributed in such way that it is trivial to pick up two thresholds to differentiate optimal, sub-optimal, and incorrect videos. For example, we used k-means algorithm [43] to cluster the weights while choosing the middle point between the cluster means as the threshold. The number of clusters is preset to or 3, depending on the number of ground truth clusters. We experimented with two other clustering algorithms- Gaussian mixture model [43] and Mini-Batch K-Means [44] - both of which generated the same results. Detailed results are reported in appendix E.
En dz label
a B1r0
label
(a) The first set.
(b) The second set.
Figure 4: Weights for two sets of videos from the Tea-Making with YuMi Robot dataset. It is trivial to pick up two thresholds to differentiate optimal ('bluc"), ) sub-optimal ('green') "), and incorrect vidcos ('red").
Note, there is one false positive (FP) in each of these two sets, d3 and d4. Further analysis on these two FP videos indicates two possible sources of errors: 1) the solver that solves the non-convex.

---

non-linear optimization problem in (3) did not converge to the global minimum or (2) the feature generator (Section 3.2) failed to identify a good set of features. Section 4.2.4 discusses the implication of generating erroneous weights.
4.2.4 Robustness of performance
We investigated the robustness of the proposed method in detecting sub-optimal and incorrect demonstrations. For this, we made a demonstration pool with 10 optimal demonstrations (seq 1) and one sub-optimal or incorrect demonstration (seq 2 or seq.3). Keeping the number of op- timal demonstrations fixed at 10, we increased the incorrect (or sub-optimal) demonstrations in the dataset until they become 44% of the dataset (i.e.. 10 optimal and 8 incorrect or sub- optimal). Tables 2 and 3 show accuracy, recall, and precision in different cases. Here, false positive (FP) indicates an optimal video demonstration inaccurately detected as incorrect (or sub-optimal) while false negative (FN) is an incorrect (or sub-optimal) video incorrectly identified as optimal. True positive (TP) and true negative (TN) are also defined accordingly. The proposed framework can detect sub-optimal and incor-
rect video demonstrations with an average accuracy of 86% and 90%, respectively. However, when detecting incorrect demonstration, the metric that carries most significance is Re- call TP The proposed framework shows 100% recall,
#Scq. [#Se Accuracy Recall Precision
10
1
90
100
67
10
4
85
100
80
10
80
both for Seq 2 and Seq 3. In other words, the framework can correctly identify every incorrectsub-optiinal demonstration in the dataset. The implication of this is significant: the framework can pass to the policy learning algorithm a training set consist- ing of purely optimal demonstrations and thereby learning highly accurate policy [15, 16, 20, 45] (more discussion in Section 4,4). While the framework generates some false positive results (as shown by Precision values 78% and 83%, respectively ), they do not directly jeopardize the quality of the learned policy. However, too many false positive may cause many demonstrations to be thrown away, resulting in a rapid shrinking of the demonstration pool which is undesirable for realistic applications where collecting demonstratinn is costly.
10
8
83
100
84
Average
86%
100
78
Table 2: Robustness with Seg 2
We performed one other experiment where used video segmentation algorithms with various accuracy and investigated the robustness of performance of the proposed framework. The detection accuracy of the proposed framework remains the same even with video segmentation accuracy of 100% (achieved through manual segmentation). Results from this experiment are in appendix E.
#Soq 1|#Soq 3 Accuracy Recall Precision
4.3 Analysis of Demonstration Weights: The 50-salads
1
67
4.3.1 The dataset
10
2
92
100
80
89
The 50-Salads is a public dataset containing 50 videos of 25 actors preparing two different types of salads. The videos are recorded from an overhead camera and the average video dura- tion is 6.4 minutes. There are 17 unique sub-tasks involved with preparing the dressing and mixing different ingredients (Cheese, cucumber, lettuce, tomato, dressing). However, the average num- ber of sub-tasks in different videos is 20 as different actors follow different sub-task sequences to prepare the salad. Since it is not custom-designed datasets, we had to devise strategy to create incorrect demonstrations. Being an activity recognition dataset, 50-salads is a balanced dataset where half of the videos starts with sub-tasks related to preparing dressing and the other half with that related to cutting and mixing different ingredients. We used the first half of the dataset to create the incorrect demonstrations by removing all frames related to preparing dressing. Thus the optimal demonstrations are those videos where actors cut and mix ingredients and then prepare the dressing while incorrect demonstrations are the ones where actors miss the sub-tasks related to preparing dressing. Because of the limitation of the dataset, we focused only on detecting the incorrect demonstration, not the sub-optimal demonstrations. Fig. 2 shows example frames from the 50-salads dataset.
10
6
88
100
86
10
89
Average
90%
Table 3: Robustness with Seq.3 3
4.3.2 Sub-task segmentation and feature generation
Using MS- M [38] we segmented each videos into q 17 sub-task segments with a per-frame accuracy of 77% which is very close to the accuracy reported in the original paper [38]. Fig. 2 shows the segmentation of video from the dataset. Feature generation followed the same strategy adopted for the Tea-making with YuMi Robot dataset.
7

---

4.3.3 Demonstration weights
To analyze the weights of optimal vs incorrect demonstrations we created a demon- stration set with 10 correct demonstrations (d1 to d10) and incorrect ones d11 to d12. Fig. 5 shows the normalized weights for different demonstrations.
Demonptietioon aedeell
4.3.4 Robustness of performance
We evaluated the robustness of the proposed method in detecting only the incorrect demonstrations in the 50-Salads dataset. We made a demon- stration pool with 10 optimal demonstrations and incorrect one, and gradually increased the number of incorrect demonstrations in the pool.
Table 4 shows accuracy, precision, and recall for different settings. From Table 4, the average accuracy is 85% which is lower than that with the Tea-making with YuMi Robot dataset. This is because Salad-preye preedd-peppration is much more complex task than Tea-making with more than double number of sub-tasks. However, the recall performance remains 100% which means we never missed an incorrect demonstration (FN = 0) desirable property for eventually learning an accurate policy. However, unlike the Tea-making with YuMi Robot dataset, we could not empirically investigate the policy accuracy for this dataset due to lack of robot joint angle data.
Demonsttatio label
Figure 5; Weights for the 50-Salads dataset.
4.4 Leveraging Vs Discarding Incorrect Demonstrations: Which Gives the Better Policy?
For the tea-making task, we created demonstration set con- taining varying number of incorrect demonstrations and in- vestigated the effect of having incorrect demonstrations on
Seq ||# Seq Accuracy Recall Precision
90
100
67
10
83
100
67
the accuracy of the learned policy. We studied three differ- ent strategies for policy learning: i) we use the proposed framework to detect incorrect demonstrations and discard them from the training set before learning a policy using BC/BC- RNN [46, 2), ii) we use BC/BC-RNY RENNC-NNNNA to learn a policy with- out discarding the incorrect demonstrations iii) we use two contemporary IL approaches DEMO-DICE [28] and R-MaxEnt [25, 26] that leverage incor- rect demonstrations for policy learning but assign poor weights to them in the learning process. Fig. 6 shows the policy accuracy in different settings where policy-accuracy is defined as (Number of correctly chosen actions Total number of executed actions until the goal reached) 100. When incorrect demonstrations are present in the training set, Demo-Dice and R-MaxEnt learned better policy than BC/BC-RNN by assigning poor weights incorrect videos. However, BC/BC-RNN learned far better policy than Demo-Dic and R-MaxEnt when the incorrect videos were removed by the proposed framework based on their poor weights. Implementation details and code links for the policy learning algorithms can be found in appendix D.
85
100
80
10
81
100
77
Average
85%
100
73
Table 4: Robustness with Seq_3 for the 50-Salads
5 Conclusion
We presented a novel approach for detecting incorrect video demonstrations of any multi-stage sequential task. The pro- posed method leverages entropy analysis to identify vidco(s) in the demonstration pool that follow a 'disruptive" seskese and assign poor weights to them. Using two different datasets we demonstrated the effectiveness of our approach. The pro- posed framework can be used as pre-processing step for any visual IL algorithm to improve the accuracy of the learned policy.
80
Accuracy
40
+
SC-99N withur. incomed durio.
BC without incomeet deno.
Demo--icc
BC RARRR/Wtt Incornect demo.
BC with incorrect deno.
0%
10%
20%
301
40%
50%s
6 Limitations and Future Work
Incorrect demonstration percentage
The proposed framework requires a value for (the number of sub-tasks) at the beginning of the pipeline. Techniques for sub-goal identification from video - an active research area in computer vision can be employed for automatically generating a value for q. The proposed work also limited in detecting missing sub-tasks without providing any information about the importance of these sub-tasks. Introducing risk parameter for each sub-tasks and learning those parameters from the dataset is possible solution.
Figure 6: Accuracy of the tea- sa--making task policy learning.
8

---

7 Acknowledgement
This work was supported in part by the National Science Foundation (IIS 1830597).
References
[1] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Visual adversarial imitation learning using variational models. Advances in Neural Information Processing Systems, 34, 2021.
[2] A. Mandlekar, D. Xu, J. Wong. S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín- What matters in learning from offline human demonstrations for robot manipulation. arXiv preprint arXiv:2108.0322 2021.
[3] Y. Zhu, P. Stone, and Y. Zhu. Bottom-up skill discovery from unsegmented demonstrations for long-horizon robot manipulation. IEEE Robotics and Automation Letters, 2022.
[4] L. Smith, N. Dhawan, M. Zhang, P. Abbeel, and S. Levine. Avid: Learning multi-stage tasks via pixel-level translation of human videos. arXiv preprint 19Xiv 02.0442204444. 2019.
[5] S. Pirk, K. Hausman, A. Toshev, and M. Khansari. Modeling long-horizon tasks as sequential interaction landscapes. Conference on Robotic Learning, 2020.
[6] R. Rahmatizadeh, P. Abolghasemi, I.. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE international conference on robotics and automation (ICRA), pages 3758-3765, IEEE, 2018.
[7] B. Wu, S. Nair, L. Fei-Fei, and C. Finn. Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks. arXiv preprint arXiv:2 (0332222099/031,,, 2021.
[8] M. Hussein, M. Begum, and M. Petrik. Inverse reinforcement learning of interaction dynamics from demonstrations. In 2019 International Conference on Robotics and Automation (ICRA), pages 2267-2274. IEEE, 2019.
[9] Nair and C. Finn. Hierarchical foresight: Self-supervised learning of long-horizon tasks via visual subgoal generation. arXiv preprint 1rXiv:1909.0882 2019.
[10] C. Lynch, M. Khansari, T. Xiao, V. Kumar, Tompson, S. Levine, and Sermanet. Learning latent plans from play. In Conference on Robot Learning, pages 1113-1132. PMLR, 2020.
[11] V. Tangkaratt, N. Charoenphakdee, and M. Sugiyama. Robust imitation learning from noisy demonstrations. arXiv preprint arXiv:2010. 10181, 2020.
[12] H. Lin, B. Li. X. Zhou, J. Wang, and M. Q.-.. Meng. No need for interactions: Robust model-based imitation learning using neural ode. arXiv preprint arXiv:2104.01399, 2021.
[13] D. S. Brown, W. Goo, and S. Niekum. Better-than-demmonsrratto imitation learning via automatically-rankel demonstrations. In Conference on robot learning, pages 330-359. PMIR, 2020.
[14] Y. Lii, J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. In Proceedings the 31 International Conference Neural Information Processing Systems, pages 3815-3825, 2017.
[15] Y.-H. Wu, N. Charoenphakdee, H. Bao, V. Tangkaratt, and M. Sugiyama. Imitation learning from imperfect demonstration. In International Conference on Machine Learning, pages 6818 2818-6827. 6827. PMLR, 2019.
[16] Y. Wang, C. Xu, B. Du, and H. Lee. Learning to weight imperfect demonstrations. In International Conference Machine Learning, pages 10961-10970. PMLR, 2021.
[17] I. Liu, Tang, I.. Li, and D. Luo. Robust imitation learning from corrupted demonstrations. arXiv preprint
2022.

---

[18] D. Brown, W. Goo. P. Nagarajan, and S. Nickum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. International conference on machine learning, pages 783-792. PMLR, 2019.
[19] K. Shiarlis, J. Messias, and S. Whiteson. Inverse reinforcement learning from failure. 2016.
[20] V. Tangkaratt, Han, M. Khan, and M. Sugiyama. Variational imitation learning with diverse-quality demonstrations. In International Conference on Machine Learning, pages 9407-9417. PMLR, 2020.
[21] Y. Wang, C. Xu, and B. Du. Robust adversarial imitation learning via adaptively-selecnt demonstrations. In Proceedings of the Thirtieth International Joint Conference Artificial Intelligence, IJCAI-21, pages 3155-3161, 2021.
[22] S. Stein and S. J. McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 729-738, 2013.
http::/:w...t.eee..t....e...rers mah1075_ http://wwww....aaaa.aa....aeaaaa e=a 9A2tA.
[24] Ho and S. Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, pages 4565-4574 2016.
[25] M. Hussein. B. Crowe, M. Petrik, and M. Begum. Robust maximum entropy behavior cloning. arxiv preprint arXiv:2101.0123 00100151,, 2021.
[26] M. Hussein, B. Crowe, M. Clark-Turner Gesel, M. Petrik, and M. Begum. Robust behavior cloning with adversarial demonstration detection. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7858-7864. IEEE, 2021.
[27] Sasaki and R. Yamashina. Behavioral cloning from noisy demonstrations. In International Conference on Learning Representations, 2020.
[28] G.-H. Kim, Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.-E. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2021.
[29] D. H. Grollman and A. Billard. Donut as i do: Learning from failed demonstrations. In 2011 IEEE International Conference on Robotics and Automation, pages 1804-3809, IEEE, 2011.
[30] Z. Wang, A. Novikov, K. Zolna, J. S. Merel. J. Springenberg. S. Reed, B. Shahriari. N. Siegel, C. Gulcehre, N. Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[31] A. Nair, A. Gupta, M. Dalal, and Levine Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.0933: 2020.
[32] I. Kostrikov, A. Nair, and S. Levine. Offline reinforcement learning with implicit q-learning. arXiv preprini arXiv:2/
2021.
[33] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, and J. Peters. An algorithmic perspective on imitation learning. arXiv preprint (rXiv:1811.067111 2018.
[34] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent advances in robot learning from demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3:297-330, 2020.
[35] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence (AAAI), volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
[36] S.-i. Amari. Information geometry and its applications, volume 194. Springer, 2016.
10

---

[37] E. T. Jaynes. Information theory and statistical mechanics. Physical review, 106(4))620, 1957.
[38] S.-J. Li, Y. AbuFarha, Y. Liu, M.-M. Cheng, and J. Gall. Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on pattern analysis and machine intelligence, 2020.
[39] M. B. Clark-Turner. High Level Learning Using the Temporal Features of Human Demonstrated Sequential Tasks. PhD thesis, University of New Hampshire, 2021.
[40] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1553 2014.
[41] M. Schlichtkrull, N. Kipf, Bloem, R. Van Den Berg, Titov, and M. Welling, Modeling relational data with graph convolutional networks. In European Semantic Web Conference, pages 593-607. Springer, 2018.
[42] Li, W. Monroe, and D. Jurafsky, Understanding neural networks through representation erasure. arXiv preprint arXiv: 1r11::661. 2016.
[43] C. M. Bishop and N. M. Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[44] D. Sculley. Web-scale k-means clustering, In Proceedings of the 19th international conference on World wide web, pages 1177-1178, 2010.
[45] Zheng, S. Liu, and L. M. Ni. Robust bayesian inverse reinforcement learning with sparse behavior noise. In Twenty-Fightt AAAI Conference on Artificial Intelligence, 2014.
[46] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991.


Reviews:
[{"review_id": "cAxh2LufzsE", "reviewer": "Reviewer_K64Q", "summary": "This paper introduces a framework for detecting incorrect visual demonstrations for imitation learning. The framework first performs temporal segmentation of sub-tasks. Then it uses a maximum entropy model to find the most consistent group of sub-tasks as the correct demonstrations. The experiments are performed on the 50-salad dataset and a customized dataset showing tea making by human.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The method is well-motivated and points out an important problem for imitation learning.\n- The method is backed up by theoretical results and analysis.\n- The presentation of the paper is overall clear.\n\nWeakness:\n- The whole framework is based on video temporal segmentation. What if the segmentation model has really bad performance? How will that affect the final performance of incorrect demonstration detection? For example, since the proposed framework uses a maximum entropy model, if most (> 50% or > 70%) of the demonstrations are wrong or imperfect, can the framework still give desired results by finding the most consistent group of sub-tasks? Or the video segmentation model has a small but the same bias of error on all demonstrations?\n- The experimented data are both small in scale as well as very limited in the variations of their scenarios, i.e. 36 videos of human making tea with fixed background and scene. How about demonstration videos in the wild, e.g. making tea in different kitchens by different people?\n- Though the overall goal of this paper is to deal with learning from demonstration, the core contribution of the proposed approach seems like an optimization approach for matching or finding consensus that uses max entropy and has no learning components in it. It raises the alert about one of the desk-reject criteria of \"No learning\" on the CoRL 2022 website: https://corl2022.org/instructions-for-authors/.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Though the problem is important, it's not clear how well the proposed method can perform if the video segmentation model is wrong or unreliable. The experiments are also a little weak for supporting the claim made in the paper.\n\n----\n\nI thank the authors for their response. I will still keep my original \"weak reject\" score.\n\nMy main concerns still remain after the rebuttal. I'm not convinced that the video model presented in this paper is able to generalize to different scenes and scenarios, due to the small scale and limited diversity of the experimented data. This is irrelevant to whether the model takes raw pixels as input or uses a pre-trained feature extraction module. Based on my experience of working on video temporal segmentation and my understanding of the state-of-the-art performance on this problem, I don't believe the proposed entropy method can scale to video data with a larger scale and more diversity in the wild.\n\nBy definition of \"weak reject\", I \"will not argue for my recommendation if the majority of other reviewers have a different opinion\".", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "The author can respond to the \"Weakness\" part of the review.", "reviewer_expertise": "2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper"}, "extra_text": {}}, {"review_id": "SPxCBWoK-o", "reviewer": "Reviewer_av9b", "summary": "This paper addresses an important problem in learning from video demonstrations, which is the presence of incorrect or suboptimal demonstrations in the dataset. Given a raw video input, it first performs sub-task segmentation on the video, and then use a feature generator to extract task-relevant features. Then it is put into an optimization problem solver to learn the demonstration weights, and eventually weight the demonstrations differently according to their demonstration quality. It is able to surpass the performance of IL models in the setting of imperfect demonstrations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Important problem setting, intuitive motivation of learning different weights for different quality samples\n\n- Interesting discussion on \"Leveraging Vs Discarding Incorrect Demonstrations\" - this mirrors the discussion between offline RL and BC on their performance on different datasets. It makes sense that this methods performs better than BC / BC-RNN for datasets with incorrect demonstrations, since it performs the filtering function that filters out suboptimal trajectories. \n\n- clear presentation of the overview figure about model pipeline and model architecture. nice visualization of segmentation results and how they correspond to the video frames\n\n- convincing results demonstrated on real, multi-step sequential tasks\n\nWeaknesses: \n\n- For Figure 6: For \"BC-RNN without Incorrect Demo\", are the number of *correct* demos the same with those in Demo-Dice? If so, I would expect Demo-Dice have similar performance with BC / BC-RNN, because Demo-Dice should be able to eliminate the effect of the incorrect demos?\n\n- For related work section on learning from suboptimal dataset, there should be discussions made to offline RL methods that learns from large dataset from different data quality. In particular, the weighted-BC line of work featuring advantage-weighted BC also learn weights for samples, and the weights from from advantage estimation. Such works include:\n\n1. Critic Regularized Regression\n\n    by Ziyu Wang, Alexander Novikov, Konrad Zolna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, Nando de Freitas\n\n2. AWAC: Accelerating Online Reinforcement Learning with Offline Datasets\n\n    by Ashvin Nair, Abhishek Gupta, Murtaza Dalal, Sergey Levine\n\n3. Offline Reinforcement Learning with Implicit Q-Learning\n\n    by Ilya Kostrikov, Ashvin Nair, Sergey Levine", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the paper addresses an important problem in imitation learning and have explained the motivations well. It gives a novel approach of learning demonstration weights that is applicable in real-world robotics setting. More clarifications and references are needed but in general the paper have convincing results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "As clarifications: \n\nRegarding the dataset labeling: \n- Does the dataset always come with ground truth labels of incorrect demonstrations? What happens when such labels are not available?\n\nRegarding the feature generator:\n- The feature generator extracts relevant task features from the videos. I'm curious whether this requires any assumption on the video demonstration type. The 50 salads dataset seems to have clear visual distinctions between different tasks. What if we are dealing with some robot manipulation tasks that does not involve large difference in visual scenes? How is the performance of this feature generator on common real robot tasks?\n"}}, {"review_id": "NolFv5hd5Wl", "reviewer": "Reviewer_key5", "summary": "The quality of demonstration data is crucial for imitation learning. This paper proposes a method to detect and filter out erroneous data from visual demonstrations (i.e., video streams) of sequential tasks, such as tea making or salad preparation. The proposed method first uses off the shelf methods to segment out subtasks from videos and then uses prior work to extract subtask-relevant features. Then, using the set of task demonstrations, it seeks to learn a predictive model of the next subtask given the current state and, in this process, identify which video demonstrations are erroneous (and therefore should be downweighted). The paper modifies the classic maxEntIRL optimization by posing a mini-max entropy optimization: find a weight for each demonstration that *minimizes* entropy of the sub-task predictive model (i.e., choose demonstrations that enables confident subtask prediction) while finding the distribution that *maximizes* entropy over the predictive model. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n*************\n–Interesting problem of identifying erroneous / low-quality demonstration data.\n\n–Valuable direction investigating erroneous high-dimensional data, like images.\n\n–Interesting approach of posing this mini-max optimization to determine which demonstrations are helpful vs. erroneous for predicting subtasks. \n\n--Interesting comparison of leveraging vs. discarding incorrect demonstrations when it comes to policy learning. \n\nWeaknesses:\n*************\n–The paper often makes claims without substantiating them properly with citations or evidence. The specifics of these instances are detailed below on a section-per-section basis. \n\n–The intuition behind why this mini-max optimization procedure should yield “good” weights that detect erroneous visual data could be strengthened. \n\n–The technical correctness of equation (3) – the dual problem to the mini-max game – seems questionable. For example, the maximization over the distribution P disappears in the dual problem. Also, the explanation for how this optimization is solved is unclear; solving mini-max games is not trivial in general, and it seems like the proposed method is solved in a way that will likely be an approximate (and not exact) solution to the non-convex game. Providing a more detailed derivation of the dual problem, and discussing the solution quality would strengthen the paper. For example, I believe that the dual problem is:\n$$\n(w^*, p^*(t|s), \\lambda^*, \\nu^*) = \\arg \\min_{\\lambda, \\nu} \\arg \\min_{w} \\max_{p(t|s)} \\mathcal{L}(w, p(t|s), \\lambda, \\nu) \n$$\nwhere $\\mathcal{L}$ is the Lagrangian which includes the entropy objective plus $\\lambda^\\top \\text{(feature expectation matching constraint)}$ plus $\\nu \\text{(valid probability distribution constraint)}$. It would then be nice to state what are the first order necessary conditions for optimality, for $w^*$, $p^*(t|s)$, $\\lambda^*$, and $\\nu^*$. \n\nIntro:\n\n–Nitpick: “Visual imitation learning …. Is probably the most feasible…” This is quite a strong claim. I’d suggest soften it by saying that “Visual imitation learning is a promising way for …”\n\n–“The state of the art…-almost all of which involve non-visual demonstrations–[is learning] weights…” Instead of [is learning] it should be [learn]. Also, it could be helpful to include a few citations of which state of the art approaches are being referred to here. \n\n–”...2) individual steps of a multi-step task - hereafter referred [to] as sub-tasks…” missing the [to] in the sentence.\n\n–”We bridge this gap…. Which does not have any restrictive assumptions.” Instead of making the broad (and somewhat unsubstantiated claim) that there are no restrictive assumptions in the proposed method, it would be good to focus on the explicit differences between the prior work and the proposed work in this context – for example, mentioning explicitly that the proposed method does not require expert labeling or a simulator for data augmentation.   \n\nSection 2:\n\n–”...where a single incorrect action may or may not cause the algorithm to eventually learn a completely incorrect policy….however, a single incorrect action can lead to an incorrect policy” This statement is not well-explained or well-substantiated. It seems as though regardless of if the task is multi-step or not, a single incorrect action can lead to either catastrophic failure or simply a marginally suboptimal behavior. \n\nSection 3.1:\n\n–”Here, q is a hyperparameter which does not require fine-tuning as we do not need very accurate sub-task segmentation.” Accurately identifying subtasks (and, related, the number of sub-tasks) seems like an essential prerequisite for the proposed method to work properly. This claim should either be removed or substantiated through some sensitivity analysis. \n\n–”The proposed framework is tolerant to typical errors in the existing video segmentation algorithms.” This claim should either be removed, or substantiated through sensitivity analysis / quantitative results. \n\nSection 3.3:\n\n–It would be helpful to provide intuition behind what the value of the weights means. If the weight is = 0, does this mean that the demonstration is not good and therefore is not used, and if the weight = 1, then it is an “optimal” demonstration and should be used to the fullest extent? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the problem addressed in this work is interesting, I recommended weak reject on the basis of the mathematical formulation of the core method seeming incorrect/incomplete (i.e., equation 3) thereby bringing into question the technical correctness of the proposed method and results, and the need to further empirically / theoretically substantiate some of the strong claims made in this paper (e.g., the \"triviality\" of picking thresholds of correct and incorrect demonstrations, sensitivity having accurate sub-task segmentation). \n\n--------------------------------------\n\nAfter discussion, I have changed my recommendation to \"weak accept\"", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Section 3:\n\n–Please clarify the technical correctness of equation (3) – the dual problem to the mini-max game. As stated above in the weaknesses, I believe that the dual problem should be:\n$$\n(w^*, p^*(t|s), \\lambda^*, \\nu^*) = \\arg \\min_{\\lambda, \\nu} \\arg \\min_{w} \\max_{p(t|s)} \\mathcal{L}(w, p(t|s), \\lambda, \\nu) \n$$\nwhere $\\mathcal{L}$ is the Lagrangian which includes the entropy objective plus $\\lambda^\\top \\text{(feature expectation matching constraint)}$ plus $\\nu \\text{(valid probability distribution constraint)}$. It would then be helpful to state what are the first order necessary conditions for optimality, for $w^*$, $p^*(t|s)$, $\\lambda^*$, and $\\nu^*$, and comment on how the solver works and if there is any concern about obtaining an approximate solution.  \n\nSection 4:\n\n–Table 1 is unclear on its own. For example, what do the rows of the table correspond to? Please clarify this. \n\nSection 4.2.3:\n\n–”The weights are distributed in a way that it is trivial to pick up two thresholds to differentiate optimal…” This is a strong claim. For example, the thresholds chosen in Figure 4 do not achieve a 100% accuracy in filtering out the optimal demonstrations vs. incorrect demonstrations vs. suboptimal demonstrations. Discussing a principled way to go about choosing a threshold, and what happens when the threshold incorrectly characterizes an optimal demonstration as incorrect, etc. would strengthen this claim. This same concern exists for Section 4.3.3 where the same statement is reiterated. \n\nSection 4.4:\n\n–”Fig 6 shows policy accuracy …When incorrect demonstrations are present in the training set….poor weights to incorrect videos.” The definition of policy accuracy relies on the number of correct actions chosen – where does the notion of a correct action come from? \n"}}, {"review_id": "1vLLWSUwlza", "reviewer": "Reviewer_JG5k", "summary": "This paper attempts to automatically detect incorrect video demonstrations for multi-step sequential tasks. The method learns a stochastic model of subtask sequence and generate weight for demonstrations using feature expectation matching with principle of maximum entropy. The approaches are verified with two datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n•\tThe paper is clearly written in detail.\n\n•\tThe authors provide a comprehensive survey over related work. The author also pinpoints the difference between the existing work and their new approach.\n\n•\tThe experiment analysis of no false negative detection is convincing.\n\nWeaknesses:\n\n•\tThe motivation emphasizes on detect visual incorrect demonstrations instead of suboptimal demonstrations. And in the statement the incorrectness often come from obvious missing steps or wrong step orders. One concern is that for such obvious mistake, it is very easy to have the demonstrator label it as incorrect or failure or directly delete the demonstration if they conduct a dangerous/incorrect behavior. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes to solve the problem of detecting incorrect visual demonstrations for multi-step sequential tasks. The paper is well-written and overall technical sound. The literature review is thorough and experiment results are convincing. The only problem is the necessity of leveraging the algorithm if human labelling is very easy.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1.\tAs mentioned in the “weaknesses” part, what are some situations where human cannot directly distinguish their incorrect behaviors or must spend lot of human efforts for doing that, so that detecting using the algorithm is necessary?\n\n2.\tFor sequential tasks, what if most parts of the demonstration are correct and only a part of it is incorrect? Could the work be adapted to address this by labelling incorrect for subtasks so that this helps keeping important information?\n\n3.\t Line 153 is a little bit unclear. Is it for each video segment or each video demonstration, that a set of ranked features is extracted? \n\nMinor issues: \n\n1.\tLine 97 The ‘proposed’ framework\n\n2.\tLine 235 Seq_2 is not fully bolded.\n"}}]

Author rebuttal:

Decision: Accept (poster)