GAIN: On the Generalization of Instructional Action Understanding
Authors: Junlong Li
Guangyi Chen
Yansong Tang
Jinan Bao
Kun Zhang
Jie Zhou
Jiwen Lu
Venue: ICLR 2023 poster (2023)
Keywords: Action Analysis
Instructional Video
OOD Generalization

Abstract: Despite the great success achieved in instructional action understanding by deep learning and mountainous data, deploying trained models to the unseen environment still remains a great challenge, since it requires strong generalizability of models from in-distribution training data to out-of-distribution (OOD) data. In this paper, we introduce a benchmark, named GAIN, to analyze the GeneralizAbility of INstructional action understanding models. In GAIN, we reassemble steps of existing instructional video training datasets to construct the OOD tasks and then collect the corresponding videos. We evaluate the generalizability of models trained on in-distribution datasets with the performance on OOD videos and observe a significant performance drop. We further propose a simple yet effective approach, which cuts off the excessive contextual dependency of action steps by performing causal inference, to provide a potential direction for enhancing the OOD generalizability. In the experiments, we show that this simple approach can improve several baselines on both instructional action segmentation and detection tasks. We expect the introduction of the GAIN dataset will promote future in-depth research on the generalization of instructional video understanding.

Published as a conference paper at ICLR 2023
GAIN: ON THE GENERALIZATION OF INSTRUCTIONAL ACTION UNDERSTANDING
Junlong Li', Guangyi Chen Yansong Tang',JJnan Bao", Kun Zhang , Jie Zhou', Jiwen Lu TTsinghua University, "MBZUAI, CCrrnegie Mellon University, University of Alberta
ABSTRACT
Despite the great success achieved in instructional action understanding by deep learning and mountainous data, deploying trained models to the unseen environ- ment still remains great challenge, since it requires strong generalizability of models from in-distribution training data to oul-of-distribution (OOD) data. In this paper, we introduce benchmark, named GAIN, to analyze the Generaliz- Ability of INstructional action understanding models. In GAIN, we reassemble steps of existing instructional video training datasets construct the OOD tasks and then collect the corresponding videos. We evaluate the generalizability of models trained on in-distribution datasets with the performance on OOD videos and observe significant performance drop. We further propose simple yet effec- tive approach, which cuts off the excessive contextual dependency action steps by performing causal inference, to provide potential direction for enhancing the OOD generalizability. In the experiments, we show that this simple approach can improve several baselines
both instructional action segmentation and detec- tion tasks. We expect the introduction of the GAIN dataset will promote future in-depth research the generalization of instructional video understanding. The project page is hitps: jun long- 1i. gi mithub.... io/GAIN
INTRODUCTION
Instructional videos play essential role for learners to acquire different tasks. The explosion of instructional video data on the Internet paves the way for learners to acquire knowledge and for computer vision community training models, for example, human can train an action segmentation model to understand the video by the dense step prediction of each frame, or an action detection model localize each step. While a number of datasets for instructional action understanding (IAU) have been proposed over the past pears(Alayrae et al. 2016 Das et al., 2013b; Malmaud et al. 2015 Sener et al., (015) and growing efforts have been devoted to learning IAU models(Zhukov et al. 2019 Huang et al. 2017), the limited generalizability of models remains to be major obstacle real-world environments. One may ask a question "Suppose the model has learned how to inflate bicycle tires, does it know how to inflate car tires?" In fact, due to potential environmental bias between the training datasct and application scenes, the well-trained model might not be well deployed in an OOD environment (Ren et al. 2019), especially when instructional videos of interest to users are not involved in the finite training dataset.
to the deployment
To encourage models to learn transferable knowledge, it is desirable to benchmark their general- izability. Though this OOD generalization problem (Barbu et al. 2019: Hendrycks et al.) 2021 Hendrycks & Dietterich 2019) attracts much attention in the field of image recognition, such Ob- jectNet (Barbu et al. (2019) and ImageNet-R (Hendrycks et al., (002)), it has barely been explored for the IAU task. related problem is video domain generalization (Yao et al. 2021) (VDG) for conventional action recognition which focuses on domain generalization when changing the scene or background of the action. However, different from conventional action, the key obstacle to the generalization of instructional action is the distribution shift of action steps under different task cate- gories, which caused by the collection bias the datasets. Fig, 3 we show that the steps under different task categories have different distributions.
"Corresponding author

---

Published as a conference paper at ICLR 2023
Training Set
GAIN
Make Lamr Kekkab Gurepore sonnniings brush CHACs)
Roast Chicken: (prepare seasonings, remove intestines, sauce, balie)
Chocolite: L-s put into molds ou taklo -
Clean Fidr .?
Make Para fi.. Slll Coll (800 piecess Ballo.e
Make Jelly: fpropare seasonings, Into molds, take cut into piaces)
Figure 1: Two examples of constructing new OOD instructional tasks by reassembling the steps of in an-distribution videos in training datasets. For example, the OOD task "Make Jelly" consists of five steps: {prepore seasonings, stir, pul inlo molds, lake oul, cul. inlo pieces), where the "prepare seasonings" step is in the task "Make Lamb Kebab", the "put into molds" and "take out" steps come from "Make Chocolate", and the "stir" and "cut into pieces" steps are in the task "Make Pizza". The steps in GAIN are consistent with those the training set, with non-overlapping task categories. GAIN encourages models to transfer the knowledge learned from training data for OOD data.
Given the motivation that action steps are the key research objects of IAU and have distribution shift when task categories change, we propose a new evaluation strategy to benchmark the generalizability by re-constructing test task categories using the steps of training tasks and evaluating the models with these new task categories. In the reconstruction, we require that training and testing task categories different but step categories new testing task "Make Jelly" with existing step categories including the "prepare seasonings" step in the task "Make Lamb Kebab", the put into molds" and "take out" steps in "Make Chocolate", and the stir" and cut into pieces" steps in "Make Pizza". This construction is non-trivial since existing IAU datasets cannot be directly used. First, for most IAU datasets (such COIN (Tang) eet al., 2019)), the steps in different videos are not shared, therefore, we cannot construct testing data by splitting itself. Second, though CrossTask (Zhukov et al.) (019)) also collects cross-task videos with partial steps shared, these shared parts are only minority in the dataset (only 14% steps are shared, i.e. 73 are shared of a total of 517 steps) and most videos have steps that are not shared with others. Besides, because the related tasks are not fine-grained annotated, they cannot be used for evaluation. Furthermore, it is built to investigate whether sharing constituent components improves the performance of weakly supervised learning. It motivates us to collect and annotate real-world IAU dataset, GAIN. consists of 1,231 videos of 116 OOD tasks with 230 categories of steps, covering wide range of daily activities. All videos in our GAIN dataset are employed for evaluation. These videos can be split into two groups: GAIN-C and GAIN-B, as counterparts of the COIN (Tang al. 2019) and Breakfast (Kuehne et al. 2014 datasets, respectively.
consistent. As shown at the bottom of Fig. T we try to find
Furthermore, we propose a simple yet effective approach to enhance the generalizability of IAU models by cutting off excessive contextual dependency by performing causal inference. is inspired by the observation that model generalizability is inevitably influenced by short-cutting with a biased context. Compared with previous methods learn the temporal dependency among video steps by the temporal networks, such as TCN (Lea et al. 2017) applying hierarchy of temporal convolutions, we propose to reduce the over-dependency between steps to mitigate the negative effect from temporal context bias. For example, if the task "Inflate Bicycle Tires" is always observed together with the "bicycle pumps" during the training process, this knowledge will be difficult to transfer to the OOD task "Inflate Car Tires" with other inflaters. In our approach, infer causal effect, and present a Monte Carlo based method to approximate the distribution after "intervention". The method is evaluated with various baseline methods on both action segmentation and detection tasks, and is shown to produce consistent improvements.
apply the Back-Door Criterion to
Contributions. (1) We propose a evaluation strategy to benchmark the generalizability of IAU models by evaluating the models on the OOD tasks. (2) We build a real-world OOD instruc- tional video dataset, GAIN, where the OOD tasks are constructed by reassembling the steps of
2

---

Published as a conference paper at ICLR 2023
training datasets. (3) We propose simple yet effective approach, cutting off excessive contextual dependency by causal inference, which provides a potential direction to enhance generalizability.
2 THE GAIN DATASET
In this section, we introduce our GAIN dataset, video-based dataset covering a large range of daily tasks reassembled via a specific framework, which collects the tasks whose categories are different from training tasks to benchmark the generalizability of LAU models. For convenience, we call the tasks whose categories are same with the tasks in the training dataset as in-distribution tasks, and the ones with different categories as OOD tasks. Note that, when we mention iindiisributtonn and "OOD", the variables are steps but not tasks. It means the steps are "in-distribution"/ "OOD" under same/different tasks. To our best knowledge, GAIN is the first dataset to evaluate the generalizability of IAU models on the OOD steps. Fig. 2 shows the pipeline to construct the GAIN dataset. Below, we describe the details of our dataset, including how to benchmark the generalizability, how to collect the data and construct the dataset, and the basic dataset statistics.
2.1 PROBLEM DEFINITION
The generalizability of critical importance to the models for the deployment in a real-world en- vironment, especially for IAU systems, e.g., we expect the model can know how to "Inflate Car Tires" after learning how to "Inflate Bicycle Tires". For this goal, we propose to benchmark the generalizability of IAU models by building an OOD evaluation dataset, in which task categories is constructed by reassembling the steps of the training set. With this construction setting, the step categories are consistent and step distribution is changed.
The training dataset X = [X? contains nt instructional videos X", where each video is composed as a set of steps X?" ST. This step set can be formulated as S = { where n, is the number of steps in video. In the conventional experimental setting, both training and evaluation data are in-distribution, which is formulated as X V.i.d ~ XSource and X i.d XSoarce where Xoouree denotes the data distribution and X denotes the videos in the evaluation set. To benchmark the generalizability of models, we collect the videos of unseen tasks with seen steps, where videos the evaluation set are OOD tasks that can be formulated follows:
X i.dd XTarget, XT
iidd ~ XSource
s.l. 27 05.
where 22 and Dg respectively denote the set of all steps in the training and evaluation set, and XOOD denotes the collected OOD evaluation dataset. As shown in Fig. we show some collected videos of our GAIN dataset, where the collected videos follow different step distributions but share the same step space. Finally, we evaluate the models trained on X with the OOD evaluation dataset X Roon to benchmark the OOD generalizability.
Here we distinguish our defined OOD gen- eralizability evaluation from other evalua- tion strategies. We summarize the com- parisons with different evaluation methods in Table [ First, compared to conven- tional supervised and unsupervised meth- ods, we focus on the OOD evaluation to benchmark the generalizability of models. Second, UDA (Busto et al., 2018 Zhang al. 2019) aims to transfer the knowledge from source domain to some known target domain. It needs the domain index (e.g. the target data) for the training process to min- imize the domain gap between the source and known target. Compared with UDA, our OOD generalization further considers how to solve the problem without any domain indices. Though zero-shot recognition (ZSR) (Sener & Yao 2019 Wang et al., 2019) also focuses on the generalizability of models, it is too difficult to conduct zero-shot analysis directly for IAU models since ZSR requires the models to understand unseen action steps. This setting can be used for task- actions (e.g. classification) given extra descrip-
Table 1: Comparisons of different evaluation strate- gies. IV: instructional video; SL: using source label during training; TD: using target data during training: SC: steps are consistent.
Methods IV SL TD OOD SC
Supervised (Tang al.
2019)
/
X
X
V
Unsuporvised Miicch ct al.
2019
x
x
X
V
UDA (Busto al.
2018
7
V
X
Zero Shot (Sener Yao
2019
/
V
X
VDG Yao al.
202T
/
V
X
Ours
V
3

---

Published as a conference paper at ICLR 2023
Task Selection
Data Collection YouTube Q
I-4SC ZCOAA RRJA TADDE
Huppe
Training Set
Separated Steps
Filter Candidates
Search Videos
Collect Videos
GAIN
Figure 2: The pipeline to construct GAIN, which includes Task Selection (left) and Data Collection (right). Given an instructional video training set, we first separate the steps of these tasks and generate a large number of task candidates. Secondly, we filter out the unqualified ones according to three principles. Then, we search for YouTuube videos related to the selected tasks and download the videos, which embrace high relevance with queries, explicit instructions, and rich diversity.
tions (Wang et al. 2019). but not for complex action understanding tasks such as action segmenta- tion or action detection. Recently, many methods in the field of image classification have attempted to evaluate the gencralizability of models by collecting or generating the OOD data, c.g. Object- Net (Barbu et al. 2019) and ImageNet-R (Hendrycks et al. 2021). However, how to evaluate the generalizability of models for more complex IAU task has barely been visited. The most related one is VDG Yao et al., 2021). which evaluates the domain generalization ability of action recognition models when changing the scene background. Unlike VDG, our setting focuses on the distri- bution shift of action steps when task categories are changed in the target domain, which is more common in the field of IAU.
2.2
TASK SELECTION
To construct evaluation dataset consisting of diverse and high-quality daily tasks, we choose the largest fine-grained annotated dataset, COIN (Tang et al.] 2019), and the widely-used instructional video dataset, Breakfast (Kuehne et al. 2014), as the training sets.
How should we select the new tasks to benchmark the generalizability? We argue that the tasks in GAIN dataset require three basic principles follows:
Task Non-overlapping: The steps in our GAIN dataset are out-of-distribution, which requires the tasks in GAIN to be non-overlapping with those in the original training set. The model perfor-
mance on these non-overlapping tasks can intuitively indicate the generalizability.
Step Consistent: Despite the step distributions are different under non-ow overrappappnn tasks, we require that the categories of steps in the testing videos are consistent with the training dataset. On the one hand, with totally different steps, IAU will be even more difficult, which deflects our goal to benchmark the generalizability. On the other hand, the steps in the training set are common in daily life, which is of critical importance for IAU.
Category Diverse: The third principle, category diverse, encourages the annotators to discover more diverse data. In other word, we argue that the larger number of task categories is the better. For example, dataset (with 3 tasks) contains 2 videos of repairing a car, 2 videos of repairing roof and a video of repairing a television is more diverse than a dataset (with only task) with 5 videos of repairing a car. More diverse data indicates more reliable benchmarking.
With the principles above, as shown in Fig. 2 we first generate a large number of task candidates (i.e. step combinations) and then filter out the unqualified ones. Specifically, we apply the steps in the training dataset as the anchor steps and generate 10 step combinations with the caption clues, where each step combination contains 2~5 steps. The captions in instructional videos often mention steps that are not in the current task but closely related to other steps in the video, which could help to generate step combinations. In total, we generate more than 8,000 task candidates.
Here we provide details to show how we filter the candidates. Given candidate, we first check whether is logical, For example, if the candidate "lifting jack, replace the tire, remove the jack", it makes sense because it could form task "Change Car Tire"; it is not acceptable if the candidate is "lifting jack, replace the tire, add the seasoning" which may never happen sensibly in daily life.
Then according to Task Non-overlapping. we drop the logical candidates if they make up tasks that already exist in training sets. Since all candidates are composed of steps directly from training sets, they inherently satisfy the principle Step Consistent.
In the first round of annotation, we ask 11 annotators to go through these candidates and annotate whether the candidate is reasonable and satisfies the above principles. We filter out the candidates
4

---

Published as a conference paper at ICLR 2023
annotated unqualified by more than half annotators and finally select 147 candidates. In the second round, annotators are asked to name the new OOD task, refine current step combinations, and collect the videos from the Internet. By filtering out the rare actions, finally collect 1,231 videos of 116 tasks. In the last round, the annotators label the fine-grained temporal boundaries of each steps in videos.
2.3 DATA COLLECTION
Given the selected tasks, we search for YouTube videos related to the task names. We use a query with exactly the task name or the task name fol- lowing "how to" prefix to locate instructional videos, e.g. for the task "Change Car Tire" we use "change car tire" or "how to change car tire". To improve the quality and diversity, adopt several criteria to select videos including: high relevance with queries, explicit instructions, and rich diver- sity. We prefer videos more relevant to the queries and containing explicit instructions with pictures, since visual models are not able to only learn from narrations. Besides, videos with explicit steps to complete tasks are also favorable, although there might be no vocal instructions. Furthermore, ex- plicit steps in a video do not need to exactly match those its task in other words, permuting and being proper subset of the task acceptable. Moreover, if similar steps are witnessed in differ- videos of task, like "add salt" and "add sugar", we regard them the same step. With regard to the undefined steps, they are considered as the background and not further annotated. On the one hand, during the data collection stage, if video contains a long stretch of undefined but important steps, this video will be filtered out according to the principle of Step Consistent. On the other hand, videos with undefined meaningless steps are acceptable and these steps will be considered as the background. After collecting the videos, we utilize the annotation tool provided in (Tang et al., 2019) to label the corresponding step categories and segments.
COIN Train
Frequency Rate
COIN Test
W.A.M Wothou Wo Mam.n
GAIN-C
M...v
Step
The step distributions of the training dataset, original di-iittribttinn test dataset, and our OOD test dataset on COIN. Under the same task categories, the step distribution similar to the original training and test datasets. With different task categories in GAIN-C, the step distribution changes a lot, which supports assumption that steps are in-distribution/OOD with same/different task categories.
Figure 3:
2.4 STATISTICS
The final version of the GAIN dataset consists of 1,231 instructional videos related to 116 unseen tasks. Our GAIN dataset is a pure evaluation dataset to benchmark the generalizability of LAU models. Each task in GAIN contains 2~24 videos with an average of 10 vidcos. We annotate 6,382 action segments in GAIN with an average of 5 steps in each video. The average length of videos is 2 minutes and 30 seconds, and the average length of steps is 12 seconds. Totally, the GAIN dataset contains OOD videos of 51.2 hours for generalizability evaluation.
GAIN can be divided into two splits counterparts of COIN (Tang et al. 2019) and Break- fast (Kuehne et al.) 2014) datasets, and we name them GAIN-C and GAIN-B, respectively. COIN is large-scale benchmark with 9,030 training videos and 2,797 testing videos of 180 tasks. As its counterpart, GAIN-C contains 1,000 videos of 100 unseen tasks with length of 41.6 hours in total, where 5,238 segments are annotated. Fig. 3 shows the step distributions on COIN Train, COIN Test, and our GAIN-C, where horizontal axis denotes the different steps and the vertical axis denotes the frequency rates of these steps. We can observe the step distributions in original COIN Train and Test sets are similar, but different from our GAIN-C. It demonstrates our assuming that under different task categories, the step distributions are different. Breakfast is composed of more than 1.9k cooking-related videos of 10 breakfast routines such as "Make Coffee" and "Cook Pancakes". Accordingly, the GAIN-B split includes 231 videos of 16 OOD tasks with an average length minutes and 30 seconds. These tasks consist of 20 fine-grained action categories. We provide more statistical data and analysis in the Appendix.
METHOD
In this section, we first construct a causal graph for the IAU problem. Then, we introduce our method which applies causal inference to mitigate the negative effect of confounding context bias.
5

---

Published as a conference paper at ICLR 2023
3.1 CAUSAL GRAPH CONSTRUCTION
The widely researched action understanding tasks include action segmentation and detection, which both focus on the steps. Without loss of generality, we formulate the task as:
P( (S)),,
(2)
where S denotes step in the video X, Y is the prediction and fo represents the model. Then,
we formulate the action understanding framework in light of a causal graph g = (V.E). where the nodes V include the step S, model prediction Y, and context steps Z. Note that X = S U Z and 5 n - = 0 denote video X can be divided by query step S and context steps Z. The links indicate the dependence (computational but not strict causal direction (Liu et al. 202T)) between two variables. For example, S + Y in the causal graph indicates variable S is the cause of variable Y. We show the casual structure of the IAU problem in Fig. 4a) and explain it as follows:
- + indicates that the model prediction depends on both the step S and the context steps Z. For example, when recognizing the current step S, temporal models (e.g. LSTM (Hochreiter & Schmidhuber 1997) and C3D (Tran et al.] 2015)) always use the temporal context clues for current prediction, which leads to Y.
S z + Y denotes that the video context steps Z simultancously affects the steps and model prediction. Z + Y has been explained above and S +- Z is intuitive due to the temporal dependency of video. Thus we call the context bias Z is a confounder (Pearl 2009), which misleads the model focus on the spurious correlation, reducing the generalizability of the model. The casual graph describes the information flow during the inference. When S is being estimated, other context steps are Z, and since S is affected by Z curing the inference, Z points to S.
Then we show that the model prediction is misled by the spurious correlations of context bias when we only consider the likelihood PSY|S). As shown in Fig. (a), we re-write PSS|S) with the Bayes rule as:
P( = EyP(Y|S, 2) =15),
which denotes that the likelihood P(Y are influenced by P(Z = z[S). Now we use an exam- ple to show that P(Z = z|S) is biased. In the video "Inflate bicycle tires" current content S is "installing the nozzle" and the context Z is "using bicycle pump". The content S and the context Z are always observed together in the training process and thus P(Z using bicycle pumpl installing the nozzle) is higher. It leads the model to predict higher probability P(Y|S = installing the nozzle) when observing the Z = using bicycle pump and vice verse. However, when we apply the model to analyze the OOD video "Inflate car tires", where 7. "using bicycle pump" is absent, the model may be confused and consequently give wrong prediction. Addition- ally, although there is actually bidirectional effect between S and Z, we find that Bayes' theorem and Eq remains unchanged. In Fig. a), we apply S <- to highlight the bias caused by the con-fonder 5 + Z + Y, which is demonstrated by the Fig. 3
Motivated by the causal inference method (Pearl [209 Glymour et al. 2016), we propose to conduct intervention to alleviate the negative effect of context bias. In causal inference, the intervention is represented as do(.). Once intervened, a variable will have no in-coming links anymore and the previous in-coming links in the causal graph are cut off. As shown in Fig. 4 [l). when we intervene with the Back-Door Criterion in (Pearl 2009), i.c. do(S), the link between and Z is cut off so as the dependency. We formulate the model prediction process under the intervention:
P(Y (())o(S)) E_ 7 = 2) =),
where = 2 independent from S. Thus, after intervention, when the model predicts from do(S) to the label Y, it fairly takes every into consideration. Please see more detailed introduction and derive about Back-Door Criterion in Section 3.3 of (Glymour et al., 2016).
However, the intervention is a great challenge, since the prediction under this intervention is subject to the prior P(Z). which is difficult to compute numerically. Thus, we simulate conducting the intervention. We replace the numeric process with a sampling process and approximate the prior P(z) by the Monte Carlo method. As shown in Fig. 4 bb) we regard each step as an individual instance and put all the steps in lottery box, i.e. step pool. Statistically, all the steps in the training set form the population Z = { 2n with n categories, and then we can sample from this population. During the sampling process, the frequency of is not affected by X anymore and is
6

---

Published as a conference paper at ICLR 2023
S: Step
Y: Prediction
Z: Context bias
S1
Z
Proboblaay
Videos
Biss P( Σ, ((P(5)2)P(((5))
Dissemble
S
24
Step Pool
dx(X)
23
Protubilley
Z
Reassemble
Zz
Intervened"
Vidcos
S1
24
S
P(Y]do(S))
((
24
(
(b)
Figure 4: (a) The causal inference illustration for instructional action understanding. (Top) presents the original causal graph of IAU and the likelihood P(Y|S). (Bottom) shows the causal graph and the causation P(Y (o(()) after intervention. (b) Approximation with Monte Carlo method. We first dissemble the videos, where $1 only occurs with &222, and sample from the step pool. The prior P(Z) ) is approximated with the relative frequency and sampled steps constitute the interrveee"" videos, where could be observed with 23 24.
only related to the statistics of the training set. We approximate the prior with the relative frequeney:
P(Z 2') Exenl(2 z²)
||]||
where S2 denotes the sampling population, ||s2|| the sample size, and is an indicator function. We use the sampled steps to assemble "intervened" videos to learn causations of X on Y, instead of the spurious correlations due context bias Z. Fig. 4b) illustrates an example of this process, which only occurs with 21 &222 in the original videos and consequently models tend to learn spurious correlations of them. After dissembling and reassembling, observed with others, like 23& and the occurrence of is not dependent on Technically, our causal "intervention" can be regarded as a new kind of data argumentation, where we dissemble the steps and reassemble them video data.
the "intervened" videos, 81 could be
EXPERIMENTS
In this section, provide performance comparisons between the in-distribution dataset and out-of- distribution GAIN dataset, and assess the effectiveness of our causal approach on both action seg- mentation and action detection tasks. We conduct experiments
three datasets, where COIN (Tang ett al., 2019) and Breakfast (Kuehne et al.] 2014) are used for both training and testing. and our GAIN dataset only used for evaluation. As mentioned in Section 2.4 COIN and Breakfast widely used for IAU, so GAIN is split into two groups as counterparts of them, named GAIN-C and GAIN-B respectively. Please refer to Section 2.4 for more descriptions of these datasets. The implementation details, results, and analysis are described below, More experimental results and visualization examples can be found in Appendix.
4.1
IMPLEMENTATION DETAILS
We conduct experiments with the following baseline methods: (1) (ochhrrrreiiterr & Schmidhu-
ber 1997) one of the earliest and most popular deep models dealing with temporal modeling. (2) ED-TCNLLa et al.) (017) applies a hierarchic encoder-decoder framework with temporal convolu- tions, pooling, and upsampling to learn temporal patterns. We use 5 convolution layers for both the encoder and decoder, whose convolutional filters' sizes are 25. (3) [ResNet(He et al., (016)) adds a residual stream in the encoder-decoder framework. We follow the network structure depicted in (Lei & Todorovic; 2018) and adopt the same experimental setting ED-TCN's. s. (4) MS- +CSTTCN+(Li ct al. (2020) proposes a multi-stage architecture, which first generates initial predictions and refines them several times. Our implementation is built upon the publicly provided codebase.

---

Published as a conference paper at ICLR 2023
Table 2: Evaluation on COIN/GAIN-C with baselines and finer results across domains. ./- denotes
performances on COIN/GAIN-C. C means we apply causal-based method.
Domain
LSTM
C-LSTM ED-TCN C-ED-TCN TResNet C-TResNet MS-TCN++ C-MS-TCN++
Nursing
64.3/59.8
65.6/61.6 61.0/60.7
64.1/61.1
62.6/59.5
64.5/60.0
65.6/62.1
68.2/62.1
Vehicle
61.9/60.7 62.6/62.8 59.2/63.7
59.9/67.2
60.6/64.2
59.3/66.1
62.9/63.8
63.0/66.9
Leisure
58.0/60.4 59.3/62.9 1/60/
56.9/61.8
54,9/60.0
57.0/60.1
61.8/63.3
61.2/62.2
Gadgets
67 4/ 68, 68.6/63.9 64.9/63.2
66.4/66.5
64.2/63.9
65.6/64.2
68.3/61,4
68.2/64.7
Electrical
64.5/44.6 65, 65.9/48.8 63.7/47.5
62.7/47.8
63.9/44.6
63.2/49.0
66.2/44.4
65.4/45.3
Furniture
61.5/57.0
63.5/59.1 58.8/59.3
59.7/58.3
60.4/60.2
60.9/60.4
63.8/60.0
63.6/61.8
Science
61.4/41.5
61.9/45.8 56.5/37.7
57.8/42.9
58.0/42.1
58.4/41.7
61.6/38.8
63.1/47.7
Pets
61.0/52.0 64.9/54.2 61.9/48.7
63.5/52.7
61,5/51.2
61.6/48.6
63.7/52.2
64.6/52.0
Drink
65.1/50.6 66,8/53.5 61.9/50.3
62.5/54.4
61.8/48.8
62.1/53.5
67.1/46.1
66.5/54.2
Sport
69.8/63.5
73.4/63.8
6
69.0/67.1
66.8/65.3
66.9/50.3
72.0/49.5
10.9/51.6
Dish
70.3/47.3
71.8/51.3
64.5/48.8
66.9/50.9
65.7/47.5
67.8/66.1
71.8/59.4
70.3/66.7
Housework
62.2/61.4
62.7/59.6
60.6/61.6
59.9/61.0
59.4/59.0
60.8/61.1
61.5/59.8
64.2/61.9
Overall
63.9/52.3 65.4/55.1 61.2/52.7 62.2/55.3
61.6/52.4
62.2/54.6
65.5/51.8
65.6/56.0
For COIN/GAIN-C, we use the temporal video resolution at 10 fps. and extract S3D(Miech et al, 2020) features with a pretrained model on HowTo (Miech et al.. 2019) as the model input. And for Breakfast/GAIN-B we use I3D (Carreira & Zisserman 2017) features (pretrained on Ki- neties (Carreira & Zisserman 2017)) sampled at 15 Ips as the model input. As for the original evaluation set, we follow the default setting and present results on split 1. For all experiments, we employ
1 convolution layer to project the features into an embedding space, whose dimension is 64. Then, we apply different baseline methods to model the spatio-temporal clues. All settings are the same with both the baseline methods and our methods.
4.2 ACTION SEGMENTATION
Setting: Action segmentation aims at assigning each video frame with a step label. This task is key step to understand complex actions in instructional videos. We adopt frame-wise accuracy, which is the number of correctly predicted frames divided by the number of total video frames. For Breakfast/GAIN-B we also adopt edit distance and F1 score (Lea et al., 2017) at overlapping thresholds 10% to further measure the quality of the model prediction.
The frame accuracy on COIN/GAIN-C is shown (the Overall row).
Results:
Table 3: Results on Breakfast/GAIN-B for action segmentation.
in Table E
Method ED-TON
C-ED-TON
TReaNer
C-TRodNet MS-TCN++
CMS-TCN-+
1.TTM
C4STM
Acc.
42/4/113
44.5135
49.1/18.0
42,2/30.8
67.6/668
664432.1
478/25.5
45.7/37.9
We observe an obvious per-
F10:10
37.832
41,63.7
43.05.5
57.13.1
54.73.7
40003
4.80.5
Falt
39.0477
44.35.
1667
46.88.8
$4644.9
4644644
3.5/1
6.2/18
formance drop of approxi- mately 10.0 on average, although the steps of the two datasets are shared. Besides, Table 3 shows the results on Breakfast and GAIN-B. The performance gap is more obtrusive since the frame accuracy decreases by approximately 60% on average. It indicates that the current methods lack generaliz- ability on the out-of-distribution tasks. We observe that causal-based methods achieve consistent improvements in the OOD scenario. Besides, the Fl score and edit score, while other bascline methods work well. By qualitatively checking the predictions, we find that LSTM only can tell actions from the background, but fails classify the action categories correctly. It may be because the LSTM model is overly dependent on temporal relations and weakens the representational capacity.
find that the performances of LSTM are poor with
Quantitative Analysis: We compare our methods with the baseline methods to demonstrate the effectiveness. Table _ (the Overall row) and Table 3 summarize the performance comparisons of our methods with all four baseline methods including LSTMMHoccreiter & Schmidhuberi 1997). ED-TCN(Lea et al., 2017). TeeeNNe(HH et al., 2016) and MS- et al., 2020). For four causal-basee methods achieve significant and consistent improvements on
baseline methods, GAIN and obtains comparable results on the original evaluation sets with the frame accuracy metric. For example, after blocking the causal link between S and Z, Causal MS-TCN+ relatively outper- forms the baseline over +8.1% on GAIN-C. Moreover, on GAIN-B the causal inference methods relatively outperform the baselines over +69.3% on average.
Domain Analysis: Following the COIN dataset, we provide more in-depth analysis with experi- mental results across different domains in Table (Domains are described and showed in the Ap-
8

---

Published as a conference paper at ICLR 2023
pendix). An obvious performance drop from COIN to GAIN-C occurs domains 'Electrical' and 'Science'. The reason is that steps in these domains often follow a fixed process, which introduces the OOD tasks. The
strong contextual dependency models and results in poor performance causal inference approach alleviates these negative effects, for example, Causal LSTM relatively outperforms the baseline with large margin of +9.4% and +0044% domain 'Electrical' and 'Science', respectively. On domains like 'Housework', models obtain comparable results on COIN and GAIN-C, which because the video collection of GAIN-C is independent of the collection for COIN. So it is possible that the videos we find are easier to be segmented than those in COIN. The "Overall" results show that OOD test set more challenging.
Qualitative Analysis: We qualitatively analyze how our method contributes to the improvement of perfor- mance. Fig. [ demonstrates the visualization of two prediction examples on GAIN-C with MS-TCN++ and the corresponding causal method, where different col- ors means different step categories. Obviously, Causal MS-TCN++ achieves higher frame accuracy both two examples. At the top, the original MS-TCN++ predicts 6 kinds of steps for the video "Wash cat", while Causal MS-TCN predicts the same causations of steps as the ground truth. This demonstrates that our model does not predict the spurious correlations caused by the context bias but focuses on the step itself. At the bottom, we show an example of the video "Scalded shrimp". the causal one outperforms the baseline method with more smooth predictions.
Ground truth
MI-TCN+
Causal MS-TCN
(o) Wash
Ground truch
MS-TCN+-
Cansal MS-TON+
(b) Scalded shrimp
Figure 5: Visualization examples of action segmentation results on GAIN-C.
4.3 ACTION DETECTION
Setting: The goal of action detection is to detect a series of steps and output the temporal boundaries. is also important yet challenging task for IAU. We follow the evaluation protocol of (Lea et al.) 2017 (Singh et al.) 2016) by reporting the widely-used segment wise metric, mean Average Precision with midpoint hit criterion (mAP@ mid). Specifically, the criterion of mAP@ mid for a true positive is whether or not the temporal midpoint of the output interval is within the corresponding ground- truth action segments.
Table 4 presents the experimental results on COIN, Breakfast, and their counterparts. In this task, we choose LSTM as the baseline. From the training set to GAIN, we observe huge performance drop by more than 80%, which is related to the weak OOD generalizability of the baseline method. We also com- pare the causal methods with the baselines. Without any performance cost on the original evaluation set, the causal methods relatively outperform baselines over +33%/455% on GAIN-CIGAIN-BB B.
Results:
Table 4: Evaluation on training sets and GAIN for action detection. '/-² denotes the performances of "Training set GAIN'.
mAP@ mid
Methods
COIN
Breakfast
3.88/600
87.7 58.6/ 13.2
Causal 1.STM
3
5 CONCLUSION
In this paper, we have introduced a dataset, named GAIN, to benchmark the generalizability of IAU models. Our GAIN dataset contains 1,231 videos of 116 OOD tasks, which are collected by reassembling the in-distribution steps of the training set. Based on GAIN, we have proposed to eval- uate the generalizability of models with the performance on the OOD tasks. We have also proposed a causal inference approach to cut off the excessive contextual dependency for enhancing general- izability. We evaluate the generalizability of some widely used methods on GAIN and demonstrate that causal inference is potential direction to improve generalization. We will release this dataset to promote the world deployment of IAU models.
Limitation:
By benchmarking the generalizability GAIN, we offer testbed and expect work to develop models that can work in dynamic environments. However, the OOD data needs to be found, which is
Thus, the GAIN dataset cannot be omniscient or cover every aspect. Besides, while causal inference shows its potential to improve generalizability. designing algorithms for generalization is still open question. It is worth devoting efforts and we leave it future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work was supported in part by the National Key Research and Development Program of China under Grant 2017YFA0700802, in part by the National Natural Science Foundation of China under Grant 62125603, and in part by a grant from the Beijing Academy of Artificial Intelligence (BAAI).
REFERENCES
Bean-Baptiste Alayrae, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Unsupervised learning from narrated instruction videos. In CVPR, pp. 4575-
2016.
Jean-Baptiste Alayrac, Piotr Bojanowski, Nishant Agrawal, Josef Sivic, Ivan Laptev, and Simon Lacoste-Julien. Learning from narrated instruction videos. TPAMI, 40(9):2194 2208, 2018.
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Danny Gutfreund, Joshua Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019.
Biagio Brattoli, Joseph Tighe, Fedor Zhdanov, Pietro Perona, and Krzysztof Chalupka. Rethinking zero-shot video classification: End-to-end training for realistic applications. In CVPR, pp. 4613- 4623, 2020.
Pau Panareda Busto, Ahsan lqbal, and Juergen Gall. Open set domain adaptation for image and action recognition. TPAMI, 42(2):413-29, 2018.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, pp. 961-970, 2015.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, pp. 6299-6308, 2017.
Silvia Chiappa. Path-specific counterfactual fairness. In AAAI, pp. 7801-7808, 2019.
Pradipto Das, Chenliang Xu, Richard Doell, and Jason J. Corso. thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In CVPR, 2013a.
pp.
Pradipto Das, Chenliang Xu, Richard F Doell, and Jason Corso. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In CVPR, pp. 2634-2641, 2013b.
Sandra Eliza Fontes De Avila, Ana Paula Brandao Lopes, Antonio da Luz Jr, and Arnaldo de Albu- querque Araújo. Vsumm: A mechanism designed to produce static video summaries and novel evaluation method. PRL, 32(1))56-68, 2011. 1.
Hazel Doughty, Dima Damen, and Walterio W. Mayol-Cuevas. Who's 's better? who's best? pairwise deep ranking for skill determination. In CVPR, pp. 6057-6066, 2018.
Hazel Doughty, Walterio W. Mayol-Cuevas, and Dima Damen. The pros and cons: Rank-aware temporal attention for skill determination in long videos. In CVPR, 2019.
Yazan Abu Farha, Alexander Richard, and Juergen Gall. When will you do what? anticipating temporal occurrences of activities. In CVPR, pp. 5343-5352, 2018.
Andrew Forney, Judea Pearl, and Elias Bareinboim. Counterfactual data-fusion for online reinforce- ment learners. In ICML, pp. 1156-1164, 1164, 2017.
Robert Geirhos, Henn-K Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nat. Mach. Intell., 2(1 2(1))66556633 2020.
Madelyn Glymour, Judea Pearl, and Nicholas Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
10

---

Published as a conference paper at ICLR 2023
Yash Goyal, Ziyan Wu, Jan Ernst, Dhruv Batra, Devi Parikh, and Stefan Lee. Counterfactual visual explanations. arXiv preprint arXiv:1904.0745: 2019.
Chunhui Gu, Chen Sun, David Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. In CVPR, pp. 6047-6056, 2018.
Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. ECCV. pp. 505-520, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. 12261, 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -iitti--iitiitttionn generalization. arXiv preprint arXiv:2006.1624 2020.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In ICCV, pp. 8340-8349, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 978) 1780, 1997.
De-An Huang, Joseph J Lim, Li Fei-Fei, and Juan Carlos Niebles. Unsupervised visual-linguistic reference resolution in instructional videos. In CVPR, pp. 1192,2192, 2017.
De-An Huang, Shyamal Buch, Lucio Dery, Animesh Garg, Fei-Fei, and Juan Carlos Niebles. Finding "it": Weakly-supervised reference-awee visual grounding in instructional videos. In CVPR, pp. 5948-54888999 1148-5957,220 2018.
Haroon Idrees, Amir R Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge action recognition for videos "in the wild". CVIU, 155: 1-23, 2017.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. ICCV, pp. 706-715, 2017.
Hilde Kuchne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal- goal-directed human activities. In CVPR, pp. 780-787, 2014.
Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual faimess. In NeurIPS, pp. 4066 4076, 2017.
Colin Lea, Michael Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager. Temporal convolu- tional networks for action segmentation and detection. In CVPR, pp. 156-165, 2017.
Peng Lei and Sinisa Todorovic. Temporal deformable residual networks for action segmentation in videos. CVPR, pp. 6742-6751, 2018.
Shi-Jie Li, Yazan AbuFarha, Yun Liu, Ming-Ming Cheng, and Juergen Gall. Ms-ten++: Multi-stage temporal convolutional network for action segmentation. TPAMI, 2020.
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie- Yae Liu. Learning causal semantic representation for out-of-distrrbution prediction. NeurlPS, 34, 2021.
David Lapez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Léon Bottou. Dis- covering causal signals in images. In CVPR, pp. 6979-6987, 2017.
Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy. What's cookin'? interpreting cooking videos using text, speech and vision. arXiv preprint arXiv: /503. 150.00553.0 2015.
11

---

Published as a conference paper at ICLR 2023
Antoine Miech, Dimitri Zhukov, Cean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Hlotto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, pp. 2630-2640, 2019.
Antoine Miech, -an-tBatitte Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zis- serman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR. pp. 9879-9889, 2020.
Karthika Mohan and Judea Pearl. Graphical models for processing missing data. arXiv preprint arXiv:1801 03003,3583 2018.
Suraj Nair, Yuke Zhu, Silvio Savarese, and Li Fei-Fei. Causal induction from visual observations for goal directed tasks. arXiv preprint /91001791, 2019.
Rameswar Panda, Niluthpol Chowdhury Mithun, and Amit K Roy-Chowdhury. Diversity- aware multi-video summarization. TIP. 26(10):4712-472 2017.
Sunghyun Park, Seung-won Hwang, Fuxiang Chen, Jaegul Choo, Jung-Woo Ha, Sunghun Kim, and Jinyeong Yim. Paraphrase diversification using counterfactual debiasing. In AAAI, pp. 6883- 0891,2019,
Judea Pearl. Causal inference in statistics: An overview. Stat. Surv., 3:96-146, 2009.
Judea Pearl. The seven tools of causal inference, with reflections on machine learning. Commun. ACM, 62(3):54 60, 2019.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark DePristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In NeurIPS, 2019.
Alexander Richard, Hilde Kuehne, and Juergen Gall. Action sets: Weakly supervised action seg- mentation without ordering constraints. In CVPR, pp. 5987-5996, 2018a.
Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juergen Gall. Neuralnetorrk-iierrb:: A frame- work for weakly supervised video learning. In CVPR, pp. 7386-7395, 2018b.
Fadime Sener and Angela Yao. Zero-shot anticipation for instructional activities. In ICCV, 2019.
Ozan Sener, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena. Unsupervised semantic parsing of video collections. In ICCV, pp. 4488, 2015.
Bharat Singh, Tim K Marks, Michael Jones, Oncel Tuzel, and Ming Shao. A multi-stream bi- directional recurrent neural network for fine-grained action detection. In CVPR, pp. 1961-1970, 2016.
Yale Song. Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tysum: Summarizing web videos using titles. In CVPR, pp. 5179-5187, 2015.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representations using contrastive bidirectional transformer. arXiv preprint arXiv: 1r6iv:1906.05743 2019.
Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR, pp. 3716-3725, 2020.
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: A large-scale dataset for comprehensive instructional video analysis. In CVPR, pp. 1207-1216. 2019.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa-
tiotemporal features with 3d convolutional networks. In ICCV, pp. 4489-4497, 2015.
Pei Wang and Nuno Vasconcelos. Scout: Self-aware discriminant counterfactual explanations. In CVPR, pp. 9981-8990, 2020.
12

---

Published as a conference paper at ICLR 2023
Tan Wang, Jianqiang Huang, Hanwang Zhang, and Qianru Sun. Visual commonsense r-cnn. In CVPR, pp. 10760-10770, 2020.
Wenguan Wang, Xiankai Lu, Jianbing Shen, David J Crandall, and Ling Shao. Zero-shot video object segmentation via attentive graph neural networks. In ICCV, pp.
2019.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In CVPR, pp. 5288-5296, 2016.
Xun Xu, Timothy Hospedales, and Shaogang Gong. Transductive zero-shot action recognition by word-vector embedding. I/CV, 123(3)3309-3333 2017.
Zhiyu Yao, Yunbo Wang, Jianmin Wang, Philip Yu, and Mingsheng Long. Videodg: Generalizing temporal relations in videos to novel domains. TPAMI, 2021.
Huanyu Yu, Shuo Cheng, Bingbing Ni, Minsi Wang, Jian Zhang, and Xiaokang Yang. Fine-grained video captioning for sports narrative. In CVPR, pp. 6006-6015, 2018.
Weichen Zhang, Dong Xu, Wanli Ouyang, and Wen Li. Self-paced collaborative and adversarial network for unsupervised domain adaptation. TPAMI, 2019.
Luowei Zhou, Nathan Louis, and Jason J. Corso. Weakly-supervised video object grounding from text by loss weighting and object interaction. In BMVC, pp. 50, 2018a.
Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018b.
Dimitri Zhukov, Jean- Caan-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross-task weakly supervised learning from instructional videos. In CVPR, pp. 3537-3545, 2019.
Appendix
A RELATED WORK
Instructional Action Understanding: With the explosion of video data on the Internet, learners can acquire knowledge from instructional videos to accomplish different tasks. Many instructional video datasets have been proposed for different goals. such as action detection datasets (Caba Heilbron
2015] Idrees et al. 2017: Gu et al. 2018), video summarization datasets (De Avila et al. Song al.) 2015), and video caption datasets (Xu et al. Krishna et al.J al. 2017). To analyze instructional videos. diverse research fields presented in recent years including action segmentation (Richard et al.
2011
Gygli et al. 2014; Panda et al. 2017
2016
Yu et al. 2018 Miech et al. 2019;
2018a Sa b; Miech et al., 2020 2019 Sun et al. (219)). procedure segmentation (Zhou et al. | 2018b). 2019), action anticipating (Sener & Yao 2019 2013a), video grounding (Huang et al. 2017
step localization (Alayrac al.) 2018 Zhukov et al.
Farha et al.) (018). dense video caption (Das et al.
2018 Zhou al. 2018a), and skill determination (Doughty et al., 2018 2019). Despite the great progress on the in-distribution environment, it is a major challenge to deploy the trained models in the real-world environment.
Out-of-Distribution Generalization: How to generalize the trained model into OOD environments key challenge in machine learning (Geirhos et al. 2020). A kind of widely-used methods are zero-shot recognition (ZSR) (Xu et al. evaluation), where the categories of samples in the testing set are apparently different from the train- ing set. For instructional video, it is difficult to directly recognize unseen step categories. Another evaluation method is Unsupervised Domain Adaptation (UDA) (Busto et al. 2018; Zhang et al.) 2019), which trains the model with the data and annotations in the source domain and target domain index (e.g. unannotated target data). Compared with UDA, OOD generalization further considers the setting without target domain information. VDG (aoo et al. 2021) is OOD generation prob- lem which evaluates the models by the videos with changing the scene or background. However, this setting is more suitable for conventional actions. On the contrast, the key domain changing in
Brattoli et al. 2020) Wang et al. (019) (cross dataset
13

---

Published as a conference paper at ICLR 2023
the instructional video is not the scene but the distribution shift of action steps in changing tasks. The detailed comparisons are summarized in Table
Causal Inference: Causal inference (Pearl 2009 2019) plays an important role in machine learn- ing, which investigates causal effects of different variables. Recently, causal inference has been suc- cessfully applied diverse fields including computer vision (Lopez-Paz et al., 2017 Wang al. 2020), natural language processing (Park et al. (219), and reinforcement learning (Nair et al. 2019 Forney al. 2017), due to its ability for removing confounding bias (Tang al. 20200 Wang al.
2020), building explainable machine (Wang & Vasconcelos) 2020) Goyal et al. 2019), 9, promoting
fairness (Kusner al., 2017: Chiappa (019), and recovering missing data (Mohan & Pearl 2018).
In this paper, we apply causal inference to mitigate the negative effect brought by confounding
context bias to enhance the generalizability of IAU models.
B PREREQUISITE: CAUSAL MODEL
In this section, we provide some prerequisites of causal model that may help to better understand our causal approach. More details could be found in (Glymour et al. (0166).
Our task for video understanding is to predict the label of step based on the observation as P(YIS).
However, the context steps Z also affect the prediction. With the Bayes rule, can re-write P(Y (S)
as:
P(Y[S) = E.P(S =) 2|S);
(6)
which denotes that the likelihood P(Y[S) are influenced by P(Z =SS). However, P(Z = SS) is changed in the OOD setting. Now we use an example to show that P(Z zSS) introduces the observation bias. In the video "Inflate bicycle tires", current content S is "installing the nozzle" and the context Z is "using bicycle pump". The content S and the context Z are always observed together in the training process and thus P(Z = using bicycle pumplS installing the nozzle) is higher. It leads the model to predict higher probability P(YIS = installing the nozzle) when observing the = -using bicycle pump and vice verse. However, when we apply the model to analyze the OOD video "Inflate car tires", where Z "using bicycle pump" is absent, the model may be confused and consequently give wrong prediction. Therefore, we aim at mitigating the influence from Z on S. Before that, we first introduce some definitions in the causal inference to help understand, and the proofs can be found in (Pearl 2009 Glymour et al., (016)).
Definition (Intervention) An intervention represents an external force that fixes a variable to constant value (akin to random assignment an experiment), and is denoted do S $ meaning that S is experimentally fixed to the value s.
Definition 2 (Confound) Consider pair of variables S and Y. Muthematically, S and Y are confounded if
P(YIS) # P(Y)do(S)).
Definition 3 (Confounder) Z is a confounder (or S and Y are confounded by Z), if is associated with Y via paths in the causal graph that are not going through S.
For example, (a) in the causal model S <- 7 - Y, Z is confounder because (1) both S and Y are associated with it and (2) it is not on a causal path going through 5 and Y (3) nor a descendant of them; (b) in the causal model S + Z + Y. Z is not confounder but mediator (Z modifies the effect of S on Y).
Definition 4 (The Backdoor Criterion) Given an ordered pair of variables (S,Y) in a causal graph, a set of variables Z satisfies the backdoor criterion relative to (S,Y) if no node in Z is descendant of S; (2) Z blocks every path between S and Y that contains an arrow into S. If variables satisfies the backdoor criterion for (S, Y), ) then we can adjust the causal effect of S on
Y with
P(Y = y)do(S = s)) = P(Y = y|S = s, Z =))((Z =2).
2).
(8)
do-operator erases all the arrows that come into S to prevent any information about S from flowing in the non-causal direction.
14

---

Published as a conference paper at ICLR 2023
In other word, with this criterion condition on Z such that we (1) block all spurious paths between and Y: (2) don't disturb any directed paths from S to Y; (3) create no new spurious paths.
With these definitions, we can show that how the backdoor adjustment can help for our OOD task. When we intervene with the Back-Doot Criterion, i.e. do(S), the link between and Z is cut off so as the dependency. We formulate the model prediction process under the intervention:
P(Y]oo (S)) P.ySS = P(Z 2),
where = 2 is independent from The only difference between Eq 6 and Eq is that we change P(Z[S) to P(Z). which shows that Z is no longer affected by S. After intervention, when the model predicts from do( to the label Y. it fairly takes every into consideration. Thus, the backdoor adjustment can mitigate the negative effect of the biased confounder Z.
Hang curtain (drill, Install shelee, humguu
Install wood flourings (cut bourds, fit bourdhy luncel realls
Training
Hangupdocko (drill, lanacle railiy hangup
Evalation
Figure 6: An example of how to generate OOD tasks in GAIN. Given two in-distribution videos of the training sets, we can generate new OOD task by reassembling the steps of original videos. Best viewed in color.
C MORE DETAILS OF GAIN
C. DETAILED EXAMPLE
To show how to construct the OOD task in GAIN, in Fig. 6. we display detailed example about "Hang Up Clock" which can reassembled by the steps in the training tasks "Hang Up Curtain" and "Install Wood flooring". Specifically, the "Hang up curtain" task consists of three steps in- cluding {drill, install shelves, hang up), and the "Install wood flooring" task is composed of fcut raw boards, fit boards, knock in nails). Our collected OOD task "Hang up clock" con- tains the "drill" and "hang up" steps in the "Hang up curtain" task and the "knock in nails" step in another.
C.2 TASKS & STEPS
In order to present more details of our GAIN dataset, we show all the selected tasks with their cor- responding steps in Table 8 and Table IO We display these two tables in the end of Appendix because of the typesetting. The GAIN dataset consists of 1,231 instructional videos related to 116 unseen tasks. Each task in GAIN contains 2~24 videos with an average of 10 videos. We annotate 6382 action segments in GAIN with an average of 5 steps in each video.
C.3 SAMPLE DISTRIBUTIONS
Fig.1 7 and Fig. illustrate the sample distribution of GAIN-C and GAIN-B. Statistically, GAIN-C contains 1,000 videos of 100 unseen tasks with a length of 41.6 hours in total, where 5238 segments are annotated. The GAIN-B split includes 231 videos of 16 OOD tasks with an average length minutes and 30 seconds. These tasks consist of 20 fine-grained action categories.
15

---

Published as a conference paper at ICLR 2023
Figure 7: The sample distributions of GAIN-C.
25
#Sample
15
Makajally
MakeBaraanaan Maliii
MakelnnttantCCC offfe
CutCarambola
CutPapaya
MakelMi ner slice
MakeSoyMilk
Make
PeelDrange
curginger
FryMealShred
MaleeCocsuujulcc
CookPatatoshred
MakeM Shkhakee
MeatWiisccabbldde deserrabbiedggg
Figure 8: The sample distributions of GAIN-B.
C.4 DURATION STATISTICS
Fig. Dillustrates the duration statistics in both video-level and step-level of our GAIN dataset, where the average length of videos minutes and 30 seconds, and the average length of steps is 12 sec- onds. Totally, the GAIN dataset contains OOD videos of 51.2 hours for generalizability evaluation.
C.5 VIEWS ANALYSIS
In Fig. I0, we display the number of views YouTube across 100 tasks in GAIN-C, which can demonstrate that with the basic principles mentioned in section 3.2, the unseen tasks meet the need of website viewers statistically. We grab the number of views from YouTube by utilizing the Python module woutubesearchpylho.. We form a query with "how to" preceding the task name (e.g. how to paint the wall) to search for YouTube instructional videos related to the tasks. Then for each task, we only extract the first 20 results and sum up the numbers of views to represent the popularity of this task.
With approximately 767.9M views, "Make popsicle" becomes the most- viewed task and the last one "Dissolve effervescent tablet" still obtains 391.8K views. The number of views per task is 44.5M on average and the average number of views for the counted videos is 2.2M. The statistical results above prove that the selected tasks in our GAIN dataset all common daily tasks and satisfy the
16

---

Published as a conference paper at ICLR 2023
Distribution Video Duration
Distribution Segment Length
2000
Count
80
80
1000
46
500
200
200
300
400
500
40
50
100
lemporal Length (seconds)
Temporal Length (secondi)
Figure 9: The duration statistics the video-level (left) and step-level (right) of GAIN.
10
10
10a
105
104
10a
101
10a
NateWatarmmebnhaa
MakeeCaee
CutWatermeis
Makelelly
HavesPinns
MakkeHilkShsie
Playdowlind
aaySnowBoord
MakeNoodde
Egof Eggfriestt PiefRiiiico
NakeBread
MakelPzzz cas Dough
Modtwith Woatiithscramb hetiiisscammddddeee mmmbiddgggg
MlyCleeiMachire
PourACupCrriln
PackGoods
FryChicken
Nakekimmhhi
CurPaaaa
WashCarr
MokeMazheaPosatie
MakeCerfic MokeCa-liclieedd Colfiereeedddd
Dychair
GetE raPaacedd
Malke siice
OpenThedtee
CaininCCnniiiie CainTrick
Wel Weter BolleeMeee Mollleet
PaintThew
MakkePueerrrare
RoastChicken
MakedBooi
WashCat
Hight Hightungg
Makerdstait mellllolssl
Goffoee
CoolDumpling
GlanWindow
PeelOrange
ScaldedString
Roasted MesstesChhickeWWi Whink
Henw HenwClothesse lithesOut
Removehhncce
Remooe www.hheehlSlSeeeo
MakeWattrmnen
Perform perform//nnichingg
10a
104
10a
102
10
10a
WachFare
Pasts PactsC ouplet
Intraveroussn
TumpUpCarriis
WashThefa
rssallPaimii aaaatingnnnWa
MilkSoyMilk
ChangeBettery
UscDryer
Shatfutt
RoastSweetPoln
InstaliFeinorr
FredDDumpiig
ChangeFlux sorfuuuree ressenntuuee
enpoyuiiicttinnn HChtooOhhhhhii Chain
Change@oereell
Vaccinate
ChangoFilter
WashsColthes
Makeel Mocel
Unicedspere
PackBookCovei
ApplyGuue
Usceltal Uscdtalonne
dlsanCatorry
UsetesselSpre
HangACleck
ChergeMremaryCarde Car.emaarrrrrrraaaaaaaaaaaaa www.Dnnaammmiiiiimmmmmmoo
Kuplyyotion
Pumplup iopuppeeeaal meeaal
CurGingger
Make Seal
TryheettSrred
PlayMhusccWWiicCe CCC.cCCCCCRRR..... Plyye
Apply Applylu wullyyybbiic www.bbicciioonn www.baaiiiiinnnnii On Door
Pastehoster
UseTcalltticc
Apply HorlyHardNWWoOO
PalloCocoo.c itjucce
Changevioinsten Snniinniiiiinngggggg
DesembteDDesk
Cookhertol Mediiieeden
CurCarambol
easunaleBookshel
ChanocTheRel CranoctheReliicce CrrGGrr
Disoovett envescenneaallee Teblet
MakeHoney
Replece
Figure 10: The views distributions of tasks in GAIN-C on YouTube.
demand of website viewers. The learning enthusiasm for diverse tasks reveals the practical value of the generalizability of IAU models.
C.6 DOMAIN ANALYSIS
Fig. I shows the domain distribution (defined in COIN) of this split. GAIN-C covers 12 semantic domains, which demonstrates the rich diversity of GAIN dataset.
17

---

Published as a conference paper at ICLR 2023
Nursing and Care
Vehicle
Leisure and Performance Gadgets
Electrical Appliance
Furniture and Decoration
Science and Craft
Pets and Fruit
Drink and Snack
Dish
Sport
Housework
Figure 11: The domain distribution of GAIN-C.
Table 5: Parameter analysis on the learning rate on COIN/GAIN-C.
Frame Accuracy COIN
Learning Rate
Methods
GAIN
MS-TCN-+
5e-4
Causal MS TCN+
64.0
52.3
1e-3
MS-TCN-+
64.7
54.3
Causal MS TCN++
63,5
56.2
MS TCN++
65.5
$1.8
2e-3
MS-TCN++
Table 6: Parameter analysis different relative sizes of reassembled videos Breakfast/GAIN-B with frame accuracy.
Method
EDTCN
Causal ED-TCN
Step #
1.x
0.5x
1.5x
1.x
1.x
Video
1.x
1.x
1.x
0.5x
1.5x
Accuracy
42.4/21.3
44.5/35.4
44.5/31.1
44.5/30.7
43.9/29.1
42.1/37.6
D MORE EXPERIMENTAL ANALYSIS
D. PARAMETER ANALYSIS
We conduct experimental analysis both COIN/GAIN-C investigate the effect of hyper- parameter learning rate with MS-TCN+((L et al., 2020). As shown in Tablç5 with a learning rate of 5e the model performs unfavorable results both datasets, while an increased learn- ing rate, i.e. le - 3 or 2e 3, can notably improve its performance on COIN(+2.3% and +2.5% as well the performance on the -ii-triiibiitto tasks (+7.5% and +7.1%).
We also conduct experiments of different relative sizes of reassembled videos on Breakfast/GAIN-- and the results (frame accuracy) are shown in Table 6 For the number of steps, 1.0x, 0.5x, and 1.5x of Causal ED-TCN denote that we use times, 0.5 times, and 1.5 times step numbers as that of the original ED-TCN. So as the number vidcos. Our methods with different setting consistently outperform the bascline on both in-distribution and OOD scenario. Besides, we find that with the same number of augmented steps and videos the model achieves better overall performance, so we adapt this setting for all experiments.
D.2 VISUALIZATION RESULTS
In section 5.2, we have visualized the ground-truth annotations and the action segmentation results. Due the limitation of space, only visualized the results produced by MS-TCN++ method and its causal version. Now in Fig, [2 we illustrate the visualization for different methods includ- ing LSTM, ED-TCN, TResNet and MS-TCN++, to demonstrate the effectiveness of our approach.
18

---

Published as a conference paper at ICLR 2023
Scalded shrimp:
prepare and boil water, pour into the water, pour out, remove the shrimp shell
Ground truth
LSTM
Causal LSTM
ED-TCN
Causal ED-ICN
TResNet
Causal TResNet
MS-TCN++
Causal MS-TCN+
Figure 12:
Visualization of action segmentation results. The video is associated with the task "Scalded shrimp". Best viewed in color.
Ground truth
MS-TCN++
Causal MS-TCN++
Make Mixed Juice
Figure 13:
Visualization of failure case. The video is associated with the task "Make Mixed Juice". Best viewed in color.
We show results of the baseline methods and our corresponding causal ones on the task "Scalded shrimp" in GAIN-C. The consistent improvements for different baselines outtt--distributton tasks indicate that our causal intervention promotes the generalizability of models.
We analyze the underlying insights from the cases that the causal-based method has lower performance than the baseline, such as the video "Make Mixed Juice" in the GAIN-C dataset. Take closer look, find that there are some strong step dependencies, such as "juice the fruit" and "pour the juice". In this situation, context bias has positive effects the evaluation. Thus, we got the following insights, despite the better performance on average, our method encourages step independency, which has negative effects on the examples where strong step dependencies occur for both training and testing data. is reasonable since these steps are in in-distribution samples where the context bias has a positive effect, e.g., "juice the fruit" and "pour the juice" are successive steps in both COIN and GAIN-C, so the prediction with the concurrence of them is better.
Additionally, show failure case of our approaches in Fig. 13
19

---

Published as a conference paper at ICLR 2023
Table 7: Tasks and the corresponding steps in GAIN-C.
Tinks
Steps
Tinks
Stops
Apply
apply glue the wall and wallpuper, passi
MaleMineduuce
cut ingrechents, juice the oranges, pour the cup
level the wallpager
orange juice
apply the cleaming agent with rowell evenly.
MakeNoolles
some water the les, mis mo-
wipe off the cleaning agent
terials, knead the dough, cut the flesh
ApplyLotion
pour glue layer
face, wipe glue
Milieeeeerraane
lold pages, paint
pager
OnClaiin
apply the luhricust on the lock, wipe off the
redundint lubricunt, cleck old chain
MakcePapereur
draw outline, cur along the edges, fold bent paper
OnDior
ApptyLutrccanon
apply the lubricant un the lock, wipe off the redundant lubricant
MakaPizzaDoughh
mix the raw materials, add wome water to the ruh and drag the materials, the
flesh
AssembleBoolchhet
assemble the frane, install hontooaral boards, inssall vertical boards
MuliPPasterMMde
mix the raw matenils, soas them water, wait for the candle until concretion, remove add mstosials, add
to
AssembleDesk
install stand of the seor, install logs on the bed
MakePupacle
pour the ingredients anto the bowl, put the
candle wick into
the melted soap block into vessel, tike
freezing. mis the now materials
ChangeBanery
open the back cover, replace the battery, in- stall the back and waterproof ring
Matickobbereall
draw an outline, carve along the outtine, ze- the pool
ChanbeDuurell
screw off the surows used to fix the switch. reset the swith and screw on the screws used to fix the switch
MakeSayMElk
sunk and wash the rice, put yoguri, honey and other ingredients into the juiser, adi water the mix maserials, pour
vessel. juice
anges
ChangeFilter
take out the old filter, remove the cap of the fitter, install dish
MalleWatemeloooFrr
cleun the interior theyuumkiin, carve
filter, the
nlong the outline. diw out line
ThangeFlurrsceent Tube
lake oui the old bulb, issall the new bulb, light sbelthousing//uppen the light shellhhosing//upprr
Make TateeeWatemmooomJuuiee
cut oranges, juice the oranges, pour the or- aage juice into the cup
ChaigeMemaryCant
use the needle to open the SIM card slot. put SIM
MeatWithscrambslllggg
pour the egg into the bowl, str the egg, pare mest, the flesh. the rials, put
into the
press the SIM card
fry
ChangeTheRetine TheRetiliO GelPen
remove cogs, put lead into the pen, buckle cut off remove
OpenThuRoer
open the bottle carcfully
ChangeVaolinSiingg
string. the new string on the lower part of the guiar, fix the new
PackBoolCover
messure the size the packing poper, cut the packing paper, fold or best
on the head of the guitar, new string
adjust tightness
CleanCimtry
remove the my and paper hed from the
PackEoooss
remove cap, put in the buttery, close cower
hamster cage, clean toys and hamster
cages,
toy paper into
hamsster cage
CleanWindow
apply the cleaning agent with towel eventy. wipe the cleaning agent
Phint
the giue, apply glue to the wall and wallpuper
CookDumpling
knead the dluggh. flatten the dough, add in- gredients into cone, knead logother, mix materials, soak them in water, load the
PasteCouplet
apply glue the wall and wallpuper, pasle and level the wallgsper
CookHerrbulMeliueee
prepare and boil water, pour the noodles into the water and stir, filtrate with filter. pour glass
PasiePoster
apply glue the wall and wallpaper, past and level the wallpuper
PotaoeSbred
peel, cut into strips pioces, put uil Byy
PoelCrrange
remove the pool, flosh
CutCarambola
off the edge, the flesh
Pertona Vannshing
shuw the glass the audience, block the glass, show the vanished glass pre-swing, push curling
CoinTrick
Cuttiinger Cuilhppya
remove the poel, slice the pulp poel, hall, dig spoon, slice the palp
PayBowling PlayClawMachine
soeds with
money the vending muchine,
press the corresponding button, take out the goods
PlayMusic CDPlayer
CurWatemmeton
cut the odge, pest, cut half, slice pulp
out the laptop drive, press SIM
card sioe back, close the fuel tank cap
Thilet
DissolveElierveseeee
ingredients the add
PlaySnow Board
ski down from the till. ski up from the hill.
the vessel
the sky
20

---

Published as a conference paper at ICLR 2023
Table 8: Tasks and the corresponding steps in GAIN-C.
hioks
Steps
lasks
Sieps
DyoRair
Apply the shampoo or hair conditionr.. scratch the hair carefillly,
Pour AuppACuporWine
open the bottle conefully, pour in after mix
body
wash away, make the dry
LiginneRiiee
lase out some rice, soak and wash the nee, the bowl, sir the mix raw mtteeiaiss put
PumpUyCCaritre
screw oit the valie and open the valve, install the sir nordle,
pour the
the nuzcle righten the valve screw valve cap instal the air woole, pamp up to the lire, remove the air aozzle
fry knead the dough. flttien the dough, cur the ftesh, add ingredicuts into oone, knead to- gethor, put the fry,
PrioiDDmpling
PumpUhyThiRBal
torials, add some water to the
PryChickes
prepare scasoningg and side dishes, prepare cut the flesh, try gril
RefueiMotochike
open the fuel tank cap, insert oil gui in the pullthe oul gun close the fuel taak
cup
FryMensStmed
prepare mest, out the flesh, put in the oil to fry. lead the dish
RemowePhomsSrren
unserew the surows used to fix the screen,
pull out the screen connector and remove screen
GelEarsPiereed
drow lines to mark the haole, find the posi- tion the hole, drill with electric drill
Remowe/eeSheelor
cat uranges, take out the shell, rime the pot,
Scallop
remove the gill, up the cover
HangAClock
drill the wall, knoek in the nails, hang up curtains
OnPad
wipe the screen, paste protector on the screen
HangClutthsSut
wrap the by hamds, hing iruned
RovenoParking
drive the (orward. drive the back-
cloetes
ward adjust front and back position
HaveAPicoie
cleas the ground, Toy the cushion evenly. load the
Chicke
propare scasonings and side disless remove
insestines and bloed vessels, brush
sauce or speinkle seasoning, bake pizza
HighJump
begin to rum up, bugin jump up, Gall to the ground
Rousseel ChickellhiikkWWinng
suak them in water, ald now maserials, mix
the materials, bake pizza
Masaal/Paitinnaa/Wa
drill in wall, knock in nails, paste and level the wallpaper
RosstSweetPPtato
cleus the paspkkn, try or rosst bmise.
cut the bread, peel
TestallPviiters
let the faat side the new noedle towards jack insert the needle, screw on the screw
ScaldedSShiing
propare and boil water, pour the noodles
the water and remove shrimp shell, pour the cooked moodles
Introwennusinecciion
tie the nouriiuet, disinfeer, pull out the needle and press with
ShotPPr
pre-swing, throw the hammer out
MakeBanaaMiikk
peel, cut into stzips and picces, add milk, shake and juice, pour the orange juice into the cup. strow berries and other fruits the juicer
UnloudSpareTire
unscrew the screw, remove the tire
MakeBanzi
add ingredieuts into cose, kneedd togother
UseHalance
put the sample to be mearured on the bail
smoo, put the weight unril the bolance hall- ancad
MakeBraod
knead the dough, run the toaster and aijustt take out the slice bread
Userryer
put the clothes neatly on ironing table, use hair dayer to blow bot wall, flip the clothes repeally
MakeCake
pour the egg into howl, now mute- rials, mix row materials, put materials into mold, nua the toaster and adjust, take out chocolate
UsNaasalSprayy
wipp nowe, fill nostril with saline the same to the other nostril, shokk and stir. remove
Maket oncosslacee
tig out the seeds with spoun, put the ingre- dients into the cos. pour the orange juice put strawberris and other fruits into juicer, put yogurt, huney and other ingredients into the juicer, shake and juice
Use leaketile
pour the lea into the vessel. hest the icapot and wash the cup, close up the cover
MakoGarfieBeaad
bread, put filker on the bread slice, put a slice of bread in, nin the boaster and adjust, ukk out the
materials,
CseWahingMMachine
open cap, some cleaner to cleun and wet the lenves and take out the lanses, close up the cover
slice of bread
MakcHoneyLLmmn
cut ingredients, add some water the tea,
Vascinase
Lajection
disinticss inject-
put the ingredients into the can, mix and pickle
ing place, inject to the muscular. pull out the neadle and press
MikehnstanCooflte
and stir. ingredients in the cofles, mix the raw ma- lerials
poweler.
WashCurr
detergent and make bubble, clean surface, wipe off the cleaning apent
Mikelelly
pour ingredients into the howl, the
WashiCit
the body wash body away
mixture, take clocodate, cut into strips and pieces. put materials into mold
Table 9: Tasks and the corresponding steps in GAIN-C.
Taoks
Steps
Tasks
Steps
MikeKimch
pickie, clean up and soak, put ingredients into the can, mix ne materials, add some water
strips and picces,
WashClothees
desergent and make bubble, homoter cages, wrap the hair by hands
to the tes, shake and juice
poel, cui potato inso sinps, sozk them water, add raw materials. mix row materi- als. propare and boil water
Washflcce
and wash hands. apply cleansing to the face, wipe up the face
MikcM
shale juice, pour orange juice into the cap
WashThoPor
diserpent and make bubble, flush and wash the interior, scrub the willlt interioe and bodl water, soak them water, load the dish
MakeMLikica
yogurt honey und other ingredients into the mix the raw materials,
WaterBloulMett
after add ice cabes
21

---

Published as a conference paper at ICLR 2023
Table 10: Tasks and the corresponding steps in GAIN-B.
Tasks
Steps
CookPotatoShred
poel fruit, cut orange, pour oil
CutCarambola
cut orange
CutGinger
peel fruit, cut fruit
CutPapaya
peel fruit, cut fruit
FryMeethhred
cut orange, pour oil, take plate
Make MananaMka
peel fruit, cut orange, pour milk, squeeze orange, pour juice, put fruit to bowl
MocoCoocout
put fruit bowl, pour juice, squeeze orange
MakelnstaniCoffee
add teabag, stir coffee
MakcJelly
pour flour, stir dough, cut bun, put fruit to bowl
ShkeeiikShake
pour milk, squeeze orange, pour juice
MakeMilkTed
pour milk, stir lea, pour water, put Iruit bowl
MakeMixedJuice
cut orange, squeeze orange, pour juice
Male
stir fruit, pour milk, pour water, tea, squeeze orange
Make WatermelonJJice
cut orange, squeeze orange, pour juice
MeatWithSeramee WilhScrambledEgg
pour egg into pan, stir fry ege, cut orange, stir fruit, pour oil peel fruit, cut orange
PeelOrange
22


Reviews:
[{"review_id": "pY4nIJwAkaB", "reviewer": "Reviewer_Ct13", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduce the benchmarking evaluation datasets GAIN constructed from the real world, whose videos contain steps from consistent categories with either COIN or Breakfast datasets but are out-of-distribution and belong to non-overlapping task categories. Furthermore, authors make attempts to alleviate performance drop on GAIN by utilizing causal inference and Monte Carlo method to intervene training samples by disassembling and reassembling.", "summary_of_the_review": "The paper is well motivated and the proposed GAIN dataset has valuable contribution to the future instructional action understanding research. Therefore, my rating to this paper is \"6: marginally above the acceptance threshold\".", "strength_and_weaknesses": "Strength:\n1. The GAIN dataset is constructed with great effort. The basic principles to construct datasets are well-designed and in line with the definition of IAU generalizability. Also, the collection process of GAIN is in detailed description. As reported in the experiment section, performance on GAIN-C and GAIN-B undergoes an obvious drop compared to COIN and Breakfast, which validate the effectiveness of benchmarking model’s generalizability on GAIN. \n2. Formulation of the causal graph is also clear. The proposed intervention method has solid theoretical proof.\n\nWeakness\n1. Action segmentation papers always report metrics of both framewise accuracy and segmental edit distance and the segmental F1 score. It would be nice to see more metrics reported.\n2. Models used in the experiment section are all pre-trained ones. I was wondering whether models trained from scratch on datasets (there have been multiple recent works focusing on this) still face a huge drop on GAIN. Also, whether they can be improved by causal-based methods is also unknown.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LsyxmccI-M", "reviewer": "Reviewer_ESEp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary:\n- The paper proposed out-of-distribution (OOD) evaluation sets for the COIN and Breakfast dataset, together referred to as the new GAIN benchmark. To formulate the novel OOD evaluation sets, the authors ensured that the training set and the OOD evaluation set share the same set of steps, but different sets of tasks. \n- Empirical results were shown that the OOD evaluation sets are indeed more challenging as they require advanced model generalizability. \n- A causal-inference-based method, operated as data augmentation, is further proposed that allows the existing models to enhance generalizability and also perform well on the OOD evaluation sets. \n", "summary_of_the_review": "The paper proposed the GAIN benchmark, out-of-distribution evaluation sets for step-level understanding tasks for instructional videos. A causal-inference-based method, operated as data augmentation, is further proposed that allows the existing models to enhance generalizability. The proposed GAIN benchmark would be valuable for the instructional-video-understanding community. \n\nHowever, clarity of the paper needs to be improved. Some assumptions, explanations, and implementation details were not clearly described. Though the proposed novel components of the paper are interesting, the paper needs more in-depth discussions. \n", "strength_and_weaknesses": "Strength:\n1. The OOD evaluation sets (i.e., the GAIN benchmark) are novel and valuable to the research community.\n2. According to the experimental results, the proposed causal-inference-based method seems to be able to enhance generalizability of models if the instruction video understanding task is a step-level task.\n\nWeaknesses:\n1. Some assumptions need to be stated. For example, the proposed method assumes the constructed casual graph shown in Figure 4(a) is correct, so the proposed method assumes the step S would not affect the context steps Z (but one may argue this is not necessarily to be true in reality).\n2. The proposed causal-inference-based method is effective only when the instructional video understanding task is focusing on the steps. I don’t think the proposed method would be beneficial if the instructional video understanding task is a more coarse-grained task such as video task recognition, domain recognition, etc., or tasks that require global context consistency, since the ressembed/intervened videos are not necessarily to be temporally semantically reasonable (because irrelevant steps are combined together to form an intervened video).\n3. CrossTask might be the most similar dataset to GAIN, but the paper only briefly describes the difference between CrossTask and GAIN in the introduction. Some statistics to support that the shared steps are only a minority in CrossTask but this is not true for GAIN, would be great to have.\n4. Some descriptions are not clear and additional explanations are needed.\n- The causal-inference-based method is technically a data augmentation method, then, how would the relative size of the augmented data (relative compared to the original data) affect the performance? What is the current size of the augmented data? Since the ressembed/intervened videos are not necessarily to be temporally semantically reasonable, I assume the relative size of the augmented data has to be small in order to be effective.\n- Why temporal boundaries of all steps of the training data is a prerequisite for finding unseen tasks (last sentence of the first paragraph of Sec. 2.2)? To my understanding, one would only need the set of steps in training, and use these steps to form novel combinations to obtain the novel task candidates, which are further filtered and collected via human annotations. \n- “... the larger number of categories is better” (from the Category Diverse paragraph). What kind of categories? For a given novel task, the collected videos of that task are supposed to be diverse. However, it is still not clear to me how the diversity is defined.\n- “... apply the steps in the training dataset as the anchor steps and generate 10 step combinations with the caption clues”. What do you mean by “with caption clues”?\n5. Failure case analysis of the proposed method is currently missing.\n6. Limitations of the proposed benchmark and the proposed method were not discussed. \n\nOther comments:\n1. In Figure 3 (a), there are some prominent spikes. It is interesting to know what are these top frequent steps in COIN Train, COIN Test, and GAIN-C.\n2. Would the causal based method reduce the number of unique steps a video has in predictions (compared to before applying the causal-based method)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8rn86p4Q7z", "reviewer": "Reviewer_2UkL", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduce a new benchmark for evaluating the generalization ability of instructional action understanding models. The main idea is to collect existing steps in training dataset and re-assembling them as new task, then collect corresponding videos. Authors also propose an effective casual-based  method to improve generability.", "summary_of_the_review": "In conclusion, this work proposes a reasonable OOD evaluation task for instructional action understanding, along with dataset and baseline method. My concern is mainly about the organization of the paper content as aforementioned. I give \"accept, good paper\" as my rating.\n", "strength_and_weaknesses": "Strength:\n* Introduce an interesting OOD evaluation definition: known steps but unknown tasks. As same steps have appearance bias in different tasks, it is suitable for generalization ability evaluation.\n* Introduce a reasonable pipeline for constructing dataset.\n* Propose an effective module to improve generalization ability.\n\nWeekness:\n* I suggest to simply introduce the definition of action segmentation / action detection tasks in Section 2 rather than Section 4. For readers who are not familiar with these two tasks, it is confusing to read Section 2. Meanwhile, I also suggest to simply introduce how a SOTA action segmentation method (such as TCN) works on Section 1 or 2. These background knowledge can help readers better understand the proposed task and method. \n* The dataset size still seems a little small for so many unseen tasks.\n* For annotation of collected  videos, I am still confusing how un-defined steps are annotated. \"Furthermore, explicit steps in a video do not need to exactly match those in its task – in other words, permuting and being a proper subset of the task are acceptable. \" If a collected video contains un-defiend step, will this step be annotated? If it is annotated, will it be considered during evaluation? This is an important point and authors should give clear description.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster