Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce the benchmarking evaluation datasets GAIN constructed from the real world, whose videos contain steps from consistent categories with either COIN or Breakfast datasets but are out-of-distribution and belong to non-overlapping task categories. Furthermore, authors make attempts to alleviate performance drop on GAIN by utilizing causal inference and Monte Carlo method to intervene training samples by disassembling and reassembling.
strength_and_weaknesses: Strength:
1. The GAIN dataset is constructed with great effort. The basic principles to construct datasets are well-designed and in line with the definition of IAU generalizability. Also, the collection process of GAIN is in detailed description. As reported in the experiment section, performance on GAIN-C and GAIN-B undergoes an obvious drop compared to COIN and Breakfast, which validate the effectiveness of benchmarking model’s generalizability on GAIN. 
2. Formulation of the causal graph is also clear. The proposed intervention method has solid theoretical proof.

Weakness
1. Action segmentation papers always report metrics of both framewise accuracy and segmental edit distance and the segmental F1 score. It would be nice to see more metrics reported.
2. Models used in the experiment section are all pre-trained ones. I was wondering whether models trained from scratch on datasets (there have been multiple recent works focusing on this) still face a huge drop on GAIN. Also, whether they can be improved by causal-based methods is also unknown.
clarity,_quality,_novelty_and_reproducibility: The paper is well-motivated as well as well-written. Each section of the paper is easy to follow. Visualizations and tables are clear and informative. 
summary_of_the_review: The paper is well motivated and the proposed GAIN dataset has valuable contribution to the future instructional action understanding research. Therefore, my rating to this paper is "6: marginally above the acceptance threshold".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Summary:
- The paper proposed out-of-distribution (OOD) evaluation sets for the COIN and Breakfast dataset, together referred to as the new GAIN benchmark. To formulate the novel OOD evaluation sets, the authors ensured that the training set and the OOD evaluation set share the same set of steps, but different sets of tasks. 
- Empirical results were shown that the OOD evaluation sets are indeed more challenging as they require advanced model generalizability. 
- A causal-inference-based method, operated as data augmentation, is further proposed that allows the existing models to enhance generalizability and also perform well on the OOD evaluation sets. 

strength_and_weaknesses: Strength:
1. The OOD evaluation sets (i.e., the GAIN benchmark) are novel and valuable to the research community.
2. According to the experimental results, the proposed causal-inference-based method seems to be able to enhance generalizability of models if the instruction video understanding task is a step-level task.

Weaknesses:
1. Some assumptions need to be stated. For example, the proposed method assumes the constructed casual graph shown in Figure 4(a) is correct, so the proposed method assumes the step S would not affect the context steps Z (but one may argue this is not necessarily to be true in reality).
2. The proposed causal-inference-based method is effective only when the instructional video understanding task is focusing on the steps. I don’t think the proposed method would be beneficial if the instructional video understanding task is a more coarse-grained task such as video task recognition, domain recognition, etc., or tasks that require global context consistency, since the ressembed/intervened videos are not necessarily to be temporally semantically reasonable (because irrelevant steps are combined together to form an intervened video).
3. CrossTask might be the most similar dataset to GAIN, but the paper only briefly describes the difference between CrossTask and GAIN in the introduction. Some statistics to support that the shared steps are only a minority in CrossTask but this is not true for GAIN, would be great to have.
4. Some descriptions are not clear and additional explanations are needed.
- The causal-inference-based method is technically a data augmentation method, then, how would the relative size of the augmented data (relative compared to the original data) affect the performance? What is the current size of the augmented data? Since the ressembed/intervened videos are not necessarily to be temporally semantically reasonable, I assume the relative size of the augmented data has to be small in order to be effective.
- Why temporal boundaries of all steps of the training data is a prerequisite for finding unseen tasks (last sentence of the first paragraph of Sec. 2.2)? To my understanding, one would only need the set of steps in training, and use these steps to form novel combinations to obtain the novel task candidates, which are further filtered and collected via human annotations. 
- “... the larger number of categories is better” (from the Category Diverse paragraph). What kind of categories? For a given novel task, the collected videos of that task are supposed to be diverse. However, it is still not clear to me how the diversity is defined.
- “... apply the steps in the training dataset as the anchor steps and generate 10 step combinations with the caption clues”. What do you mean by “with caption clues”?
5. Failure case analysis of the proposed method is currently missing.
6. Limitations of the proposed benchmark and the proposed method were not discussed. 

Other comments:
1. In Figure 3 (a), there are some prominent spikes. It is interesting to know what are these top frequent steps in COIN Train, COIN Test, and GAIN-C.
2. Would the causal based method reduce the number of unique steps a video has in predictions (compared to before applying the causal-based method)?

clarity,_quality,_novelty_and_reproducibility: - Clarity: some descriptions in the paper are currently not clear enough.
- Quality: the quality of the paper is generally ok.
- Novelty: the contributions of the paper are novel.
- Reproducibility: Some additional details are required for one to reproduce the results in the paper, e.g., how to determine the size of the augmented data. In addition, the authors provided the code but I did not check. 

summary_of_the_review: The paper proposed the GAIN benchmark, out-of-distribution evaluation sets for step-level understanding tasks for instructional videos. A causal-inference-based method, operated as data augmentation, is further proposed that allows the existing models to enhance generalizability. The proposed GAIN benchmark would be valuable for the instructional-video-understanding community. 

However, clarity of the paper needs to be improved. Some assumptions, explanations, and implementation details were not clearly described. Though the proposed novel components of the paper are interesting, the paper needs more in-depth discussions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduce a new benchmark for evaluating the generalization ability of instructional action understanding models. The main idea is to collect existing steps in training dataset and re-assembling them as new task, then collect corresponding videos. Authors also propose an effective casual-based  method to improve generability.
strength_and_weaknesses: Strength:
* Introduce an interesting OOD evaluation definition: known steps but unknown tasks. As same steps have appearance bias in different tasks, it is suitable for generalization ability evaluation.
* Introduce a reasonable pipeline for constructing dataset.
* Propose an effective module to improve generalization ability.

Weekness:
* I suggest to simply introduce the definition of action segmentation / action detection tasks in Section 2 rather than Section 4. For readers who are not familiar with these two tasks, it is confusing to read Section 2. Meanwhile, I also suggest to simply introduce how a SOTA action segmentation method (such as TCN) works on Section 1 or 2. These background knowledge can help readers better understand the proposed task and method. 
* The dataset size still seems a little small for so many unseen tasks.
* For annotation of collected  videos, I am still confusing how un-defined steps are annotated. "Furthermore, explicit steps in a video do not need to exactly match those in its task – in other words, permuting and being a proper subset of the task are acceptable. " If a collected video contains un-defiend step, will this step be annotated? If it is annotated, will it be considered during evaluation? This is an important point and authors should give clear description.
clarity,_quality,_novelty_and_reproducibility: There is no reproducibility issue as codes provided. This paper propose a novel OOD evaluation for instructional action understanding. 
summary_of_the_review: In conclusion, this work proposes a reasonable OOD evaluation task for instructional action understanding, along with dataset and baseline method. My concern is mainly about the organization of the paper content as aforementioned. I give "accept, good paper" as my rating.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to study the out-of-distribution problem for instructional action understanding in videos, and therefore introduces a new benchmark for it. The OOD problem is well motivated and formulated. The pipeline for constructing the new OOD evaluation dataset from the existing COIN and Breakfast datasets is also reasonable. In addition, a causality-based method is introduced to improve the generalizability. The main concerns from the reviewers are about the clarity, presentation, and result discussion (so no critical concerns regarding the task and method formulation). Overall, the AC agrees with the reviewers that this work would be a valuable benchmark for OOD study for video understanding and thus recommends an accept. but we expect that the authors could revise the camera ready version according to the reviews by improving the presentation and adding more result discussion and analysis. 
justification_for_why_not_higher_score: Reviewers have some concerns regarding the clarity, presentation, and result discussion of this paper. The benchmark is useful but not broadly applicable.  
justification_for_why_not_lower_score: The OOD problem is well motivated and formulated. The pipeline for constructing the new OOD evaluation dataset from the existing COIN and Breakfast datasets is also reasonable. In addition, a causality-based method is introduced to improve the generalizability. Overall, the AC agrees with the reviewers that this work would be a valuable benchmark for OOD study for video understanding and thus recommends an accept.
Decision: Accept: poster