Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces EgoTaskQA, a new benchmark with questions and answers for diagnostic analyses on spatial, temporal, and causal relationships between entities in goal-oriented task videos. The dataset is extended from the ego-centric multi-agent LEMMA dataset with four different types of QA annotations: descriptive, predictive, explanatory, and counterfactual. The paper also provides experiments with state-of-the-art models on the benchmark along with human performances for diagnoses on the reasoning tasks in goal-oriented task videos.
strengths: * Building upon an existing dataset considering what's missing in the existing works.
* The data annotation process is systematic and the authors provide the reasoning and procedures behind the annotation in the paper.
* Provides extensive evaluation of state-of-the-art models along with naive baseline and human performances.
* Provides detailed information including procedures of annotation, statistics, model setup, and datasheet in the supplementary materials.
* Has a website providing data exploration along with code and model checkpoints.
weaknesses: * The tables are hard to follow without bolding. For example, the highest numbers in Table 2 and the increase of performance in HCRN in Table 3 should be easier to read if bolded.
* Terms like "Most Likely" in Table 2 are vague and confusing. The authors should consider explaining more about the term to clarify.
* There are four columns but only two category columns in Table 3. The authors should consider adding an appropriate name for each column.
correctness: The dataset construction process is systematic. The authors provide the procedure of the data annotation process. The evaluation methods are straightforward.
clarity: In general, the paper is well-written. However, the authors should consider clarifying terms and tables for the first-time readers.
relation to prior work: Yes, prior work is well-cited with claims about the proposed dataset differing from prior works.
additional feedback: * Can this annotation paradigm extend to non-ego-centric or goal-oriented video understanding tasks?
* What is the role of diagnostic datasets like the proposed work? It would be interesting to show the diagnosis of more models more extensively.
review_title: Review of EgoTask QA

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Augmented two datasets based on Lemma dataset - a normal and indirect dataset. Both datasets had contributions of four question types (descriptive, predictive, explanatory, and counterfactual) across different question scopes (world, intents and goals, and multi-agent) from an egocentric view. The main difference between the indirect dataset from the normal dataset was the usage of words like "something" instead of object names to limit textual exploitation. They compared results between these two datasets and the indirect dataset results were worse than the normal dataset results, shedding light on the flaws of general SOTA video text alignment models as exploiting textual relationships rather than using sufficiently knowledgeable spatial temporal reasoning modules.

strengths: Good contributions of augmenting datasets, novel Q/A for a video QA dataset (as described in table 1), good results and discussion, limitations described are relatively small, good figures in general (aside from grammatical issues described in weaknesses), normal/indirect dataset idea was a great idea and showed good results, 
weaknesses: Poor grammar in benchmark vocabulary/spatial relations, certain figures' grammar is distracting from the paper (fig. 1 and 2), certain figures can be very difficult to understand due to complexity, doesn't explore model architecture/reasoning modules at all, lack of forward guidance for future papers/works, weak explanation for broader impact.
correctness: The claims made in the paper are generally correct. There are concerns, however, on the correctness of relationships, grammar, and terms in the datasets augmented. As these datasets were one of this paper's main contributions, this is a relatively big issue. However, to my knowledge other claims made in the paper are correct and the benchmarking setup is correct.
clarity: The text of the paper is pretty well written and clear. However, the figures have a lot of grammatical issues which makes apparent that the dataset likely also had substantial grammatical issues. These grammatical issues did at times reduce clarity.
relation to prior work: Yes, a table makes clear differences between this and other works as well as a comprehensive related works section.
additional feedback: The figures and figure captions need to be improved before publication for grammatical correctness as well as clarity. Also, the 3rd contribution mentioned of "devising challenging benchmarking splits over EgoTaskQA to provide a systematic evaluation" is unclear, it would be best to immediately mention the benchmark splits created (normal and indirect), as well as their purposes, rather than briefly mentioning them in the paragraph. Additionally, I was wondering if there was potential for mislabeled data as a cause (or one of many) for low benchmarking accuracy.
review_title: Overall great contribution to QA understanding in Ego Videos

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper created the EgoTaskQA benchmark for a direct evaluation of question-answering on real-world egocentric videos. The designed questions are aiming for video understanding of the human tasks from different perspectives, action dependencies and effects, intents and goals, and agents’ beliefs about others. 

This benchmark will help to evaluate agents capability in a comprehensive way with descriptive, predictive, explanatory ad counterfactual questions and thus help to develop more intelligent agents. 

strengths: - This paper introduces a new benchmark, EgoTaskQA that contains 40K balanced question-answer pairs. Those questions are targeted to understand the video from multiple perspectives to evaluate the agents’ intelligence.

- The questions are very broad, ranging from descriptive, predictive, counterfactual, and explanatory to evaluate the agent over the spatial, temporal, and causal understanding of the tasks.

- The generated questions in the proposed benchmark take care of the diversity and balance of each kind.

- The splits of normal and indirect of the dataset will help understand whether the model is using the correlation between the questions in the training and evaluation set without understanding the task, instead using language shortcuts in relations among questions.

- Evaluate model performance on question scopes, types, targeting semantics, and overall answer categories will show its overall capacity in understanding. 

- The ablation of object information and language-only shows the objects are very important visual clues in QA tasks. 

weaknesses: - The data scope is in-door goal-oriented tasks so this might cause limitations for a broader community.
correctness: The claims made are well supported by experiments in the paper. The evaluation methods and experiments design are appropriate and performed correctly. 
clarity: This paper is well written and easy to read and understand.
relation to prior work: The difference from the previous benchmarks is shown in Tab.1.

additional feedback: No.
review_title: Well designed QA dataset 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This dataset uses videos from the LEMMA dataset (egocentric videos of human-human / human-object interactions) for studying video QA. The authors extend LEMMA with annotations of objects, agents, and their relationships from Amazon Mechanical Turk. Then, the authors build causal dependency relationships between agents and objects in the videos. The questions in the dataset used in QA is then automatically generated with 4 types of questions (descriptive, predictive, counterfactual, explanatory). The dataset is evaluated on a set of 6 existing video models, showing a gap from the model to human performance.
strengths: - Based on Table 1, and the authors review of related works, there is a richer set of questions in this dataset compared to baseline (although I have some concerns about the questions themselves, see first point in weaknesses). The proposed question types (descriptive, predictive, counterfactual, explanatory) and the generated causal dependency relationships are interesting for understanding the performance of Video QA models.

- The authors benchmark a set of state-of-the-art video models on their dataset, with both normal split as well as a split for studying indirect references. These benchmarking efforts helps the community understand the strengths & weaknesses of existing models. 

weaknesses: - The full dataset is not available at this stage, even to the reviewers (please correct me if I'm wrong). Also, based on samples I'm seeing from the data, the automatically generated questions in this work seems a lot less clear than prior work such as AGQA [A]. From the author's website, some samples "Q: What does the person want watermelon to be for doing the action cut something using something in the video?" "Q: What will the status of the last object that has status change change to if the actor pour from something into something in the future?". Could the authors clarify why the generated questions here are less clear?

- Due to the above, I have concerns about the ability of this dataset to evaluate QA of the different question types proposed by the authors. Furthermore, comparing Table 2 here with Table 2 in AGQA, many of the question categories are fairly similar. Since this dataset (40K questions) is also a lot smaller than AGQA (3.6M), the significance of this dataset is not clear to me - would appreciate clarifications from the authors.

- There was just 1 trial run for all the benchmarking. While I understand that running video models are expensive, this does bring some concerns on the variability of results across runs and reproducibility. The ML reproducibility framework was not used here.

[A] AGQA: A Benchmark for Compositional Spatio-Temporal Reasoning
correctness: - The full dataset is currently not available to reviewers (as far as I'm aware), but some samples are available on the website.

- The benchmark evaluations are present in the paper (I just have some concerns on reproducibility, see documentation section below).

* Post rebuttal - The dataset is presented to reviewers now and authors have added more comments on the documentation & reproducibility.
clarity: The paper is generally well written, but I would appreciate more explanation on how the questions are generated in Section 3.2 (see my comments on weaknesses above).
relation to prior work: - The authors clearly discuss related works in many areas.

- I'd like more discussions from the authors on the significance of this benchmark with respect to AGQA (see weaknesses).
additional feedback: Could the authors let me know where reviewers can download the dataset? The link provided by the authors links to their public website, which currently requires a signed form for download and says "*During the review process, we only provide model checkpoints and temporarily forbid full data download".
review_title: Updated Review for EgoTaskQA 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work introduces a new video question answering benchmark that consists of egocentric videos with fine-grained annotations of object states, object-object, human-object and multi-agent relations, and causal action relations. The dataset also contains four types of question-answer pairs, including counterfactual and explanatory questions, in addition to questions that aim to capture intents, goals and object states and changes. Finally, the work compares 6 video question answering models on two data splits.
strengths: - The work significantly expands an existing egocentric dataset with 40K questions covering 4 different types. This dataset will be very useful to the embodied AI and VQA/VQC research communities.
- The paper comes with a comprehensive related work with a summary table that contrasts the contributions of this work w.r.t. relevant literature.
- Evaluation compares 6 VQA models. Several ablation studies are also present to demonstrate the usefulness of object information and language supervision (albeit masking with a common term vs. using the noun terms does not necessarily imply better action understanding and further analysis is needed)
weaknesses: - Unfortunately, reviewers are not able to access the data without exposing their identities. The website mentions filling in a form/license data agreement for downloading the data with a note:  *During review process, we refer to the website for data examples and temporarily forbids full data download.* It is unclear how the dataset will be distributed afterward, i.e., i) on which platform (the datasheet in the supplementary material mentions *dataset could be accessed on our website* but would that be restricted access?) and ii) whether all code will be released and scripts for ease of reproducibility of the reported experiments will become available.

- It is unclear how the correctness of answers generated by functional programs is verified. Similarly, how questions are machine-generated remains fairly unexplained. To my understanding, neither the code for the QA construction is not open-sourced nor the paper contains sufficient details on QA quality. Is the evaluation of the quality of the generated answers limited to the one described in lines 239-241 (randomly sampling 50 questions from each category)? It seems that some categories have very low accuracy.

- The observed performance increase of text-only models on object state change questions is worthy of further analysis. Perhaps the associations captured are not action-aware or context-aware, but rather simplistic linguistic co-occurrence patterns, as the experiments in Section 4.3 suggest. However, the performance drop between normal and indirect splits for text-only models is marginal. Would be nice to open source models, both for reproducibility and to allow future research to perform more exploration on what exactly language-guided models are able to learn.
correctness: It is difficult to judge correctness with a few examples presented in the paper and website. Most importantly, the paper does not contain a lot of information on whether it is safe to assume that template-based QA generation is accurate, or whether any additional verification has taken place. 

Other minor comments:
- Please further explain how spurious correlations are avoided by including direct and indirect references to actions and objects (lines 51-52)
- Please explain lines 130-132, specifically add a brief description of the vocabulary of relationships and state attributes and how manually filtering of irrelevant relationships is performed.
- What is the inter-annotator agreement for AMT workers on tasks described in the Multi-agent Relationships section?

clarity: The paper is generally well-written and easy to follow.
relation to prior work: There is some closely related work missing, in particular, similar datasets in embodied AI frameworks [1,2] and models that learn world dynamics (object state changes and object interactions) [3].

[1] Padmakumar, Aishwarya, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. "Teach: Task-driven embodied agents that chat." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 2, pp. 2017-2025. 2022.

[2] Gao, Difei, Ruiping Wang, Ziyi Bai, and Xilin Chen. "Env-QA: A Video Question Answering Benchmark for Comprehensive Understanding of Dynamic Environments." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1675-1685. 2021.

[3] Zellers, Rowan, Ari Holtzman, Matthew Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, and Yejin Choi. "PIGLeT: Language grounding through neuro-symbolic interaction in a 3D world." arXiv preprint arXiv:2106.00188 (2021).
additional feedback: - Have the authors considered an object-centric split to evaluate how models generalize to unseen objects?
review_title: Solid new egocentric QA benchmark but no option to review the actual dataset

Meta-review: metareview: The reviewers are positive regarding the high level of the contribution of the work for the NeurIPS 2022 Track Datasets and Benchmarks. The authors properly addressed all reviewers comments and concerns during the rebuttal period.
confidence: 5: The area chair is absolutely certain
Decision: Accept