Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper presents an extension to the CLEVER dataset augmenting it with causal judgments of physical events. The causal judgments are collected in the form of graphs and then converted to a QA format to test existing methods.  Evaluation shows that existing methods that perform well on CLEVERER, seemingly fail on the proposed dataset. The paper suggests that this is because of improved diversity, human annotated causal judgments and smaller training set not allows models to overfit (I am not too convinced by the third argument. Question below.).
strengths: - Well written paper
- Well constructed dataset
- Evaluation indicates the dataset is challenging to existing models
weaknesses: - The paper might benefit from some additional analysis of the gap between current models and what the benchmark offers. One way to do this might be to present illustrative examples of common errors that current models make on the new benchmark. 
- I think the performance of humans on the final QA formulation of the data should be included. 
- The argument about lower training data size is unclear to me. What would be the performance of models on the CLEVERER dataset if their training set were similarly restricted? 
correctness: Yes. The data construction method is presented in detail. Along with relevant evaluations. 

One evaluation that is missing is the human performance on the finally constructed dataset. While the dataset is constructed using human annotations, there is some final automatic conversion of graphs to questions and threshold based filtering (lines 197 to 200). How can one verify that these choices are appropriate? 
clarity: Yes. the paper is fairly easy to understand. 
Minor point: In Figure 6, several plots of "Edge score" as the label for the x-axis. I don't believe those are correct. 
relation to prior work: Yes. 
additional feedback: -
review_title: Interesting, useful dataset. Paper might benefit from analysis of failure of popular models.

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This work proposes a new physical causal reasoning benchmark similar to the explanatory task of the CLEVRER. They discuss that template-based questions / descriptions lack diversity and causal relationships based on rule-based heuristics differ from human judgments. Different from the original explanatory task, they _iteratively_ collect _human-annotated_ causal event descriptions for a subset of CLEVRER examples. They use this collected data to train a trajectory-based event description generation model. Finally, they apply a refining / filtering process to the generated descriptions with the help of human annotators.
strengths: - This paper introduces the first human-annotated video question answering dataset about physical reasoning and causal relationships.
- The paper is well-structured, clearly written and very easy to follow.
- The paper includes detailed statistical analysis of the generated dataset.
weaknesses: - They claim that the existing benchmarks lack diversity in terms of events / descriptions, but they still use the same visual data with the CLEVRER.
- One could use synonyms (e.g. collide vs. hit) or different templates (e.g. passive vs. active voice) to increase the diversity of language-component. When I check Figure 6 (f), I see that these event types are actually very similar to each other (e.g. bump, collide and hit). Different verb forms are categorized as different individual categories (e.g. pushes vs. pushed, they should be one single category).
- Could the authors explain the second issue that is mentinoned in the abstract more? They only address this issue on the lines 29-33, but it is not clear "_how_ humans interpret _A is responsible for B_ differently from _A causes B_". This needs more explanation.
- The dataset is small-scale (the conclusion section refers it as a large-scale dataset). I'd rather design this dataset as a zero-shot evaluation platform (pretrain on CLEVRER, evaluate on the generated benchmark, and actually the authors performed this experiment).
- The paper puts emphasis on the causal event graphs (CEGs). The CRAFT benchmark also includes a similar representation which is named as causal graphs. There are no discussions about how they relate to / differ from each other. The paper does not even mention the CRAFT's causal graphs.
correctness: - Please just see the weaknesses section for the claims.
- The dataset is constructed in a sound way, I don't see any problems. 
- I think the experiments are well designed, it's good to have pretrain, from scratch, and pretrain-then-finetune experiments.
- Since the data is small-scale and the dataset expanded the original vocabulary, I think it'd be good to experiment with a stronger text encoder like BERT or RoBERTa instead of LSTM.
clarity: The paper is clearly written and very easy to understand.
relation to prior work: - It could also include a subsection for conditional text generation.
- The paper needs to make a comparison of CEGs and CRAFT's causal graphs.
- CATER should not include any form of questions, please double check.
additional feedback: I think it's a well-written paper, the idea is also good but the paper needs revision. Please see my comments on the weaknesses section.
review_title: CLEVRER Explanatory Task with Human Annotations

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: This paper proposes CLEVERER-Humans (CH) dataset, where physical events (such as causal relations between objects in case of a collision) are described using natural (human-annotated) language. A multistage pipeline is presented to efficiently collect these annotations. Authors conduct preliminary testing on question-answering task. According to authors. this dataset presents the challenge of interpreting diverse human-written event descriptions, making human-like causal judgments, and data-efficient learning (due to smaller training set). However, while this claim maybe true, as it stands, in my opinion, it may not be true or inconclusive (please see weaknesses).
strengths: Dataset is more diverse (partly resulting from subtleties occurring in human annotated descriptions) than previous counterparts. Pipeline to collect human annotations might be useful/applicable in other works.
weaknesses: Paper is not exactly self-contained. For example, L31-33, authors say "responsible for" is not same "causes", but did not explain why that's case, they simply mentioned a some references, and left for readers to go through those works and understand. 

In my opinion, motivation could have been given more importance. L21-22: "we can use natural language as a lens to evaluate machine understanding of physical events and causal judgments." Any particular why authors want to do this? Is it to generate better human-understandable explanations in case of explainable AI scenario? 

Dataset is rather small, and may not be sufficient for training models. Although in vision domain, smaller datasets when curated carefully have shown to be sufficient to train deep networks (YUP++ dataset). This might be indicative of problems in collecting the CH dataset? Please correctness section.

Information on the profile of MTurks is missing. On what basis were the MTurks hired? What's their background? Were they qualified to annotate the dataset? Did they have sufficient knowledge of English language in order to understand the annotation process and questions in the interface and annotate?

Was there any quality checks on human annotations? How many annotators annotated each individual sample? Was any sort of agreement among annotators measured in order to ensure quality checks?
correctness: I think quality checks in Stage 3 of human annotation is currently missing. This could lead to wrong annotations, and subsequently the models could be learning from rather ambiguous/noisy data, and that might be the reason for poor performance --- more so than the diversity in the dataset. For example, random baseline is performing more or less on par with trained models (or even better in some cases). Therefore, I don't think the findings are conclusive. I have recommended rejection for this reason. If I am mistaken, please correct me. I will revise my rating based on authors' response.
clarity: Please see weaknesses. 
relation to prior work: Sufficient, but maybe including prior work on natural language description generation might be better.
additional feedback: N/A.
review_title: Potentially Useful, but Probably Lacking Necessary Quality Checks 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors use the CLEVRER dataset consisting of video sequences of movable objects and associated physical events and use a sophisticated methodology to annotate these videos with human-generated judgments of causal relationships between the depicted objects. The authors introduce "causal event graphs" which are graph representations of the scenes. Nodes in these graphs are events (e.g., "red sphere moves towards green block" and edges indicate causal or non-causal relationships. To be compatible with existing benchmarks and methods, the authors also provide (question, set of possible answers) pairs.

The data collection methodology consists of three stages. 

(I) Collecting human-generated / corrected event descriptions for video sequences. 
(II) Event description (single and pairwise) generation using machine learning
(III) Human subjects improve/annotate the event descriptions and conversion of resulting CEGs to QA
strengths: - Well-written paper, that presented and clearly described methodology
- Novel graph representation of events (CEGs). An interesting data structure that could be a simplified / more accessible alternative to CGMs 
- Using human judgment for causal events 
- Well-designed and insightful experimental results showing that the benchmark is challenging to existing methods
weaknesses: I would like to see an (at least brief) discussion of the similarities and differences between Causal graphical models (CGMs) and the proposed causal event graphs (CEG). For instance, CGMs encode strong (binary) independence and conditional assumptions. In contrast, the questions in stage III allow for a graded type of causal relationship (a bit responsible, moderately responsible, etc.). What is the advantage of this graded way of assessing causality? It would also be interesting to study how consistent subjects are wrt these questions. 
I am also a bit worried about the wording here:  For instance, "quite responsible" vs. "moderately responsible". I could imagine that some subjects would rank the word "quite" to be weaker than "moderate". What type of thought process and justifications went into these questions?  A lot hinges on these words as you place the threshold at 4 ("quite"). Personally, I think it would have been better to drop labels for answers with scores 2, 3, and 4 and have 1: not responsible and 5: fully responsible or similar. 

Since this is a benchmark paper, I would have liked to have seen more of a discussion of data storage and persistence practices. yes, there is a website and a link, but no DOI, no versioning, etc. 
correctness: The dataset is constructed in a sound way and the evaluation and experiments are performed correctly. 
clarity: The paper is extremely well written. It was a joy to read it. 
relation to prior work: Prior work is, to the best of my knowledge, fairly discussed. A table contrasts and compares the proposed benchmark with prior work. 
additional feedback: Most of my feedback and questions are provided in the "weakness" section.
review_title: Human-generated causal judgements for physical events

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors build on the CLEVRER video reasoning dataset by introducing more varied natural language descriptions and human-annotated causal graphs for CLEVRER visual data. The paper proposes a strategy for producing annotations of this nature in a efficient manner using a combination of human annotations and model-generated labels. They illustrate the challenge presented by this dataset by comparing existing causal reasoning models’ performance on it to their performance on the CLEVRER dataset. 


[UPDATE: the authors have adequately responded to concerns]
strengths:  - The authors identify an interesting problem in the field: Many physical causal reasoning datasets have limited natural language annotations and causality annotations that do not effectively capture detail and nuance in the visual data.	

-  The authors apply an annotation method used in various works in pure NL to a visual domain

 -  The paper is well-written and has a detailed appendix. The website is informative and easy to navigate. The online annotation interface is very helpful.
weaknesses: - It is not clear that the CLEVRER-Humans dataset captures meaningful nuances in human causality reasoning that could not be captured by a well-designed heuristics-based annotation approach (which would be more cost-effective and less noisy). The paper also raises some questions about the quality of the produced annotations.

- Figure 6 shows that the new set of descriptions includes significantly more verbs than the original dataset (one of the primary contributions of the new dataset), but there is no evidence in the paper that these verbs successfully capture meaningful details in the visual data. What is the average agreement among annotators w.r.t. verbs? Would humans agree that the verbs chosen by the language model are the most accurate? In the stage 3 annotation process it seems that annotators only have the option to flag a description as technically incorrect, but not to flag a description as unnatural or poorly worded. Therefore, to a reader it may be unclear whether these model-generated descriptions successfully capture nuances of human causal reasoning.

- Furthermore, it is not clear what the descriptions “a bit responsible” vs. “moderately responsible” vs. “quite responsible” mean in the context of physical causality, there is no analysis of this with examples provided.
correctness: 
In the introduction, the authors claim they show that their dataset is challenging because of the diversity of event descriptions and data-efficient learning. However, based on the experiments section it is not clear that these reasons are the primary causes for the low model performance given the very small dataset size and domain shift from the CLEVRER dataset.

Other than this and the points addressed in the “weaknesses” section, the claims presented in the paper seem to be justified and correct.
clarity: The paper is mostly well written and easy to follow. However, the dataset creation process is described both in terms of a 2- and 3-stage process (in the introduction, and in the rest of the paper) which is difficult to follow.
relation to prior work: The authors provide a comprehensive overview of the limitations of existing physical and causal reasoning datasets. However, they omit relevant work in other areas of research (for example, cloze tasks in NLP, graphs for visual event reasoning, etc.), which would add helpful context.

As some examples:

“The language of causation”  argues for a new framework that departs from Talmy and later by Wolff.  Does that now mean anyone within this group of collaborating authors believes that literature is no longer worth referencing?

I disagree with the decision to stick with the binarization as used in CLEVERER as the default for this new dataset. Various works advocate for non-binary treatments, like COPA (which requires alternatives):

https://people.ict.usc.edu/~gordon/copa.html (COPA)
https://aclanthology.org/L18-1316/  (visual COPA)

Or the JOCI work, which generated entailing language then had humans similarly filter and label in a graded way, keeping the graded annotations:

https://aclanthology.org/Q17-1027/

Either way, it would be nice to see more discussion on when there was uncertainty by humans in their judgements.

additional feedback: Neat idea :)
review_title: Neat idea, could use more discussion of uncertainty, and fair pay

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This dataset collects human judgement annotations over events and causal relations of a previously existing physical events dataset called CLEVRER. The aim is to address the two issues in CLEVRER: the diversity issue in natural language description and the causal judgement discrepancy between human and heuristic-based generations. It collects 1724 events descriptions from 100 CLEVRER videos from humans, and then us them to train some neural networks to produce more human-alike annotations, in total to 1108 videos with 8581 descriptions and 21167 event relationship annotations, and generated 1076 question-answer pairs.

strengths: It is important for AI to understand physical events. In previous datasets, the description of what happens (events) and their causal relations are based generated manually by heuristics. But these annotations might not be precisely how humans understand physical events. This work aims to bridge the gap by collecting human annotations. This is the strength of this work.


(UPDATE: I moved my score from 4-reject to 8-clear accept after the author made addressed the core weakness - that is, how is like of this gap between heuristics. It is an important question worth the field's notice, and their dataset provides a good benchmark for it, so I give an 8.)
weaknesses: However, one central problem they aim to address - the difference between human understanding and heuristic generation of causal relationship and it’s importance - is not justified and presented clearly in the paper. Though this paper seems good elsewhere, if this fundamental flaw is not fixed, the whole benchmark could turned out to be meaningless and mislead other people’s research direction. This is why I give a rejection for now.

Specifically:

1, How is humans' causal judgment different from the template-generated one in CLEVRER? Why and how do they matter?
There seems to have no comparison and analysis. The reference paper given for this general topic does not address this issue directly. 

2, Also, why would humans have different causality scores on an event? What makes human uncertain? Do these different scale correspond to different types of causality or just because of lack of information? Does it matter?


Also notably:

3, if the point is to collect human diversity, doesn't it defeat the purpose to use NN to generate descriptions? I know it's slow and costly to let all tasks to be annotated by human, but this execution difficulty doesn't justify whether the NN method is valid. Even if you did some verb distribution post-processing to alleviate this issue, but how can you prove they are sufficiently diverse and human alike? Humans might just come up with some novel way to look at the event and describe differently.


correctness: As stated in the paper, there're only 1076 question answer pairs, so the dataset size is very limited to enable meaningful algorithm training (at least NN based). So it isn't clear to me that the evaluation is meaningful.
clarity: The content included in this paper is clearly written, but as mentioned above, some key justifications and analyses are missing. This caused trouble in understanding some key contribution of this work.
relation to prior work: The relation to prior work is clear.
additional feedback: One more question: you retrieved 1108 videos with 8581 descriptions and 21167 event, but why can you generate only 1076 question-answer pairs?
review_title: human judgements annoation over CLEVRER dataset with NN data augmentation

Meta-review: metareview: In this paper, the authors present video reasoning benchmarks. The feedback from reviewers is largely positive, while concerns mainly focus on the diversity of proposed benchmarks and clarification in the presentation, which the authors have tried to fix during the rebuttal period. I would recommend the authors to carefully address these in the revised version (especially those for NWDY and XFZ1)

Overall, I would recommend this paper to be accepted.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept