Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This work presents an interesting new dataset supporting the research on visual question answering that requires causal reasoning about physical forces and object interactions. Compared to previous work [23, 18, 24], the proposed dataset contains more complex scenes. The data is generated automatically from virtual scenes. 3 sets of baselines are established for the newly created data. However, there are several major weaknesses, mostly regarding the baselines and the data bias and quality.
strengths: The main contribution of this paper is the creation of a new visual question answering that requires causal reasoning about physical forces and object interactions. This dataset differs from previous work [23, 18, 24] as it contains more complex scenes, i.e., the moving objects can interact with both static and dynamic scene elements while in [23, 18, 24], they can only interact with dynamic scene elements. I believe the move from simple environment to a more complex one is a reasonable and natural next step for the community. The dataset, if of good quality, could be useful for the research community.  
weaknesses: - The fact that `Text only` models perform so well (e.g. in Table 1, especially under hard setting, the LSTM model is better than or comparable to a few single frame and video models) might suggest there is a certain degree of language bias in the dataset, i.e., the questions and answers are correlated. While the authors put some efforts on balancing the answer distribution (L206), this correlation is not studied. It would be good to identify and eliminate these biases before releasing the dataset.

- Follow the last point, I am also curious how the models perform if using stronger language features/embeddings like GloVe and BERT. I feel the text only models could be further improved using these features/embeddings.

- Data quality. See Correctness section where I pointed out potential data quality issue.
correctness: - The answer to the 2nd Counterfactual question in Figure 1 does not look correct to me. For example, if the large cyan triangle is removed, the small gray box may still end up in the platform (where the large cyan triangle initially sets.). To me, I would answer "Not sure, removing large cyan triangle increases the chance that the small gray box enter the basket, but this chance is not 100% since the small gray box may stuck in the platform."
- Following the question above, I am a bit worried about the overall human performance of this dataset. I am aware that the authors performed a filtering process for the final dataset in L163, but this filtering is not described in detail. The relatively low human performance in Table 1 also makes me wonder the potential quality issue of the data.
- The paragraph starts from L310 makes a few false claims about Table 1, e.g., "Among the evaluated baselines, the text only models perform the worst,". As noted in the Weaknesses section, the best text only model `LSTM` is better than one Single frame model and two video models under hard setting.
clarity: Some parts can be further improved:
- Figure 1: Please add a legend for bucket as well. I was initially confused that the whole figure represents the bucket, as it looks the same as the smaller one.
- Figure 1: Showing the whole video in a single frame looks nice and succinct, but it lacks a bit clarity: I cannot tell whether the small brown circle collided with the small yellow square or not. Thus cannot answer the 1st Enable question. Similar for the 2nd Enable question. I recommend the authors to represent it as a sequence of frames like Figure 3.
- In L 346, could you detail why only consider the ones who responded at least 75% of the questions? How many questions are left after this filtering? If the pool is too small, the human performance might not be accurate.
relation to prior work: There are 3 prior work TIWIQ [23], CLEVRER [18], and CLEVR_HYP [24] that are most similar to this work. Compared to all the 3 work, this paper is unique as it involves more complex scenes (L71). By compare the paper and the 3 references in detail, I notice the main difference lies in that this work contain both static scene elements and dynamic scene elements to interact that a moving object can interact with, while in the 3 references, the moving object can only interact with other dynamic elements. I believe this part can be better explained and added to the related work section — so readers can quick get a sense of where the difference is, it is rather hard to get it from a `explained later` sentence, "however the scenes in our benchmark are more complex, as explained later." (L71).
additional feedback: - I noticed in L108 the train/val/test splits in the hard setting is based on different scenes? Could you confirm that different `scenes` means different `environments``?
- I don't quite see a difference between `Cause` and `Enable` question types. Maybe they should be merged? Otherwise, please distinguish them clearly.
- Could you explain why TVQA+ is trained using full precision? It should be good to add one more comment explaining this in the paper.


---------

-- after discussion, final review (copied from my reply)


Thanks for responding to my question and concerns. I am convinced the language bias is not an issue for this work. However, there is one question that is not answered or not answered clearly. In `Additional Feedback`: `I don't quite see a difference between Cause and Enable question types. Maybe they should be merged? Otherwise, please distinguish them clearly.` I am aware that the authors responded to a similar concern from Reviewer `Bpwc`, but that still looks confusing to me. For example, in Figure 1, for the 1st `Enable` question, I can replace the word `enable` with `cause` and the question would still be valid, in that case, should it be categorized as a `Cause` question? Likewise, I can also replace the word `cause` with `enable` for the 1st `Cause` question, and then it might be categorized as an `Enable` question. With this confusing categorization, I don't think the current evaluation of these different categories is meaningful. For this reason, I'll keep my original rating, and encourage the author to address this in the 2nd round.

BTW, for the negative value with high magnitude, you can probably replace it with a smaller value then fp16 should work well. I.e., replace `1e10` with `1e5`.
review_title: Novel dataset but with some bias and quality issue

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The authors develop a dataset for Visual Question Answering (VQA) that is motivated by "intuitive physics": humans' natural ability to reason causally about physical forces and events. They evaluate a number of baseline and state-of-the-art methods for question answering and show that these methods do not perform well on this dataset relative to human reasoning abilities. This suggests that their dataset will be useful for detecting and quantifying advances in this domain.

If I understand the Related Work section correctly, this represents a valuable contribution toward the development of AI that is able to mimic humans' physical reasoning abilities. The context vis-a-vis previous datasets merits some clarification, and I have some concerns about the language used in the dataset and about the need for additional quality control via human performance baselines.
strengths: The construction of scenes and the corresponding question-answer pairs are usefully grounded in cognitive and linguistic theories that attempt to account for humans' physical reasoning capabilities. The construction of the dataset is clear and well principled. I am not familiar with the related work, so it is difficult for me to evaluate the novelty, but based on their description I understand this to be a useful and novel extension to existing VQA datasets.
weaknesses: This seems like a self-evidently valuable type of dataset to construct. I have two concerns with this work that I hope the authors can address.

First, insofar as this dataset is intended to measure advances in AI's ability to mimic human intuitive physics, it seems vital to obtain more robust measurements of how well humans can reason about these scenes. The baseline study that they conducted is useful and informative, but it's small (481 questions, 101 adults). In addition to measuring individuals' accuracy, it seems important to measure inter-annotator reliability. Perhaps these measures wouldn't be needed if the dataset were intended to be prescriptive with respect to physical causal reasoning, but the cognitive theory behind it suggests that it's intended to be descriptive of human reasoning processes. 

Second, I have some concerns with the language in the question-answer pairs. (For context, I'm a native English speaker.) (1) The "do" construction in English signals a "Yes"/"No" question. It sounds odd to respond "True"/"False" to these questions. (2) I'm not surprised that annotators had more difficulty with Enable questions than with the other categories of questions, because in the context of questions like this it seems that "enable" could signal either active facilitation or a lack of interference. The description in the second paragraph of the Question Types section suggests that the first meaning is intended, but I think the word "enable" is ambiguous. I think that I myself wouldn't know how to answer without further instructions about which of these meanings is intended. If the goal is to measure humans' intuitive physics, then some clarification is probably required for respondents. That probably goes for the other categories as well.
correctness: The claims are correct to the best of my knowledge. The dataset seems to be well motivated and soundly constructed, and the evaluations seem appropriate and robust.
clarity: The paper is generally clearly written. There are a couple points that I think need clarification. I can guess at most of the answers, but it would be helpful to reword or briefly clarify these.
- What exactly does it mean that "the language component is largely missing" in previous work?
- What does it mean that previous datasets "lack visual variations"?
- What does it mean that CRAFT questions "require multiple counterfactual simulations to be explored"?
- Why is it "crucial to have language variety"?

More substantially, I don't understand the examples in Figure 1. For instance, the first question-answer pair says that 2 objects fall to the ground. Looking at the scene, I would have thought the answer was 4. I'm also not sure how I would go about answering the other four categories of questions. How can I tell that the small brown sphere causes the tiny yellow box to enter the basket? Perhaps these categories can't be meaningfully represented without animation, but this needs to be clarified.
relation to prior work: This is generally well described, though there are a couple points that merit clarification as described above.
additional feedback: No additional comments.
review_title: Valuable dataset for physical causal reasoning. Some additional baseline performance measures required.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper proposes a new visual question answering task which requires understanding complex physical reasoning, in the sense of "decoding visual data" to answer questions about physical simulations under alternative configurations of the data. The dataset emphasizes visual question-answering about “cause, enable, and prevent” and claims this is related to “commonsense knowledge” 

===
Thanks for the response! 
I have raised my score a bit but would defer the substantive questions of intuitive physics engines as benchmarks to other reviews/the AC. 

strengths: The paper proposes an interesting task that (under appropriate parameter initializations) is easy for humans via "commonsense reasoning" but difficult for machine learning. The textual questions are about objects in the scene in different arrangements. The difficulty of the task incorporates text parsing, visual decoding, and learning effective physics simulations. 
weaknesses: Figure 2 shows that the size of the dataset is generated in part by perturbing within small bound shte location of objects in fixed configurations. On the other hand, the paper also notes that “Humans do not accurately reason about the objects for some counterfactual cases as subtle changes in the scenes result in very different outcomes.”, presumably because of predicting events such as collisions between moving objects that can be quite sensitive to absolute position information. 

It is not exactly well-argued by the paper why the visual question-answering task corresponds necessarily to “human commonsense reasoning”. The task itself has the structure of visual synonyms layered on top of a physics engine --- if a neural network appropriately parses the sentence structure (even with verbal variation) to an appropriate mapping in the physics engine, and had access to a physics engine (let’s say), then it could sufficiently “solve” the task. As to the question of whether a neural network can “learn” a physics engine, it’s not necessarily well argued why this is a good benchmark: if we really wanted to solve physics problems with complex sensing, we should endow the machine learning pipeline with the physics engine. 
Overall, since ultimately the dataset is generated from an underlying physics engine, it’s not clear to me that the difficulty introduced by the dataset maps well onto the space of machine learning, in the sense that the difficulty seems a bit artificial in the sense that it is difficult for neural networks (for example) to learn physics engines. The paper does not provide evidence, however, that a suitably engineered approach to visual question answering (decoding the vision state by CNN and passing scene reconstruction into a physics engine) would solve the task. I surmise that if one really wanted to solve the task, that is what one would do instead. 

correctness: Correct to the best of my knowledge. 
clarity: Clarity can be improved. 
relation to prior work: Relation to prior work and other types of similar benchmarks could be improved. 
additional feedback: Perhaps framing the paper more narrowly could help the sharpness of the claims, eg not emphasizing "commonsense reasoning" (since after all, the "commonsense reasoning" is more about "intuitive physics", e.g. that humans can think as a worse version of an actual physics engine.
review_title: review 

Meta-review: Reviewers agree that this paper is proposing an interesting task of visual question answer tasks using complex physical reasoning. However, there are a few issues that we hope authors clarify to resubmit in the 2nd round. First issue around human performance (raised by all reviewers). For example, inter-annotator kappa coef seems very low (0.26-0.56). While author’s argue that the way study was conducted did not request folks to annotate qualitatively, it remains unclear what would it mean to have the human labels if humans can’t agree with each other. While authors argue that the performance itself is not bad given the high complexity of the task, the fact that humans don’t agree means that some (maybe most?) humans get the answer ‘wrong’ in some way. How do we reconcile this for the goal of this dataset? It would be very helpful to further clarify this point. Second, related to the first one, the final goal of this dataset could use more clarification. One reviewer put this elegantly, so I paste their words here "I remain uncertain about whether this dataset should be understood as reflecting (1) prescriptive truth about causal relationships and events, or (2) humans' intuitive physics. As in, should this dataset be used to test whether computers can understand complex physical causal relationships, or to test whether computers can learn to mimic humans' intuition about these types of relationships?” 
Decision: Reject