Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round1

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The paper presents a new dataset with several benchmark models. The goal of the dataset is to learn physical understanding, where the physical understanding is measured by the model's/human's ability to predict collisions of basic objects based on a short video.
strengths: The dataset is very relevant for learning physical understanding of the world. The evaluation is very general, such that many types of models can be used.
The dataset is challenging. Humans cannot get perfect scores, and vision-based models are just slightly better than random.
Substantial baseline work: Several relevant baseline models are presented. This covers vision-based models (VGG + Transformer based), particle-based models (that takes the GT physical states as input), and human-evaluation (where humans are asked to estimate collision/ no collision based on small video clips)
weaknesses: The environments seems rather similar (fig. 1) (same floor, walls etc.), thus we cannot expect vision-based models to generalize to real world or other datasets, if trained on this dataset.
correctness: It seems to me that the dataset and baseline models are designed in a sound way. 
clarity: The paper is clear and well written.
relation to prior work: Yes, the dataset is the first 3D data encompasses a wide range of physical scenarios. Furthermore, the dataset is the first large-scale dataset to compare to human level performance for action prediction.
additional feedback: On the github, it would be nice with some small videos to illustrate the input. This would give users a bit better intuition about the difficulty of the dataset.

#########

Final score after author feedback is unchanged. I think the dataset can be relevant for the community and the methodology well presented.
review_title: Physion: Evaluating Physical Prediction from Vision in Humans and Machines

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Here, the authors provide a dataset of 17,200 videos of simulated physical interactions, containing 2,000 training videos and 150 test videos for 8 distinct problem scenarios. The dataset provides a clear-cut problem of predicting a final contact state between objects based upon an initial configuration and momentum. Overall, the main claims of the paper are that the dataset provides a more comprehensive challenge than existing benchmarks for physical prediction and reasoning. 
strengths: The authors provide a resource for the critical problem of testing physical prediction and reasoning capabilities of machine learning models. The dataset is comprehensive, and encompasses a diversity of physical scenarios, and the problem formulation is elegant, distilling the task of physical prediction down to a centralized definition that is granular enough to be useful. I really like the baseline dataset that the authors provide of human physical prediction ability: the authors collect annotations from 800 human participants and compare models against human reasoning capabilities on the same dataset. That current models fall short of this baseline means that there is still ample room for our current machine learning methods to grow, and having benchmarks like these can really support this aim. 

Overall, this work promises to provide a first-of-its-kind dataset for a challenging problem that has potential to advance the state-of-the-art. However, as I will explain in the weaknesses and feedback to authors section, I feel that there are some cumulative problems with the work in its current state that make it fall short of achieving this goal on a practical level. 
weaknesses: I really want to like this paper. However, a major issue is that the training dataset is difficult to access because it has to be generated directly from code the authors provide, and the code is not documented well enough to explain how to generate both training datasets. But additionally, there are numerous smaller issues with the clarity of the writing, the explanation and characterization of the dataset, and plan (or lack thereof) for maintenance of the dataset, that are not fatal in isolation, but in combination result in a manuscript that is rough around the edges. These cumulative issues detract from the utility and reproducibility of the dataset,  as well as knowledge of what claims are possible with it. I provide a more detailed list of these weaknesses in the "Feedback to Author" section. 
correctness: The high-level details seem okay, and a brief browsing of the test dataset does not seem to indicate any major issues. Related to point 2 in the feedback, I'm not sure what the authors did to make sure the datasets were well-represented in contact/no-contact: was there any attempt to generate a diversity of videos for both cases, or were the videos just simulated without any prior design expectation of either? Related to point 3/2, I'm not exactly sure of how diversity of data was produced, and what the distribution of the test versus the training data is expected to look like. 
clarity: Generally, yes - the introduction especially does a good job of outlining the motivation and utility of this dataset. However, as I point out in the feedback section, there are some issues with the explanation of how this dataset was produced (point 3), and their notation and clarity in their problem characterization (point 5). 
relation to prior work: To my knowledge, yes, the authors provide a good overview of existing benchmarks. I am not an expert in this specific field, so I cannot comment on if there are any foundational datasets they may have missed. 
additional feedback: 1. While the authors provide mp4 and HDF5 files for their test dataset as zip files, the training dataset has to be generated from the code they provide. This limits the accessibility of the training data, especially since this problem may attract researchers from different backgrounds who may not have experience with ThreeDWorld, or have hardware limitations. I understand there may be some difficulty in making the HDF5 files for the training datasets accessible due to their large size (although the authors could consider dividing by their 8 physical scenarios), but I would like to see them make at least the mp4 files accessible. Additionally, I would prefer to see these datasets uploaded to a stable public repository (e.g. Zenodo) over a private Amazon instance, especially given there is no plan for the maintenance of these Amazon links described. 

1a. How exactly do I generate the training data from the tdw_physics repository? The repository contains a README of how to generate datasets in general, which is great - but I'm not certain how I would actually reproduce the exact training data used in this manuscript. I see a "generate_original_data.sh" file, but there's no comments, and it seems to refer to a non-existent directory "target_controllers". More documentation for reproducibility purposes is essential. 

2. Could the authors provide summary statistics for each of the datasets? In particular, I am interested in the class balance of contact versus no contact labels for each problem. This could just be how I sampled videos, but in the subset of mp4s I looked at for the "collide" problem, it seemed that a large majority of the scenarios contact was not made. What would be the expected accuracy if a model just predicted no contact for all inputs? 

3. The authors explain their principles for producing a generated dataset with variability within each scenario, but the manuscript is unclear on what exact procedures they followed to make this happen. Examining the test data, I see that parameter sets seem to occur in "batches" - for example, I see 15 movies for the Collision scenario generated with parameters "assorted targets", "box_2", "dis_2" and "occ". Within these batches, there seem to be other parameters that are kept controlled - for example, I see the texture and friction of the flooring is consistent between all 15 movies. This is not explained in the manuscript. Importantly, since I see so many similar subsets of images in the test dataset grouped by parameter, is the test dataset images of totally different parameter sets than the training images, or a randomly held-out subset? This is critical for readers to understand exactly what makes this problem hard: I would like to know if these baselines are failing on a problem that's so hard that it can't be done in-sample, or if the authors are stacking an out-of-sample generalization problem on top of this. 

4. I agree with the authors' goal of removing confounds from prior knowledge of object configurations. I can't help but notice in the videos, the authors seem to denote friction by whether the floor resembles office carpeting, or kitchen tile. Especially as this is cultural context that not all human participants would share, I can't help but wonder how a participant's ability to recognize the simulated floor as carpet/tile would have impacted their ability to solve the problem. Is there any commentary that the authors can provide on this, or statistics from the participant labels that would provide insight on this issue? 

5. I have a hard time understanding the authors' explanation of the OCP readout fitting set: what does y denote here? I assumed this is P(contact), but if so, I'm unclear on how this differs from the training dataset of 2000 videos per scenario where the movies (I'm assuming t_vis in this formula is actually supposed to be in subscript) are paired with labels of P(contact)? More clarity in their explanation here is needed. 

6. The authors mention adversarial examples where the dynamics are hard to capture in their methods as a case for testing if the models are learning non-causal shortcuts. I see some examples of adversarial cases in the supplementary, but what I don't see is any testing of if the models perform well on the adversarial examples.
review_title: Review of "Physion: Evaluating Physical Prediction from Vision in Humans and Machines"

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper studies the problem that whether machines can make future predictions for "real-world" physical events and proposes a new benchmark named Physion. It simulates photo-realistic videos with the TDW simulator[24]. The benchmark contains diverse physical scenarios, including dominoes, support, collide, contain, drop, link, roll, and drape with dense and accurate annotations like color, depth, normals ... . Various baselines are conducted and analyzed, among which DPI with object-centric graph neural networks performs the best.
strengths: 1. The benchmark is valuable for evaluating machines' intuitive physical understanding of the world, which is of great importance for human cognitive understanding and robotic planning and control;
2.  Specifically, the benchmark provides dense pixel-level annotations while containing more diverse physics events and being more similar to real-world videos than previous videos from simulators (e.g. [6, 29]);
3. Various baselines are evaluated and analyzed, showing to what extent the previous dynamic models understand the physical world;  
4. A valuable and time-consuming human testing is provided, showing how humans perform on the new benchmark and its comparison with machine algorithms. 
weaknesses: While the reviewer is positive about the benchmarks, the reviewers still have some concerns or suggestions on the project.
1. One concern is about the implementation of the baselines. The reviewer appreciates the authors provide original code links. However, according to the reviewer's understanding, baselines like RPIN and DPI are based on object/ proposal-centric representations. How do the authors handle and model the soft-body collisions for them? Will the implementation of the baselines for the new benchmark available to the public?
2. The performance comparison in Fig. 5 is easy to understanding. However, it will be hard for future researchers to copy the numbers for comparison. It will be better to provide its table versions in supplementary.
3. The DPI is based on ground-truth annotations from the simulator, which makes its comparisons with other baselines are unfair, although the reviewer knows that a fair performance comparison is not the main focus of the paper.
4. Recently,transformers [A, B] has shown strong performance for spatio-temporal reasoning. It is interesting to evaluate their performance on this new benchmark.
5. The benchmark provides dense pixel-level annotations like depth and normals. However, it seems that the baselines make no use of them. How will these visual properties help with physics understanding?  
[A] Zhou, Honglu, et al. "Hopper: Multi-hop transformer for spatiotemporal reasoning." ICLR 2021.
[B] Ding D, Hill F, Santoro A, et al. "Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures", Arxiv 2021.
 
correctness: Please refer to the weakness section.
clarity: It is well written.
relation to prior work: It has discussed how the work differs from previous contributions.
additional feedback: None.
review_title: An interesting benchmark for evaluation of phyiscal prediction from vision in humans and machines

Meta-review: The paper proposes a challenging dataset to learn physical understanding along with several benchmark models. The paper is very well-written, and the reviewers agreed that the paper should be accepted. 

The physical understanding of the world is a very fundamental problem in AI research. This paper proposes a set of challenges to investigate it further. It is a very important challenge to build better model-based RL algorithms and general-purpose AI agents. The paper tests several different ML models and compares them against the human performances on those tasks. The experiments seem to be carefully designed and well-thought-out.

The authors addressed the concerns raised by the reviewers during the rebuttals quite well. 


Decision: Accept