SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Results of model performance on the Physion++ dataset. For testing videos, 'w/' and 'w/o property' denote if the model makes predictions based on the property inference stage. 'fully observed' - model using the ground truth prediction frames.
Subset	Training	Testing	MCVD	DeiT-mlp	ResNet-mlp	VGG-mlp	ALOE	SlotFormer	DPI-Net
Mass	unified	w/ property	52.6	50.5	48.0	53.6	52.1	53.6	52.1
separate	54.2	56.3	54.7	55.2	53.1	58.9	75.9
w/o property	53.1	55.0	54.2	56.3	52.6	57.3	60.1
fully observed	67.1	69.8	67.7	60.4	56.3	66.7	96.9
Friction	unified	w/ property	52.6	54.2	50.5	50.0	52.1	50.5	53.8
separate	52.6	52.6	52.1	52.6	56.8	54.2	52.5
w/o property	54.2	52.6	52.1	51.6	53.1	54.2	54.7
fully observed	56.3	53.6	53.6	54.2	54.2	58.3	84.6
Elasticity	unified	w/ property	53.1	52.1	55.2	51.6	51.6	52.6	51.0
separate	52.6	55.2	54.2	52.6	52.6	58.3	52.1
w/o property	51.6	55.2	54.7	52.6	53.1	56.8	52.4
fully observed	55.7	59.4	54.7	54.2	53.1	64.6	95.8
Deformability	unified	w/ property	58.9	56.3	55.2	52.1	51.6	52.1	55.1
separate	57.8	57.3	59.4	61.5	51.0	55.2	60.2
w/o property	58.3	53.6	58.3	59.9	51.0	56.3	62.6
fully observed	60.4	72.9	68.8	69.8	58.3	70.8	91.3
Overall	unified	w/ property	54.3	53.3	52.2	51.8	51.9	52.2	53.0
separate	54.3	55.4	55.1	55.5	53.4	56.7	60.2
w/o property	54.3	54.1	54.8	55.1	52.5	56.2	57.5
fully observed	59.9	63.9	61.2	56.0	55.5	65.1	92.2
truth 3D information. VGG-mlp achieves good performance on the deformability set and SlotFormer achieves good performance on mass and elasticity sets.
In summary, most models fail to perform physical property inference, except DPI-Net in the mass scenario (60.1% for w/o property and 75.9% for w/ property), potentially indicating this model has learned to infer just the property of mass. This suggests that future work focusing on better physical property inference for deep models would be worth exploring in order to build systems that more generally reason about physical scenarios.
4.2 Human experiments
We additionally gathered predictions from human participants in order to test whether the models (a) reached human levels of performance, and (b) performed the task in a way similar to people, as measured by the same pattern of errors. 200 participants (50 for each of the mechanical properties) were recruited from Prolific and paid $15.50 per hour for their participation. We selected 192 trials from each scenario, then shuffle and split the 192 trials into 8 disjoint sets, where all sets have half of the trials containing contact between the red and yellow objects, and half do not. Each participant was shown 32 stimuli drawn from one of these disjoint sets of trials drawn from a single mechanical property, and we ensure participants do not observe both parts of any matched pair of trials. Data is balanced, so each participant will see 16 stimuli with the ground truth label 'YES' and 16 with 'NO'. These studies were conducted in accordance with the MIT and UC San Diego IRBs.
Task procedure. The structure of our task is shown in Figure 3. Each trial began with a fixation cross, which was shown for a randomly sampled time between 500ms and 1500ms. The stimuli will start playing right after the fixation. To indicate which of the objects shown was the agent and patient object, participants were shown the agent and patient objects were overlaid in red and yellow respectively, at the first frame during the prediction phase of the video for 2000ms. During this time, the overlay flashed on and off with a frequency of 2Hz. After this, the stimulus will keep playing until it ends and gets removed, and the response buttons were enabled. Participants proceeded to the next trial after they made a prediction by selecting either 'YES' (the red and yellow objects would touch) or 'NO' (they would not). The order of the buttons was randomized between participants. Before the main task, participants were familiarized with 4 trials that were presented similarly to the
Figure 4: Comparisons between humans and models. First row: task accuracy (I, II, III), Second row: Pearson correlation between model output and average human response (IV, V, VI). We evaluate the metric in three settings: with the whole testing dataset (all trials), with trials (in pairs) that humans perform particularly well (above  $67\%$  accuracy; easy trials) and poorly (below  $33\%$  accuracy; hard trials). We use the letters 'D, F, M, E' to denote the task deformability, friction, mass, and elasticity, respectively.
test trials, except the full stimulus movie and accuracy feedback were presented after participants indicated their prediction. Familiarization trials were always presented in the same order. After the test trials were completed, basic demographics were collected from participants. Finally, participants were informed of their overall accuracy.
Results and human-model comparisons. Following [1], we compare a model's outputs under the 'separate' training and 'with property' testing protocols to human responses on each physics property's testing stimuli with two standard metrics: overall accuracy and Pearson correlation between model and average human responses across stimuli. See Figure 4 I and IV, respectively. From the results, we found that humans are performing reliably above chance but not particularly well at the task (acc:  $60\%$ ), yet all video models are performing slightly worse than humans. Only DPI-Net, which operates on ground truth states from the physics engine performs comparably to humans. While this is a challenging task, using trials with a range of human performance (including many below chance) allows us to test whether model and human predictions are based on the same competencies, or whether they excel on different trials.
We found that no model correlated well with humans. SlotFormer holds the highest correlation with human predictions  $(r = 0.12$  vs. split half human correlation  $r = 0.37)$ , suggesting that object-centric approaches may be better suited for physics inference than alternatives, but they are not sufficient for producing human-like predictions.
Performance on easy/hard trials. We further examine the model and human performance on trials where humans perform particularly well and particularly poorly. This allows us to understand whether the models are differentiating trials that humans find trivial from those they find difficult, or whether model performance is driven by features not used by people. We evaluate on the "easy" trials by selecting stimuli from some of the same-initial-configuration pairs (see Stimulus Design for more details) if one of the stimuli in the pair has mean human performance higher than  $66.7\%$ , and we also evaluate the "hard" trials where one of the paired stimuli has mean human performance lower than  $33.3\%$ . These results are shown in Figure 4 II and V for the easy trials, and III, VI for the hard trials.
Table 3: Ablation study on the number of training trials per property.
#Trials	DeiT-mlp	ResNet-mlp	ALOE	SlotFormer
200	49.0	50.0	50.5	51.6
500	53.6	51.6	52.1	54.2
1000	54.7	54.7	54.2	56.3
2000	55.4	55.1	53.4	56.7
From these results, we see that all models appear to be making predictions in a way different from people: they all underperform humans on the easy trials, but outperform humans on the difficult trials. These results suggest that existing visual learning models still fail to capture human behaviors in general physics understanding tasks, especially in cases where efficient inference of the underlying physics properties is required.
Performance on each of the four physical properties. According to Figure 4, most models perform better than humans in the deformability category, while their performances for all the other three categories are worse than human judgment. This might result from the fact that humans are better at learning intuitive physics hence the deformability property is harder to predict than the other three (see hard trials in Figure 4 III). However, these models do not utilize physical property inference and learn from data distributions, therefore the preferences for the four categories are less pronounced.
Performance on different number of training trials per property. We conducted additional ablative experiments with different numbers of trials by selecting four representative models (DeiT-mlp, ResNet-mlp, ALOE, and SlotFormer). The models learn dynamics separately for each mechanical property with property inference, and we report the average performance of the four mechanical properties. The results are shown in Table 3. We can see that when the data size is small (200), the performance of all models is close to random guessing. As the number of training data increases, the accuracy of the all models improve and gradually saturate, with only minor performance increases from 1000 to 2000 trials. Therefore, we believe that 2000 training trials for each property are reasonable, and that any performance differences we observe are related to architectural differences. In the future, we will continue to enrich our data to further investigate asymptotic performance across these and other models.
Physical scene understanding itself is a challenging task, requiring knowledge of visual perception, spatial relation, physical property understanding and inference, distance and dynamics estimation, etc. Despite recent successes from neural-network-based video frame prediction models, none of these models correlate with human predictions on Physion++. This indicates that current computational models do not make online inferences about physical properties. This could be for a number of reasons. It could be that property inference is a difficult task; these models were not designed with this inference in mind, and 2,000 scenarios might not be enough for this capability to emerge. In addition, these models might struggle with video prediction that involves multi-stage long-term dependencies. Our current findings suggest that while the accuracy levels may be both close to 0.6, the correlation between human and model performances is low, indicating their proficiency in distinct domains. Humans and models excel at different scenarios and trials. It shows that although the models outperform humans in some scenarios, they still have huge room for improvement, e.g., they still fail in many scenarios that humans can easily handle.
The videos in our tasks often involve several phases: inference events, the transition phase, and the test phase, which causes the videos to last for 13-20 seconds. On the other hand, these models are often trained and tested on shorter videos. We believe with future video and dynamics model development that enables accurate predictions on multi-phase videos, our proposed Physion++ dataset can again serve as the benchmark to evaluate how well these models can be used to infer mechanical properties and how well their performance correlate with humans. The proposed dataset has no ethical or societal issues on its own, except those inherited from physical scene understanding.
Acknowledgement. This work was supported by ONR MURI grant N00014-22-1-2740.
DiCarlo, Josh H. McDermott, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Threadworld: A platform for interactive multi-modal physical simulation. CoRR, abs/2007.04954, 2020.
[44] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[45] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[47] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01566, 2018.
[48] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. arXiv preprint arXiv:2205.14065, 2022.
[49] Christopher P Burgess, Loic Matthew, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11390, 2019.
[50] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé Iii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12):86-92, 2021.
A Project Page and Dataset Release
We have released the Physion++ dataset at our project page. All three subsets (training set  $^2$ , readout fitting set  $^3$ , and testing set  $^4$ ) and the human data  $^5$  are publicly available. We also post a list of all test filenames in the test set (192 entries per physical property). The dataset is organized as follows.
Listing 1: Organization of the dataset.
# Download and unzip the three data files
-- Training_data
...
-- Readout_data
...
-- Testing_data
-- Folder organized by physical properties # folders named with copy0 and copy1
indicate two matched trials (e.g., the same video location from copy0 will have a
different property/outcome in copy1 but the same initial conditions)
-- Subfolder organized by scenarios and objects # e.g. bouncyplatform-
use.blocker_with_hole=0-target=cube
-- [id].json # object id and instance information over time
-- [id].pkl # meta information of the physical event and the video
-- [id].image.mp4 # raw input, RGB video
-- [id]seg.mp4 # segmentation masks of the video
-- [id].map.png # indicating the yellow and red objects
# Below are lists of test filenames and ids (192 entries per physical property)
-- Mass.csv
-- Friction.csv
-- Elasticity.csv
-- Deformability.csv
The .json file contains the object id and segmentation mask of each instance over time. The .pkg file contains the meta information of the scene, including 1) the dynamic friction, static friction, initial position, initial rotation, mass, elasticity, color, and mesh of each object; 2) the camera matrix and projection matrix for each frame; 3) angular velocities, positions, rotations, and velocities of all objects over time; 4) the collision events, including the object ids, relative velocities, time and states; 5) the trial seed used to generate the video, the label for the video, and 'start_frame_for_prediction', which is a timestamp indicating the part of the video before the timestamp is visible, and the part after the timestamp is required to be predicted.
Author statement. We confirm that we bear all responsibility in case of any violation of rights during the collection of the data or other work, and will take appropriate action when needed, e.g. to remove data with such issues.
Hosting, licensing, and maintenance plan. We host the dataset on Amazon AWS. We ensure access to the data and will provide the necessary maintenance. All products created as part of this project is shared under the MIT license. We used a number of third-party software packages, each of which typically has its own licensing provisions. Only TDW [43] was used in the creation of the dataset; all others were models used for assessment. Table 4 contains a list of these licenses for many of the packages used.
B Scenario Details
Mass-Dominoes. This scenario starts with an inference phase, where a set of "dominoes" (equally sized cuboids standing long end up) are placed approximately in a row with semi-random spacing and orientations. One of the dominoes is visually marked with a different texture, indicating a different material. At the start of the video, a domino at one end starts to fall as if it had been pushed over, starting a sequence of the dominoes being pushed over. The video continues past the point where the textured domino is hit by or hits one of the other dominoes, providing information about the textured dominoes' mass. A transition phase is then required (as all relevant objects have toppled)
$^{2}$ Training data: https://physion-v2.s3.amazon.com/train_data.zip
3Readout data: https://physion-v2.s3.amazon.com/readout_data.zip
4Testing data: https://physion-v2.s3.amazon.com/test_data.zip
Human data: https://physion-v2.s3.amazon.com/human_data.zip
Table 4: Table of open-source code used.
Name	URL	License
MCVD [39]	https://github.com/voletiv/mcvd-pytorch	MIT License
ALOE [37]	deepmind/object_interest_for_reasoning	Apache License 2.0
ResNet [45]	https://github.com/pytorch/vision	BSD 3-Clause License
DeiT [38]	https://github.com/facebookresearch/deit	Apache License 2.0
VGGNet [46]	https://github.com/pytorch/vision	BSD 3-Clause License
DPI-Net [47]	https://github.com/YunzhuLi/DPI-Net	N/A
TDW [43]	https://github.com/threedworld-mit/tdw	BSD 2-Clause License
SlotFormer [41]	https://github.com/pairlab/SlotFormer	MIT License
and the dominoes are reset. In the reset scene, there is a mat on the floor, and one of the dominoes is indicated as the 'red' object while the mat is the 'yellow' object. The textured domino is placed in the chain so that its mass will influence the chain of dominoes: e.g., if it is too heavy it would not topple when another domino strikes it, but if not it will continue the sequence of collisions and cause the red domino to land on the yellow mat.
Mass-Waterpush. The inference phase begins with an object at rest and a stream of water shooting towards the object as if out of a hose. This stream may move the object, and thus give information about its mass. The transition phase then occurs where the object is moved to another location (and placed upright if it has tipped over) and marked as 'red'. In addition, a new object is added falling from midair, marked as 'yellow'. The yellow object might be positioned above the red object, in which case, depending on the masses, the water might knock the red object out of the way or fail to move it. Or the yellow object might be positioned further along the path defined by the water stream, so that the stream might cause the red object to slide into the yellow object, or fail to move.
Mass-Collision. The scenario is identical to the Mass-Waterpush scenario, except that instead of a stream of water that pushes the object, a ball rolls into and collides with the object in both the inference and testing phases.
**Elasticity-Wall.** In the inference phase, an object is in ballistic motion towards a wall, bounces off, and lands on the floor. This provides information about the elasticity of the collision. The transition phase then occurs, and the object is again placed in ballistic motion towards the wall (which does not move) and marked as the 'red' object, with a 'yellow' mat being placed on the floor. Depending on the elasticity, after bouncing off of the wall, the red object may land on, or over- or under-shoot the yellow mat.
**Elasticity-Platform.** This scenario contains a raised platform that ends and drops onto a surface, with a wall at the end of that surface, followed by a mat (marked as 'yellow') on the floor. It starts with an object (marked as 'red') bouncing onto the platform and continuing to bounce/slide to its edge. This provides information about the objects' elasticity, and thus how it will bounce when it falls onto the surface. The key question is whether the red object will touch the mat. In some cases, the wall is short, so the object must have high elasticity to bounce over and hit the mat. In other cases, the wall is tall but has a slot at the bottom; in these cases if the object bounces too much it will hit the wall and stop, but if it is less elastic it will slide through the slot and contact the mat. In this way we decorrelate the elasticity from the outcome.
Friction-Slide. There were two subtypes of scenarios for Friction-Slide. In both cases, an object (marked as 'red') is positioned near the top of a ramp and begins to slide down. In the 'gap' situation, there is a pivot in the slope, and the 'yellow' mat is positioned in that gap; thus depending on the friction of the red object it might slide into the gap and contact the mat or fly over the gap and miss it. In the 'no-gap' situation, there is no pivot, and the 'yellow' object is placed in the runout area of the slope; thus the red object might stop before hitting the yellow object or might continue to slide into it. We use both subtypes so that there is not a correlation between low- or high-friction objects and the outcome. No transition phase is needed in this case; the inference can be performed from the first part of the video when the object is sliding down the slope.
Friction-Collision. This scenario starts with the key 'red' object sliding along the floor, providing information about the friction of that object. The transition phase occurs and then the red object is reset to a different position with a new velocity, and a 'yellow' object is dropped from above (similar
to the Mass-Collision and Mass-Waterpush scenarios). The crucial judgment is whether the friction of the red object will slow it down just enough so the yellow object will land on it, or whether the red object will under- or over-shoot the mark.
Friction-Clothslide. This scenario is identical to the Friction-Slide scenario, except instead of a rigid object sliding down the ramp, the 'red' object is a piece of cloth.
Deform-Roll. This scenario starts with a cloth hanging from posts either vertically or horizontally, and a key object is launched or dropped on the cloth respectively. This provides information about the deformability of the cloth. The transition phase occurs and the cloth is now hung from posts at an angle. The critical 'red' object is dropped from above the cloth, and the 'yellow' object is set in one of two places. It is either positioned on the ground towards the base of the cloth, so that it is important to determine whether the red object will sink into the cloth, or roll off of it and hit the yellow object. Or the yellow object is dropped from above the red object, so that if the cloth is deformable enough both will sink in and touch, but if it is not, the red object will roll off before the yellow and they will never contact.
The examples of all 9 scenarios are shown in our project page.
C Details of the Dataset
In some scenarios, the inference and prediction phases can be included in the same video (e.g., Elasticity-Platform, Friction-Slide, and Friction-ClothSlide). However, in many cases, the physical event that provides information in the inference phase irrevocably changes the configuration of objects so that there is no way to use the inferred information for future predictions (e.g., judging mass from seeing one domino topple into another leaves them both on the floor at the top row of Figure 2 in the main paper). In these cases, we include a "transition phase": a curtain slides in to block the scene, then while the scene is occluded the objects are rearranged for the prediction phase, and finally the curtain moves out of the way. And the cueing of the two target objects is done immediately after the transition phase, followed by a short observation of the rearranged objects in motion. Examples of all 9 scenarios are shown in our project page.
Training set. The training dataset is used for the agents to learn dynamics prediction, and the learned representations can be discriminative enough to distinguish whether the red object hit the yellow object, and can generalize to the testing dataset. We generate 2000 trials for each mechanical property without YES/NO labels for dynamics pretraining. For each physical scenario, we make half of the trials where the output answers are YES and half of them are NO, so as to ensure the balance of learning.
Readout fitting set. The readout fitting set is a small dataset containing 192 trials used to map the dynamic representation learned in the training set to YES/NO of the video question-answering (i.e., OCP) task.