SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hsiao-Yu Tung $^{1,2*}$  Mingyu Ding $^{1,3*}$  Zhenfang Chen $^{4}$  Daniel M. Bear $^{2}$  Chuang Gan $^{4,5}$  Joshua B. Tenenbaum $^{1}$  Daniel L. K. Yamins $^{2}$  Judith Fan $^{2}$  Kevin A. Smith $^{1}$
$^{1}$ MIT 2Stanford 3UC Berkeley 4MIT-IBM Watson AI Lab 5UMass Amherst {sfish0101, kevin.smith3}@gmail.com myding@berkeley.edu jefan@stanford.edu
General physical scene understanding requires more than simply localizing and recognizing objects – it requires knowledge that objects can have different latent properties (e.g., mass or elasticity), and that those properties affect the outcome of physical events. While there has been great progress in physical and video prediction models in recent years, benchmarks to test their performance typically do not require an understanding that objects have individual physical properties, or at best test only those properties that are directly observable (e.g., size or color). This work proposes a novel dataset and benchmark, termed Physion++, that rigorously evaluates visual physical prediction in artificial systems under circumstances where those predictions rely on accurate estimates of the latent physical properties of objects in the scene. Specifically, we test scenarios where accurate prediction relies on estimates of properties such as mass, friction, elasticity, and deformability, and where the values of those properties can only be inferred by observing how objects move and interact with other objects or fluids. We evaluate the performance of a number of state-of-the-art prediction models that span a variety of levels of learning vs. built-in knowledge, and compare that performance to a set of human predictions. We find that models that have been trained using standard regimes and datasets do not spontaneously learn to make inferences about latent properties, but also that models that encode objectness and physical states tend to make better predictions. However, there is still a huge gap between all models and human performance, and all models' predictions correlate poorly with those made by humans, suggesting that no state-of-the-art model is learning to make physical predictions in a human-like way. These results show that current deep learning models that succeed in some settings nevertheless fail to achieve human-level physical prediction in other cases, especially those where latent property inference is required. Project page: https://dingmyu.github.io/physion_v2/
The human visual system allows us to perceive rich contents in the dynamically changing physical world – from a static scene, we see objects, their shapes, sizes, colors, and textures. From objects' motions, we further see their mechanical properties, e.g., mass, friction, and elasticity, much more beyond simple object appearance. Knowing these mechanical properties enables us to adjust our prediction of the likely future – for instance, if we see a box sink into a couch cushion we might infer it is filled and so will be heavy when we pick it up. Any artificial system that is designed for general
*Equal contribution. This work was done when Mingyu was at MIT.
Figure 1: Different mechanical properties can lead to different physical outcomes. In this work, we design Physion++ consisting of 9 scenarios (Mass-Dominoes/Waterpush/Collision, Elasticity-Wall/Platform, Friction-Slide/Collision/Clothslide, and Deform-Roll) that try to probe visual physical prediction relevant to inferring four commonly seen mechanical properties – mass, friction, elasticity, and deformability. See Section 3.1 for more details.
physical prediction should also be able to infer the mechanical properties of objects and leverage that information for future predictions.
Prior benchmarks of physics [1, 2, 3, 4] have pushed the field to show that some models that purport to be good at physics are not actually capturing physically relevant outcomes, and that often model architectures that explicitly represent objects and their geometries outperform those that don't. However, such benchmarks do not require any understanding that objects have individuated properties, e.g., all objects had the same density and elasticity. Agents (e.g., dynamics models and human participants) in these benchmarks often make predictions based on visual appearance rather than physical properties, which can be ambiguous in real scenarios.
Take the mass-dominos task in Figure 1 (row1, left) as an example. The domino on the left (light brown) is falling down and might possibly hit the domino in the middle (dark blue), and the goal of the agent is to predict whether the dark blue domino would hit the light region on the right. However, the outcome of this scenario highly depends on the relative masses of the two dominos: (1) If the dark blue domino in the middle is much lighter, then as long as there is contact, then most likely it would fall down and hit the target region. (2) If the mass of the dark blue domino is much heavier, then it is possible that even if there is contact between the dominos, the dark blue domino would fail to fall down and therefore not hit the target region. The two scenarios have exactly the same shape and arrangement and thus would present the same observation without any information to disambiguate the masses. Therefore, how to evaluate agents' capability to predict latent mechanical properties of objects becomes a key issue.
To further evaluate visual physical prediction under circumstances where accurate physical prediction involves efficient inference of the latent mechanical properties of the objects, we introduce Physion++.
Table 1: Qualitative comparison between our Physion++ and other physics/dynamics-related video reasoning benchmarks. 'Diverse Phenomena' refers to whether the dataset covers diverse scenarios and types of properties, e.g., our dataset has 9 scenarios, including rigid bodies, fluids, soft bodies, objects of various shapes, and various physical properties. 'Few-shot Reasoning' means that our dataset allows physical properties to be judged from a few reference video frames, i.e., property inference.
Dataset	Realistic	Object Dynamics	Physical Properties	Diverse Phenomena	Few-shot Reasoning
KITTI [5]	✓	×	×	×	×
Human3.6M [6]	✓	×	×	×	×
ShapeStacks [7]	✓	✓	✓	×	×
Cater [3]	×	×	×	✓	×
RoboNet [8]	✓	×	×	✓	✓
CLEVRER [4]	×	✓	×	×	×
CoPhy[9]	×	✓	✓	✓	×
PHYRE [10]	×	✓	×	✓	×
ESPRIT [10]	×	✓	×	✓	×
CRAFT [11]	×	✓	×	✓	×
Physion [1]	✓	✓	×	✓	×
IntPhys [2]	✓	✓	✓	×	×
Comphy [12]	×	✓	✓	×	✓
Physion++ (ours)	✓	✓	✓	✓	✓
This dataset is built using the same engine and testing procedure as Physion [1], but is designed so that individual mechanical properties affect the physical outcomes. It consists of test suites on four commonly seen mechanical properties – mass, friction, elasticity, and deformability (see Figure 1), providing a more comprehensive challenge than previous benchmarks. For example, our dataset rigorously includes physical interactions between objects with different mechanical properties, i.e., objects with varying masses, frictions, softness, and bounciness. The key difference compared to Physion [1] is thus that we further include extra frames (stimuli) depicting object interactions to enable agents to infer the mechanical properties before the actual prediction task. An example of our full observation is in Figure 2. We design the tasks such that, to accurately forecast the outcome in the prediction phase, the agents need to infer the mechanical properties in the inference phase. These additional mechanical properties also give us a chance to introduce more scenarios with soft objects and fluids (Figure 1) that are less explored in previous works like Physion.
Using Physion++, we test whether a set of state-of-the-art models that have been trained to make physical predictions also learn to make and leverage these property inferences. We further compare these models' outputs to human responses on each physical property to test if they perform physical predictions in a human-like way. Our main contributions are thus three-fold: (1) introducing Physion++, a novel benchmark to test mechanical property inference, (2) analyzing the performance of a set of state-of-the-art physical prediction models on this benchmark, and (3) comparing the performance of these models against human predictions on the same stimuli.
Summary of key findings. We found that all video models fail to capture human-level prediction across all four mechanical properties. Similar to the finding in [1], we found neural networks that encode objectness perform better than models that are not explicitly trained on these object-relevant tasks. We also find particle-based models which operate on 3D representations and have direct access to the latent mechanical properties perform better than models that operate on 2D representations. Yet, we found that no model was reliably using mechanical property inference, and therefore all models failed to correlate to human predictions. These results suggest that existing visual learning models, despite their success in a variety of video tasks, still fall short of human performance in general physics understanding tasks.
Physics and dynamics prediction benchmarks. Humans can easily infer the rough dynamics and physical properties of objects from videos. For example, recent work [13, 14] has shown that people might approximate detailed 3D shapes with simple convex hulls or geometry for physics
understanding tasks. With the tremendous success of deep learning, many benchmarks [5, 6, 7, 3, 8, 4, 9, 10, 11, 1, 2, 12] are designed to evaluate the similar dynamics prediction capability of deep models. However, these benchmarks are either unrealistic, do not contain dynamics/physical properties, or only focus on a single scenario. The qualitative comparison between our Physion++ and other physics-related video reasoning benchmarks is shown in Table 1.
The next question is: How do we evaluate the physics knowledge learned by existing learning-based visual dynamics models and compare it to human behaviors on general physics understanding? Physion [1] provides a benchmark suite consisting of stimuli generated from realistic simulations on a wide range of dynamic scenes. The benchmark has helped evaluate progress towards human-like visual physical scene understanding algorithms and has provided insightful suggestions for model design. Yet, Physion includes physical prediction tasks that focus mainly on object shapes and arrangement, and not mechanical properties, while these properties can make a huge difference in the dynamics outcomes. Using Physion++, we specifically look at scenes that require making inferences about object properties from physics in order to get to the correct prediction - taking the best parts of Physion [1] and extending it to interrogate models of physics in deeper ways. Physion++ is purposefully designed to introduce the additional complexity that was ignored in Physion by incorporating different mechanical properties that influence the resulting physical interactions. This dataset includes a series of test suites focused on four fundamental mechanical properties: mass, friction, elasticity, and deformability. This broader scope presents a more comprehensive challenge compared to previous benchmarks, which is also different from what the Physion benchmark investigated.
Mechanical property inference. Recent work in material perception has demonstrated that people can quickly infer and use mechanical properties to make predictions about physical events [15, 16, 17]. While some work suggests that these properties are extracted using Bayesian inference [15], other work suggests that our visual system uses a set of task-specific heuristics to perform this estimation that could easily be learned by standard vision models. For example, the shape and motion fields of fluids could be used to estimate their viscosity [18, 19, 20]; deformation speed and contour of objects, as well as optical cues, could be used to estimate the elasticity of nonrigid objects when bending or pushing in [21, 22, 23]; and the trajectory length could be used to estimate the elasticity of an object bouncing around a room [14]. However, these visual heuristics leave the open question of how to choose which heuristics to use, and how to integrate them into a cohesive model that can account for human judgments in various physical reasoning tasks. There are also some works that integrate differentiable physics engines or learning-based physics simulators to perform property inference [24, 25, 26, 27, 28, 29] for video/physics property prediction and video question-answering tasks, but these models typically are based on a known dynamics model and do not learn physics from scratch.
Video and dynamics prediction. One popular hypothesis is that general intuitive physics knowledge can be learned through pixel prediction on large video datasets of natural physical scenes [30, 31, 32, 33, 34, 35, 36]. This provides a unified theory for intuitive physics knowledge learning and mechanical property inference without depending on task-specific heuristics [22, 23, 14], which is hard to obtain in general scenarios. This also provides a data-driven way to obtain the abstraction and approximation needed for a human-like intuitive physics engine [15], which is not limited by the availability of a realistic physics engine. These video models [37, 38, 39] are directly applicable to any video inputs, making it possible to handle natural dynamical scenes and even long-term video frame prediction on realistic scenes. Also, recent video diffusion models [39, 40] and transformer-based prediction models [37, 41, 42] show tremendous success.
Here we want to evaluate whether these models already encode general physics knowledge that includes mechanical property inference. We are also curious about how these models can explain human visual physical scene understanding, i.e., if they perform physical predictions in a human-like way.
In this work, we introduce Physion++, which tests models' capabilities to infer four commonly seen mechanical properties from dynamics, including mass, friction, bounciness, and deformability. We first explain our stimuli design for each task, and then detail how we construct our benchmark.
Figure 2: Stimuli are in the format of single videos. The videos consist of an inference period where artificial systems can identify objects' mechanical properties and a prediction period where the model needs to predict whether two specified objects will hit each other after the video ends.
3.1 Stimulus Design
We generate both the stimuli and the training data for visual dynamics learning algorithms with the ThreeDWorld simulator (TDW), a Unity3D-based environment [43]. We create 9 physical scenarios across 4 mechanical properties (see Figure 1), which are used to evaluate the model and human physics prediction ability that requires efficient inference of objects' mechanical properties from motion cues.
Stimuli are in the format of single videos, so they match better with circumstances that a real human might encounter in the real world. Following [1], we probe physics understanding with the object contact prediction (OCP) task, where the goal of the artificial systems is to predict whether two specified objects, cued in red and yellow in the middle of the video, will hit each other if physics continues to unfold after the video ends. In Table 1, we show Physion++ is more comprehensive than previous physical scene understanding benchmarks.
To provide sufficient information for artificial systems to infer objects' mechanical properties, we design the videos to consist of an inference period where the artificial systems can identify objects' mechanical properties and a prediction period where the model needs to predict whether two specified objects will hit each other after the video ends (see Figure 2). Before the prediction phase, we indicate the target two objects with blinking lights in red and yellow. And we task the model with the question "Will the red object hit the yellow object?"
The nine scenarios in the Physion++ dataset are (see Figure 1 for an example):
1.1 Mass-Dominoes – a sequence of collisions that depend on the arrangement and masses of objects may or may not cause the red block to fall on the yellow mat
1.2 Mass-Waterpush - a collision between liquid and a rigid body red object may or may not cause that object to collide with a yellow object
1.3 Mass-Collision - a collision between an object and the red object may or may not cause the red object to move out of the path of a falling yellow object
2.1 Elasticity-Wall - the red object is thrown towards a wall and may collide with the yellow mat, depending on their placement, trajectories, and objects' elasticities
2.2 Elasticity-Platform - the red object is thrown towards a surface and may bounce onto or miss the yellow mat
3.1 Friction-Slide – the red object slides down a ramp and may make contact with the yellow object, or may stop beforehand due to friction
3.2 Friction-Collision – the red object starts with velocity and slides across the floor and may make contact with the yellow object, or may stop beforehand due to friction
3.3 Friction-Clothslide - a piece of red cloth slides down a ramp and may land on the yellow mat, or may under- or over-shoot it
4.1 Deform-Roll - the red object is dropped onto a piece of cloth and may either sink in or roll off in order to possibly hit the yellow object
We aim to avoid strong associations between superficial visual cues with the final YES/NO outcome by designing "paired" trials, where the paired video scenes are visually identical in the first frame during the prediction phase yet they unfold into different event outcomes due to different latent physical properties assigned to the objects in the videos.
In some scenarios, the inference and prediction phases can be included in the same video (e.g., Elasticity-Platform, Friction-Slide, and Friction-Clothslide). However, in many cases, the physical event that provides information in the inference phase irrevocably changes the configuration of objects so that there is no way to use the inferred information for future predictions (e.g., judging mass from seeing one domino topple into another leaves them both on the floor; Figure 2, top row). In these cases, we include a "transition phase": a curtain slides in to block the scene, then while the scene is occluded the objects are rearranged for the prediction phase, and finally the curtain moves out of the way. The cueing of the two target objects is done immediately after the transition phase, followed by a short observation of the rearranged objects in motion.
Further details of stimulus creation are provided in the Supplemental Materials.
3.2 Dataset Design
Each scenario of Physion++ consists of three stimulus sets: dynamics training, readout fitting, and testing. We describe them below, and provide further details in the Supplemental Materials.
Dynamics training set. The training dataset is provided so the agents can learn the dynamics of the environment, and thus learn representations that can be discriminative enough to distinguish whether the red object will hit the yellow object in the testing dataset. We generate 2,000 trials for each mechanical property without YES/NO contact labels for dynamics pretraining. For each physical scenario, in half of the trials the red object contacts the yellow object, and in half there is no contact, so as to ensure the balance of learning. Scenarios with an object rearrangement phase were included in the dynamics training so that models could learn that it was unsurprising for objects to be rearranged behind the curtain, but that objects retained their properties.
Readout fitting set. The readout fitting set is a small dataset containing 192 trials (96 pairs) for each mechanical property, used to map the dynamic representation learned in the training set to YES/NO of the OCP task. Trials were balanced for equal contact / no contact events to avoid bias.
Testing set. The final testing benchmark also consists of 192 trials (96 pairs) for each mechanical property. It has the same visual and physical statistics as the readout fitting set so that the learned mapping from the readout set can be directly evaluated on the test set.
4.1 Model Performance
Prediction Models. We evaluate state-of-the-art video models that fall into four main categories representing different levels of learning versus built-in knowledge. 1) Video models that learn a visual encoder and physics prediction model through pixel-wise prediction: MCVD [39]. 2) Visual encoders learned from supervised pretraining on ImageNet [44] that are designed to output responses using extended MLPs, which are trained in an unsupervised way on the benchmark readout data: pRESNET-mlp [45], pVGG-mlp [46], and pDEIT-mlp [38]. 3) Video models that learn physics prediction models on top of object-centric representations, obtained from self-supervised image pretraining on its encoder: SlotFormer [41] and ALOE [37] 4) 3D particle-based graph neural
Figure 3: Demonstration of the human task. Trial structure for the familiarization trials (left) and test trials (right) indicating the Cue, Stimulus, and Inter-trial periods.
network that takes ground truth 3D state and latent properties of objects as inputs (without a visual encoder, i.e. assume perfect observability of physical dynamics) and learns object transformation and dynamics as explicit 3D flow: DPI-Net [47]. More details are provided in the Appendix.
Training and readout protocols. We use 2 different training protocols and 3 readout protocols. For training, we test both models that learn dynamics separately for each mechanical property ('separate'), and models that learn unified dynamics models for all scenarios at once ('unified').
We test the models on three variants of each video. The 'w/ property' video is the standard video described in the stimulus design section. The 'w/o property' video does not include the initial frames of the video where the critical object first interacts in the scene, and therefore there is no property inference possible. This setting was used to test whether the model performance was in fact driven by using mechanical properties, or based on predictions without that information. Finally, the 'fully observed' video extends the standard video to include the red object contacting or missing the yellow object. Similar to [1], we use these videos as a test of the quality of the representations learned by each model - can we read out the OCP answer even if it is directly observed? We trained a separate readout model for each video type.
Training and Evaluation Pipelines. For all models, we first pretrain the learning of dynamics using the dynamics training set by predicting future frames under a future prediction loss. Then on the readout fitting set, we perform rollout using the pretrained dynamics model and extract representations from both observed and predicted frames. Based on these representations we train a readout model as logistic regression. Finally, we evaluate the model by extracting representations and reading out the OCP task from the testing set.
Results. The results of baseline models on Physion++ are shown in Table 2. We can observe that: (1) Training on all mechanical properties (unified) typically leads to inferior performance to individual training (separate), suggesting that learning joint models of dynamics produces worse predictions than learning individual scene dynamics. (2) Most models perform only slightly above chance, suggesting that this task is challenging for all current models. (3) Most models with property inference perform similarly to those without property inference, indicating that these models are not utilizing physical property inference. (4) With full video observations, the prediction performance generally improves, suggesting the learned representations do contain additional object information that could be predicted by their dynamics models. (5) DPI-Net achieves the best overall performance as it leverages ground