SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Testing set. The final testing benchmark consists of 192 trials (96 pairs) for each mechanical property. We aim to avoid strong associations between superficial visual cues with the final YES/NO outcome by designing the readout and test dataset to be "paired" trials, where the paired video scenes are visually identical in the first frame during the prediction phase yet they unfold into different event outcomes due to different latent physical properties assigned to the objects in the videos. We achieve this by fixing object configuration during the prediction phase and regenerating the stimuli with uniformly sampled mechanical values on a target object until we get one stimulus with a positive outcome, and the other with a negative outcome. For each scene configuration, we sample a maximum of 5 different property values, and we drop scenes where we sample all true or all false outcomes. It has the same overall visual and physical statistics as the readout fitting set so that the learned mapping from the readout set can be directly evaluated on the test set.
D Pipelines and settings
For each video, we truncate (or pad) both the inference phase and the prediction phase to 160 frames, and sub-sample the videos by a factor of 5 for training the representation or dynamics models. All frames are resized to  $128 \times 128$  to reduce the computational cost. For SlotFormer [41] and ALOE [37], we first pre-train the object-centric models STEVE [48] and MONet [49] on all
Table 5: Ablation study on implicit physical property inference.
Method	DeiT-mlp	ResNet-mlp	ALOE	SlotFormer
implicit (OCP only)	55.4	55.1	53.4	56.7
explicit as auxiliary	54.2	55.7	54.7	56.7
sub-sampled frames for scene decomposition, and extract all slot representations for subsequent training. The dynamics models are then trained on the slot representations from the training set under future prediction loss. For MCVD, the frames are directly fed into the model for dynamics prediction under image reconstruction loss. For pRESNET-mlx, pVGG-mlx, and pDEIT-mlx, we leverage pretrained ResNet50 [45], VGG16 [46], and DeiT-small [38] on ImageNet as our feature extractors. For DPI-Net, we represent the scene with particle representation provided by the annotation. For the oracle model with property inference, we add the ground-truth property values into the attribute embedding input of DPI-Net in both training and testing. For the model without property inference, we simply mask all property values with padding zero vectors. For the full video observed, we feed the ground-truth particle-based representation to the model. We calculate the distance  $d_{min}$  between the closet particles in the two target objects. We consider the two objects will contact if  $d_{min}$  is smaller than a threshold  $\eta$  that is learned from the training set.  $\eta$  is set to 0.075. For other parameters, we follow the same setting as Physion [1].
For models to generate YES/NO responses from their learned representations, we use the readout fitting set for the models to learn to map from their latent representation to the target response. We perform rollout to generate future scene representations (e.g. feature maps for image-based methods, or object slots for object-centric methods) based on the inference phase in the readout set. We implement a multilayer perceptron (MLP) with intermediate dimensions of 256 and 64 as our readout model, which is trained on rollout scene representations from the readout set to classify whether the two cued objects contact. All experiments were run on 8 NVIDIA Titan X GPUs using the Adam optimizer and a learning rate of 1e-4. The models learned on the training and readout sets are then evaluated on our final benchmark (testing set) by applying the learned visual representations and the readout model. The best testing results among all readout training epochs are reported.
E Explicit and implicit physical property inference
We selected four representative models (DeiT-mlp, ResNet-mlp, ALOE, and SlotFormer) for the experiment. A parallel layer of MLP is added at the end of the model (two-branch multitask: OCP and explicit property estimation) during the readout fitting process. With explicit property estimation as an auxiliary task, we report the performance comparison with our original setting as in Table 5. With explicit inference, performance barely improves, suggesting that the networks do not have access to the properties even when prompted, not that they understand properties but fail to use them for prediction.
F Datasheets for dataset
Here are our responses in reference to the Datasheets for Datasets [50] standards.
- For what purpose was the dataset created? To measure deep models' physical future prediction abilities and latent property inference capabilities, and compare these to predictions made by humans.
- Who created the dataset and on behalf of which entity? The authors listed on this paper, including researchers from MIT, Stanford, UC Berkeley, MIT-IBM Watson AI Lab, and UMass Amherst.
- Who funded the creation of the dataset? The various granting agencies supporting the above-named researchers, including both grants to the PIs as well as individual fellowships for graduate students and postdoctoral fellows involved with the project.
- What do the instances that comprise the dataset represent? Each instance is a video of a simulated physical scene (e.g. a tower of blocks as it either collapses or remains steady), together with some metadata about that video, including map-structured metadata with segmentation maps and information about object-object collisions at each timepoint.
- How many instances are there in total? The dynamics prediction model training dataset consists of 2000 examples for each of the 4 physical properties. The OCP readout fitting dataset consists of 192 examples per each of the 4 physical properties. The test dataset (on which human responses were obtained) consists of 192 examples per physical property.
- Does the dataset contain all possible instances or is it a sample of instances from a larger set? Data is generated by a simulator; in a sense, the set of datapoints we created is an infinitesimally small subset of data that could have been generated. However, we are all here releasing all the examples we did actually generate.
- What data does each instance consist of? It consists of a video depicting a physical situation (e.g. a tower of blocks falling over), together with simulator-generated metadata about the situation.
- Is there a label or target associated with each instance? For the training dataset, there are no labels. For both the OCP readout fitting dataset and the human testing dataset, there are binary labels describing whether the red object collided with the yellow zone during the duration of the trajectory.
- Is any information missing from individual instances? No.
- Are relationships between individual instances made explicit? Yes. All data is provided in a simple data structure that indicates which instances of data are connected with which instances of metadata.
- Are there recommended data splits? Yes, for each of the scenarios in the datasets, there are three splits: (a) a large training split for training physical prediction models from scratch; (b) a smaller readout-training set that is to be used for training the yes/no binary readout training as described in the paper, and (c) the test dataset on which human responses were obtained.
- Are there any errors, sources of noise, or redundancies in the dataset? We have not found any as of this publication. As these are discovered, they will be fixed and versioned.
- Is the dataset self-contained, or does it link to or otherwise rely on external resources? It is self-contained.
- Does the dataset contain data that might be considered confidential? No.
- Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? No.
- Does the dataset relate to people? No.
- How was the data associated with each instance acquired? What mechanisms or procedures were used to collect the data? How was it verified? Videos (for training, readout fitting, and human testing) were generated using the TDW simulation environment. Online crowdsourcing was used to obtain human judgements for each testing video. During the creation of the simulated videos, the researchers looked at the generated videos by eye to verify if the scenarios were correct (e.g. actually depicted the situations desired by our experimental design). Prior to running the actual data collection procedure for humans, we verified that the experimental websites were correct by having several of the researchers complete the experiment themselves.
- Who was involved in the data collection process and how were they compensated? PIs, students, and postdocs generated simulator-generated videos. For human responses, 200 participants (50 for each of the mechanical properties) were recruited from Prolific and paid $15.50 per hour for their participation.
- Over what timeframe was the data collected? All simulator-generated scenarios were created and human data was collected during the second half of 2022.
- Were any ethical review processes conducted? All human data collection was approved by UC San Diego IRB.
Preprocessing, clearing and labelling.
- Was any preprocessing/cleaning/labeling of the data done? We reviewed the test scenarios to make sure we videos with non-informative situations were not included (e.g., one of the key objects is fully occluded during the entirety of the video). No other preprocessing was done, and labeling was produced automatically by the system.
- Has the dataset been used for any tasks already? Yes, the participants in the human experiments used the data for the single purpose for which it was designed: obtaining detailed characterization of human judgments about physical prediction and latent property inference in simple scenes.
- Is there a repository that links to any or all papers or systems that use the dataset?. No other papers use the dataset yet.
- What (other) tasks could the dataset be used for? None.
- Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? No.
- Are there tasks for which the dataset should not be used? The dataset can only be used to measure abilities of humans or models to make physical prediction based on latent property inference.
- Will the dataset be distributed to third parties outside of the entity (e.g., company institution, organization) on behalf of which the dataset was created? Yes it will be completely publicly available via our project page and the links listed thereupon.
- How will the dataset will be distributed? It will be available via links to the project page and which will refer to permanent Amazon S3 resources.
- When will the dataset be distributed? Immediately.
- Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? The dataset and associated code will be licensed under the MIT license.
- Have any third parties imposed IP-based or other restrictions on the data associated with the instances? No.
- Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? No.
- Who is supporting/hosting/maintaining the dataset? The dataset is hosted on Amazon S3 resource. The associated Amazon S3 account is the institutional account for the CogTools lab (at Stanford).
- How can the owner/curator/manager of the dataset be contacted? The corresponding author of the paper can be contacted via email as described in the front page of the paper.
- Is there an erratum? No. If needed, any future errata will be posted on the project page.
- Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? As this dataset becomes used by a larger audience, we will review the instances for errors that users uncover. These errors will be corrected as they are discovered on an ongoing basis.
- Will older versions of the dataset continue to be supported/hosted/maintained? If newer versions of the dataset are created, these will only be in additional to the existing data. Old versions will be maintained indefinitely.
- If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? No. Making contributions to this dataset requires a very detailed understanding of a variety of components and how they interconnect – physics simulators, scenario generation modules, online psychophysical experimentation platforms, etc. – and we do not contemplate allowing third parties to (e.g.) add new examples of physical scenarios.
Structured metadata. We have not created structured metadata for our project in a format like that in schema.org or DCAT as yet, because we expect that through the review feedback process, the exact structure of what metadata we should provide may change. We will be happy to do this once review is complete. In the meantime, all of our data is available through our project page, which provides a certain level of metadata about the project that we think is appropriate for the review process.
Dataset identifier. At the moment, we provide access to the dataset via Amazon S3 links that are visible via our project page. We have not yet pushed out data into a standard data repository or created a DOI for it. This is because we expect the specifics of how the data is made available to develop during the paper review process. Once this is complete, we will push the data into a standardized data repository and generate a DOI for it.
G Checklist
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes]