Benchmarking Progress to Infant-Level Physical Reasoning in AI
Authors: Luca Weihs
Amanda Yuile
Renée Baillargeon
Cynthia Fisher
Gary Marcus
Roozbeh Mottaghi
Aniruddha Kembhavi
Venue: Accepted by TMLR ()

Abstract: To what extent do modern AI systems comprehend the physical world? We introduce the open-access Infant-Level Physical Reasoning Benchmark (InfLevel) to gain insight into this question. We evaluate ten neural-network architectures developed for video understanding on tasks designed to test these models' ability to reason about three essential physical principles which researchers have shown to guide human infants' physical understanding. We explore the sensitivity of each AI system to the continuity of objects as they travel through space and time, to the solidity of objects, and to gravity. We find strikingly consistent results across 60 experiments with multiple systems, training regimes, and evaluation metrics: current popular visual-understanding systems are at or near chance on all three principles of physical reasoning. We close by suggesting some potential ways forward.

Published in Transactions on Machine Learning Research (10/2022)
Benchmarking Progress to Infant-Level Physical Reasoning in Al
Luca Weihs, 1 Amanda Rose Yuile, 2.+ Renée Baillargeon, Cynthia L Fisher,? 2 Gary Marcus, Roozbeh Mottaghi, Aniruddha Kembhavi
Allen Institute for AI, University of Ilinois at Urbana- Champaign, New York University (lucaw, roozbehm, amik) Gallenai.ory (amandah3, rbaillor, oifishe @illinois.ed edu
{afmarcus @gmail.com com
Reviewed on OpenReview: https: // openreview. net/foruee id- (N
Data, Code, and Videos available at: https: //allenai. org/ project/ inneettiifleee level
Abstract
To what extent do modern AI systems comprehend the physical world? We introduce the open-access Infant-Level Physical Reasoning Benchmark (INFLEVEL) to gain insight into this question. We evaluate ten neural-network architectures developed for video understanding on tasks designed to test these models' ability to reason about three essential physical prin- ciples which rescarchers have shown to guide human infants' physical understanding. We explore the sensitivity of each AT system to the continuity of objects as they travel through space and time, to the solidity of objects, and to gravity. We find strikingly consistent re- sults across 60 experiments with multiple systems, training regimes, and evaluation metrics: current popular uisuua-understanddngg systems are at or near chance on all three principles of physical reasoning. We close by suggesting some potential ways forward.
1 Introduction
As artificial intelligence increasingly begins to power automated vehicles and robots' actions, generaliza- tion to the infinite variety of real-world environments requires that these Al systems reason use reliable understanding of the physical world. For instance, automated vehicle that does not understand that permanence is a general attribute of all objects may not react appropriately when a novel object becomes occluded on a roadway. While there is an active community of researchers pursuing this direction (Lake et al., 2017), most current AI work is focused on end-to-end learning with deep-learning architectures (Le- Cun et al., 2015). As they have grown in size and leveraged increasing amounts of data, they have shattered records across numerous computer-visioe
benchmarks, including image classification (Zhai et al., 2021),* object detection (Xu et al., 2021),² action recognition (Ryoo al., 2021),* 3 visual question answering (Wang et al., 2021),* and complex reasoning tasks in synthetic environments (Yi et al., 2018). And yet, even these remarkable results give researchers little confidence that today's 's best models have anything approaching human-level world understanding (Serre, 2019; Yuille & Liu, 2021).
Developmental psychologists have shown that by 4.5 months of age, infants correctly reason about objects' displacements and interactions in many different physical events. The presence this ability at such a young
* -Wok completed, in part, while intern at the Allen Institute for Al.
1 1990% Top-1 accuracy ImageNett classification
260 box AP COCO object detection
885% Top-1 accuracy at Kinetics-400 action recognition
4 400 Top-I accuracy at visual question answering
5999 accuracy and 990 Ave-per-q at the diagnostic CLEVR and CLEVRER datasets

---

Published in Transactions on Machine Learning Research (10/2022)
Figure Evaluating physical-reasoning abilities of AI models. Inspired by studies performed with infants, INFLEVEL asks AT models if they surprised by videos of physically plausible and implausible infants surprised by Solidity violation where toy rolls through box (Baillargcon, 1986). Bottom: schematic and example from INFLEVEL; models that understand Solidity should be surprised that the object does not move with the cover, as though it somehow passes through the back of the cover.
events. Top:
age has led to the formulation of the core-knowlledge hypothesis, which states that infants are born with an innate understanding core laysscal--easoniig principles (Spelke et al., 1992). These principles are now thought include Persistence (all other things being equal, objects persist as they are in time and space), Inertia (absent external force, moving objects follow straight trajectories and stationary objects remain stationary), and Gravity (objects fall unless supported). Persistence is composed of five subcategories: Continuity, Solidity, Cohesion, Boundedness, and Unchangeableness,, for total of seven principles (Lin et al., 2022). We capitalize the names of these principles to emphasize that these terms have meanings within developmental psychology that may differ slightly from everyday use. While infants cannot identify 1000 object categories or answer complex questions about images, their innate understanding of these core principles is the foundation upon which remarkable capacity for physical understanding is built. We believe that assessing
aideo-understanding model's knowledge must include a direct evaluation of its understanding of these core principles. If model's representation of the world failed to embed the understanding that objects should fall when unsupported, then would have serious concerns that this model was performing any meaningful physical reasoning, regardless of how strongly it performed on computer vision benchmark.
To assess contemporary AI systems' ability to reason about some of these essential aspects of the physical world, we present the Infant-Level Physical Reasoning Benchmark (INFLEVEL). We focus on three principles that have been extensively studied, Continuity, Solidity, and Gravity, modeled on past experimental work in developmental psychology (Lin et al., 2022; Baillargeon, 2008; Baillargeon et al., 1985; Luo al., 2009; Needham & Baillargeon, 1993). These principles enable infants to correctly interpret myriad of simple object interactions and thereby they provide good objective for study. These principles can also all be studied with a consistent experimental setup (a single human manipulating two objects on table). In this prior work, infants' understanding is typically assessed using the filltiionof-expeettaion (VoE) paradigm (Lin et al., 2022) in which infants are presented with a physically implausible event, which violates the principle under investigation, and a physically plausible event, which conforms to this principle. With appropriate controls, greater surprise the implausible event, with looking time as the canonical surrogate measure quantifying surprise, is taken to indicate that the child brought to bear the relevant principle to form expectations about how the events would unfold and subsequently detected the violation in the physically implausible event. In physical-reasoning experiments, infants tend to look longer at unexpected outcomes, somewhat analogous to an adult having a double-take upon seeing magic trick. In essence, the premise is that any agent that can understand the world should be able to discriminate between ordinary events (e.g., inert objects falling towards the earth) and unusual events (e.g. incrt objects floating midair, unsupported). With this VoE paradigm in mind, INFLEVEL contains two video datasets, INFLEVEI-LAB and INFLRVEL-SIM, with videos involving physically plausible and implausible object interactions (see Fig. 1).
2

---

Published in Transactions on Machine Learning Research (10/2022)
INFLEVEL-LAB contains ~5,700 videos filmed in an infant-cognition lab, divided into 3 subsets focusing Continuity, Solidity, and Gravity. These principles have the advantage that they do not require elaborate experimental set-ups, even when creating physically implausible videos (see Sec. 3). We propose quantifying a model's surprise using metries designed for out-of-domain (OOD) detection. This novel perspective on quantifying surprise in artificial models allows us to evaluate any visual model that produces a veclor representation of an input video. We pair our natural video benchmark INFLEVEI -LAB with INFLEVEL- SIM: ~75,000 videos collected using the robotic-arm enabled agent within the A12-THOR environment (Kolve et al., 2019; Ehsani et al., 2021). Mirroring its natural video counterpart, LEVELLLM includes videos of plausible and implausible interactions between objects but contains an order of magnitude more videos due to the inexpensive nature of creating videos within a simulated environment. Unlike related benchmarks (see Sec. 2), our inclusion of a real-video datasel enables us to Lest models trained on natural- video datasets with minimal domain gap. Likewise, as we also wish to evaluate embodied agents (generally trained in synthetic environments like A12-THOR), our inclusion of INFLEVEL-SSM reduces the domain gap in evaluating such agents.
INFLEVEL has several unique design considerations that break convention with usual benchmarks in computer-vision: (1) Evaluation only: INFLEVEL. is designed to be used only for evaluation. As discussed in Sec. 3, allowing training (even of a linear probe) may result in evaluation metrics confounding physical understanding with feature extraction and matching. This result underlines the subtlety of evaluating the expectations embedded in model representations. We stress that INF is designed to assess the physical reasoning abilities of already trained models; we do not allow for any training or ine-tuning on our data as this would lead us to evaluating "what model can learn" and not "what model currently understands." Disallowing training makes INFLEVEL challenging; this is by design and is how infants are evaluated. (2) Easy for humans: As INFLEVEL builds on infant experiments, adults evaluated on it obtain near 100% ac- curacy in our exploratory experiments. Given this "triviality," " a model's failure emphasizes the substantial gap between the plysical-reasoning capabilities of infants and artificial agents. (3) Controlled: The events depicted in INFLEVEL videos are highly controlled and mirror events shown to infants. This tight control opens the door for designing heuristics that can obtain high performance. We emphasize that LEELEVEL is intended to be a diagnostic methodology for studying the physical knowledge embedded in learned rep- resentations: poor performance suggests fundamental failure of the representation to embed physical principles; high performance, on the other hand, must be understood in the context of how the model was trained and the inductive biases it employs.
We evaluate a range of otate-of-thh-art (SoTA) video models trained on popular large-scale datasets (e.g.. HowTo100M Kinetics400, SSv2, elc. (Sigurdsson et al., 2016; Miech el al., 2019; Kay et al., 2017; Goyal et al.,, 2017; Damen et al., 2018). These models perform poorly (marginally above random chance) when presented with the Continuity violations; on Solidity and Gravity violations, they perform even worse. On INFLEVEL-LAB, all models attain near-random performance across all principles suggesting that learning from embodied interaction may require more than training agents for task completion.
In summary, our contributions include: (i) INFLEVEL, a benchmark containing thousands of natural and synthetic videos designed to evaluate AI systems' understanding of core physical principles, (ii) a proposal to use OOD detection as a flexible proxy for surprise in artificial agents that does not require retraining, and (iii) exhaustive experimentation demonstrating that SoTA vidco understanding and embodied models find INFLEVEL exceptionally challenging. Data and code will be made publicly available.
2 Related Work
Physical Reasoning in Infants. The study of physical reasoning in infants has long history dating to the seminal works of Piaget (Piaget, 1952; 1954). Piaget's experiments led him to hypothesize that infants' physical reasoning abilities were relatively crude, even lacking object permanence until 8-9 months of age. In part due to the introduction of the nfolatinn-o-eeppettatin (VoE) paradigm (Baillargeon et al., 1985), these hypotheses have boen largely overturned. Infants have surprisingly sophisticated physical-reasoning capabilities; for instance: 2.5-month-olds expect objects to become visible when moving between occlud- ers (Aguiar & Baillargeon, 1999), 4.5-month-olds expect inert objects to fall when unsupported (Needham
3

---

Published in Transactions on Machine Learning Research (10/2022)
& Baillargeon, 1993), and 3.5- to 4.5-month-olds expect objects to follow straight-line trajectories and to collide with hidden objects (Baillargeon, 1986; 1987). See the chapter of Tin et al. (2022) for an in-depth survey of these advances and for a discussion of the core-knowledge hypothesis which suggests that infants are imbued, at birth, with a "skeletal framework of core physical principles" " which they leverage to understand the world. Inspired by this work, we design INELEVEL to test artificial agents' understanding of a subset of these core physical principles.
Physics Reasoning Benchmarks and Models. There is substantial history of work studying physical reason- ing in artificial models (Lake et al., 2017). Here we will highlight the pieces most relevant to INFLEVEL. In INFLEVEL, agents are required to reason about objects that are hidden behind occluders, inside containers or tubes, or under covers. Relatedly, researchers have considered knowledge-bbee basigge--beeeeeeee systems for reasoning about container manipulation (Davis et al., 2013), built generative models of containers' contents to guide active search (Wong et al., 2013), designed systems to reason about containment in videos (Liang et al., 2016) and to track occluded objects through long temporal sequences (Liang et al., 2018), and considered parse graphs as means by which to track object states from multiple views with potential occlusion (Yuan el al., 2020). Other works have studied building models of intuitive physics by using inverse graphics to approximate an observed scene inside of a game engine (Wu et al., 2017), Xu et al. (2019) introduce DenscPhysNet uses imput depth maps to predict object state changes after actions in self-supervised manner, and the Dynamic Concept Learner (DCL) uses object-centric inductive biases, a learned dynamics model, and graph NNs to obtain SoTA performance on the CLEVRER benchmark (Chen et al., 2021). These works present new methodology and models while our focus is on designing a benchmark to evaluate existing models, potentially trained for unrelated end tasks.
Of recently proposed physical reasoning benchmarks, most similar to INFLEVEL are IntPhys (Riochet et al., 2018) and two unnamed datasets from Piloto et al. (2018) and Dasgupta et al. (2021), which we call PolotoVoE and Dasgupta VoE respectively. These benchmarks explicitly reference the VoE paradigm and evaluate models by testing their ability to discriminate between physically plausible and implausible events. Our work differs from theirs along two critical dimensions. First, these other benchmarks are set within stylized simulations, with simple geometric objects that move without any apparent external force acting upon them. In contrast, we include natural vidcos in INFLEVEL-LAB, use diverse houschold objects, and always include agent who explicitly manipulates objects. Second, these benchmarks are designed with different use-case in mind: they all include large datasets of plausible videos meant to be used to train models to physically reason explicitly. INFLEVEL is evaluation-only: we have no training dataset by design. This is because we are interested in testing the capabilitics of models that have already been trained, to understand whether these models have learned to reason about core principles through their normal course of training. We also note that we focus on very simple physical violations; by contrast, violations in IntPhys are sufficiently complex that even humans obtain high error rates (up to an 47% error rate in the most challenging setting!). In recent concurrent work, Piloto et al. (2022) have released the Physical Concepts dataset, a dataset similar to VoF but with richer coverage of physical concepts; they also introduce "PLATO" model which, given preprocessed information in the form of groundtruth semantic masks and object indices, attains impressive accuracy on their dataset. Our work differs from this work just as it does for the original Pol benchmark as described above. Additionally, our benchmark could not be directly tested on their model as is, since their system, unlike the systems that we did examine, docs not work purely from pixcl-basod input.
Several other benchmarks have recently been proposed. Physion (Riochet et al., 2018) is set in the simu- lated ThrleDWorld environment (Gan et al., 2020) and studies physical understanding by requiring mod- cls to predict whether two objects will come into contact in the future. PHASE (Netanyahu et al., 2021) presents 2D animations of abstract social interactions with associated social recognition and prediction tasks. AGENT (Shu et al., 2021) generates simulated 3D videos to probe understanding of how agents plan and execute efficient, goal-directed actions. BIB (Gandhi et al., 2021) tests agents' capacity for predicting other agents' behavior using simulated 2D/3D videos. CLEVRER (Yi al., 2020), built upon CLEVR (Johnson et al., 2017), tests models' ability to answer questions about potential collision events. Physics 101 (Wu et al., 2016) presents natural vidcos to study how models may learn physical attributes from unlabelled videos. We also highlight that recent years have seen the proliferation of high-fidelity, physics-enabled, sim-
4

---

Published in Transactions on Machine Learning Research (10/2022)
Revealing the open bottom the tube
Implaus ble: object doesn't fall through the open bottom
4.
Figure 2: Implausible Gravity examples. Four frames taken from Gravity trials INFLEVEL-LAB and INFLEVEL-SIM. The operator shows that the primary object is tube and then drops an object into The object fails fall through the open bottom of the tube, creating an implausible outcome.
ulated worlds (Kolve et al., 2019; Xiang el al., 2020; Szot et al., 2021; Li et al., 2021) which allow explicit embodiment and learning-frood foamiingfoom--nteracctoo
3 INFLEVEL: The Infant-Level Physical Reasoning Benchmark
INFLEVEL is designed to evaluate the core plysical-reasoning abilities of existing vidco-understanding models. To accomplish this goal, INFLEVEL must balance two competing goals:
1. Simplicity. To ensure that we are testing models' core physical understanding, the videos in INFLEVEL should be simple, closely replicating the lab settings used in infant studies. While ideal models would be both robust to distraction and able to reason precisely in complex multi-object scenes, such complexity would only add noise to our desired signal: would be challenging to know if model's failure was due to lack of physical understanding or, instead, to confusion in the presence of distractors.
To achieve simplicity, make several design decisions. All events depict interactions between precisely two objects, a primary and a secondary object. The primary object is a container, cover (an upside-down container), or a tube (a bottomless container); the secondary object is placed into the container, under the cover, or into the tube. All objects are simple, household objects (e.g., cups, bowls, blocks, or crayons) that infants may encounter everyday life (see Fig. 5). The primary and secondary objects are manipulated on table by an operator (a human in INFLEVEL-LAB and a robotic agent in INFLEVEL- SIM). The interaction between the two objects is either physically plausible violates a core physical principle (see Fig. 2).
2. Diversity. We must have enough visual diversity in our videos that high performance is not simply product of a model accidentally latching onto some statistical correlation present across videos. To diversify our videos, we use a large collection of objects, provide three camera angles, and vary the direction in which trials are performed. See App. Fig. A.1 for an example of such diversity. In INFLEVEL-SIM, we also change LN the human operator
the lexture and color of the table and background programmatically also varies their clothing. Importantly, our strategies for increasing diversity are cosmetic and do not require agents reason about additional objects new physical concepts.
3.1 Experimental Designs
We now discuss the physically plausible and implausible event videos used to test models' understanding of Continuity, Solidity, and Gravity. In each case, we include two plausible-implausbbe pairs of trials.
Continuity. The principle of Continuity, a corollary of Persistence, states that an object should not spon- taneously appear or disappear (Lin et al., 2022). Infants as young as 2.5 months of age are able to reason about Continuity violations. For instance, they detect a violation when an object that passing behind two occluders positioned a short distance apart fails to become visible between them (Aguiar & Baillargeon, 1999). Similarly, they detect violation when cover is lowered over object, slid to the side, and then
5

---

Published in Transactions on Machine Learning Research (10/2022)
lifted to reveal no object (Wang al., 2005). Our Continuity trials involve covering events, and each trial has three parts (separated by short, 0.5s, cuts):
Plausible
Implausible
1. Secondary Object Familiarization - The operator picks up the secondary object, moves it to the frame's center, and then places it back onto the table.
Continuity
2. Primary Object Familiarization The operator picks up the primary object (always cover in these trials), rotates its opening toward the camera. to show that it is empty, and then returns it to the table.
Object continues to esist (V)
Cbyect disappears (VI)
Solidity
3. Continuilyy Test - The operator picks up the cover, low- ers it over the secondary object, slides it across the table, and finally lifts it into the air. In a physically plausible event, the secondary object revealed when the cover lifted. We label this VV (the secondary object is visible at the start and end). In physically implausible event, the sccondary object is absent when the cover is lifted, labeled VI Disible-to-invisible See Fig. 3 for schematics of VV and VI.
Object moves cover (UI)
Objject passes through cover (UV)
Gravity
Cloject through tube
Object dessn't fall through tube
Illustrations of (VV,VI), (UI,UV), & (CV,CI) events. In Continuity, Solidity, and Gravity trials respectively,
Figure 3:
Our Continuity trials also include II and IV, where no secondary object is present at the beginning. Evaluations now reverse, with IV being physically implausible (this also a Continuity violation), and II being physically plausible. These additional trials ensure that the high-level statistics of our trials are well-balanced, e.g. there is not clear bias where all plausible videos include two objects in the final frame.
Cover on tob
Unclipedd Cover
Clpped Cover
Solidity, Another corollary of Persistence is Solidity, which states that two solid objects should not capable of passing through one another (Lin et al., 2022). By 3.5 4.5 months of age, infants are sur- prised when screen rotates through the space oc- cupied by an object in its path (Baillargeon, 1987), when a toy car rolls through the space occupied by
Solicity Famili inliiintooonn
Backside cover remaved
Consainor on table
Unclipped Container
Cipped Containor
Bottom vontaine: removed
Grovity Famillorcaton
an object (Baillargeon & DeVos, 1991), or when an object is lowered into a container that is then moved
forward and to the side to reveal the object standing in its initial position (Hessos & Baillargeon, 2001). Our Solidity trials' primary object is cover (U); additional trials involve clipped cover (C), which has been modified so that it is open at the back and functions as an occluder, see Fig. 4. Each Solidity trial has two parts (separated by short cuts):
Figure 4: Clipped and unclipped objects. In the Solidity and Gravity trials, primary objects can be ci- ther clipped or unclipped. In the Solidity trials (top), covers have no back; in the Gravity trials (bottom), clipped containers have no bottoms.
1. Primary Object Fomiliarization The U cover initially rests on the table. The operator lifts it, rotates its opening forward to show that it emply, and then returns it to its original position.
2. Solidity Test The U cover and secondary object stand next to each other one side of the table. The operator lifts the cover, lowers it over the object, slides the cover forward, and finally slides it to the other side of the table. In the physically implausible event, the object is visible in its original position, as though it did not move along with the cover (UV); in the physically plausible event, the object is not visible (UT), as though it did move along with the cover. See Fig. for schematic examples of the UI and UV cases.
Additional trials use the clipped cover (C); the missing back of the cover is revealed in the familiarization event when the cover is rotated forward. Plausibility physically implausible (this is inertia violation, as the clipped cover cannot exert force on the object), and theh-e- (CV) being physically plausible. These additional trials ensure that our Solidity
reverses, with invisible-at-the-end (CT) now being

---

Published in Transactions on Machine Learning Research (10/2022)
Continulty Solidity Gravity
600
2000
500
1500
400
300
1000
200
500
100
(b) Figure 5: Object type statistics. Number of Limes various secondary object types appear in videos for (a) LNFLEVEL-LAB and (b) INFLEVEL-SIM.
(a)
evaluation set is balanced: there are equal numbers of plausible/impllusibbbe videos where secondary objects are/aren't visible in the original positions.
Gravity. This principle states that unsupported objects should fall. By at least 4.5 months of age, infants are surprised when objects float unsupported (Needham & Baillargeon, 1993; Luo et al., 2009). Our trials involve a clipped container (C) that has been modified so that it is open at the bottom and functions as a tube; additional trials include a standard, unmodified container (U), see Fig. 4. These trials occur in two parts (separated by short cuts):
1. Primary Object Familiarization The C container rests on the table. The operator lifts it, moves it to the center of the frame, rotates it to show that it is bottomless, and then returns it to its original position.
2. Gravity Test To start, the C container and secondary object rest next to each other on one side of the table. The operator lifts C in one hand and moves it to the center of the frame (because C is upright, its open bottom cannot be seen). The operator then lifts the secondary object, lowers it into the opening of the container, and then lets go of the object. In the physically plausible event, the object. falls through the open bottom of the container and onto the table (CV). In the physically implausible event, the object remains in the C' container, even though there is solid surface to block its fall and it is, therefore, unsupported (CI). See Fig. 3 for schematic examples of the CV and CI cases.
Additional trials use U, whose solid bottom can be seen in the familiarization event when the container is rotated forward. Events now reverse, with visible at the end (UV) being implausible (this is a Solidity violation, as the object appears to pass through the bottom of the container), and invisible at the end (UI)
being plausible. As for Solidity, these additional trials help balance the evaluation set.
Concept Overlap. While the above trials would ideally evaluate each principle in isolation, it is impossible, even in infant studios, to completely disentangle all physical reasoning principles in a single event. Our design choice to use similar events with hidden objects across trials means that all of our violations involved Continuity to some extent: to detect them, an agent had to assume that the secondary object continued to exist after it became hidden under the cover (Continuity and Solidity violations) or inside the tube (Gravity violation). That issue aside, cach violation focusod primarily on one principle (i.e., the hidden object appeared to wink out of existence, to pass through a solid surface, to remain stable without support), making it possible to compare responses across principles.
3.2 INFLEVEI-LAD: Natural Videos
LNFLEVEI-LAB consists of 5,772 videos filmed within an infant-coguition lab (2268 Contimuity 912 Solidity, and 2592 Gravity vidcos). We use 54 unique secondary object instances from 29 categories (sce Fig. 5a) and 20 unique primary object instances from categories (bowls, cups, and pots). Videos were filmed at 29.97 FPS in 1280x 720 resolution. Continuity, Solidity, and Gravity trials take approximately 21, 16, and
7

---

Published in Transactions on Machine Learning Research (10/2022)
14 seconds to complete respectively. Further details about this datasct, inchuding a description of how we film events that appear physically implausible without visual special effects, can be found in Appendix A.
3.3 INFLEVEL-SIM: Synthetic Videos
While our primary interest lies in understanding the capabilities of models trained on real-world videos, researchers have argued that embodiment and interaction are critical for developing truly intelligent agents (Fridman & Malik, 2020) and presented evidence that developing general AI requires cmbodi- ment (Kremelberg, 2019). Also, recent work has shown some evidence that represenlations learned from gameplay in simulated environments have desirable properties (Weihs et al., 2021). As large collection of the research in embodied AT is done in simulation (as real-world training is expensive and slow), we developed the INFLEVEL-SIM benchmark to evaluate embodied agents' physical-reasoning capabilities.
LNFLEVEL-SIM consists of 75,336 videos collected in the simulated A12-THOR, environment (Kolve et al., 2019) using the robotic-arm-enablled ManipulaTHOR agent introduced in (Ehsani et al., 2021). We chose to use AI2-THOR as it, for our purposes, provided useful feature set: physics-based arm interactions, high visual fidelity, material randomization, and a diverse set of object instances. Alternative potential embodied-Al simulators include iGibson 2.0 (Li et al., 2021) and Habitat 2.0 (Szot et al., 2021). To record our trials of interest, we had to extend the existing arm functionality to include new action types (e.g. arm rotation) and to add a second arm for use in our Gravity trials where two objects must be manipulated at once (soe Fig. 2). These 75k videos are approximately evenly distributed across Contimuity, Solidity, and Gravity trials (each having >24 videos); they involve 574 unique secondary objects from 57 categories (see App. Fig, 5b) and 167 unique primary objects from eight categories. Videos were recorded at the same FPS and resolution as in INFLEVEL--LA and the Continuity, Solidity, and Gravity trials take approximately 30, 23, and 25 seconds to complete respectively. See App. B for more details.
3.4 Evaluation
We wish to evaluate models that have already been trained on external data sources (e.g. Kinelics400) without making strong assumptions on how they structure their output. To this end, we will assume merely that a given model f can produce a vector representation of an input video v. A common strategy used to measure the quality of representations is via finear-probing the model f is frozen and a linear decoder is trained using the representation from f as input to accomplish some auxiliary task. This approach works well in most settings it captures a common use-case for representation learning methods: low-sample transfer learning. Unfortunately, as we discuss in detail in Appendix E.1, even training simple linear decoder on INFLEVEL can result in conflating physical understanding with other related quantities.
For this reason, INFLEVEL is an evaluation-only benchmark: no training on INFLEVEL is allowed. This has an unfortunate implication we must. overcome: models that wish to report scores on INFLEVEL must be able to provide a scalar "surprise" score for every input video. This requirement, used by other bench- marks (Riochet et al., 2018), is limiting it requires that anyone wishing to evaluate on INFLEVEL to train special "surprise" decoder on their model using some external data source. To circumvent this problem, we model surprise as out-of-domain (OOD) detection using the intuition that models with sufficient physical understanding should consider physically implausible events more out-of-domain than physically plausible ones. In Sec. 4, we propose several OOD metrics, each of which takes a representation of a video and returns a scalar quantifying how out-of-domain the video is. While we show, in Sec. 4, that this OOD approach is empirically promising, it is easy to show that there is no surprise metric which can be used to detect physical understanding for all possible models (see App. E.2). Given this, anyone evaluating on INFLEVEL is free to define their own surprise metrics so long as: (1) the same metric is used across all subsets of INFLEVEL (i.e. there should not be one metric for Continuity and another for Gravity) and (2) these surprise metrics are neither trained on INFLEVEL nor designed explicitly to exploit regularities in INFLEVEL data (which would be an implicit form of training). Going forward, we hope that researchers will continue to improve and refine the model-agnostic OOD surprise measures we propose.
Now, without loss of generalization, assume that a model has produced a surprise scalar s; for every video 02 in INPLEVRI-LAR. We would like to aggregate these scores into single number representing the model's
8

---

Published in Transactions on Machine Learning Research (10/2022)
performance. Surprise scores are often incomparable: if U2 is a video with sponge and by is video with an apple, may be larger than 8j simply because the model has never seen a sponge during training but has seen apples. To account for this our evaluation is conditional. Recall (Sec. 3.1) that in our Continuity trials, for each primary/ /sconndary object pair p we have four associated trials VV, II, IV, and VI. Let (p); and svI(p) be the four surprise scores associated with the p pair. Then we compute the aggregate score as
Syv()), s (1)()))11v
IE
1
E (ps < sy((p)
4.n
=E(VV.II) VE(IV.VTy
where the first sum is over the set of all distinct pairs p in the Contimuity set. This quantity can be interpreted as the average accuracy in discriminaling between plausible and implausible events. Evaluations for Solidity and Gravity are analogous.
Metric
Max-Softmax Sh&:PS
NN (1.2)
Mahulanobis Sh&PPS
von Misess Fisher
4 Experiments
Ang.
Sh&PS
Sh&PPS
Name
0.70
0.85
0.90
0.80
PS
0.61
0.83
0.92
0.85
4.1 Metrics.
Sh
0.65
0.53
0.50
0.66
Detecting whether samples are out-of-domain (or, relatedly, outliers) is challenging problem with long history (Grubbs, 1969). Recent work (Fort et al., 2021), has shown that very simple OOD met- rics, when paired with representations from large- scale models, can result in strong performance in outlier-detection benchmarks. These results sug- gest that such OOD metrics have the potential to function a flexible proxy of surprise for deep net- works. In INFLEVEL, physically plausible videos are OOD visually (being filmed in controlled setting from a third perspective) while physically implausi- ble videos are OOD both visually and dynamically (violations of physical principles demonstrate OOD enviromment dynamics). Thus, for OOD metrics to be useful as a measure of surprise for INFLEVEI. where we wish for implausible videos to be more surprising than plausible ones, we must have evidence that these metrics are "additive". That is, we wish for these OOD metrics to consider videos with both visual and dynamic abnormalities to be more OOD than those with only visual abnormalities.
Metric
Mox-Softaxx Fr&BBW
NN (1.2)
Mahalanohis Fr&BBW
Fisher Fr&-BW
Aug.
Fr&
Nome
0.69
0.85
0.57
0.84
Ft
0.69
0.56
0.49
0.78
BW
0.53
0.90
0.63
0.77
Table 1: Additivity of OOD Metrics. For four OOD metrics: frequency with which videos with com- posed column-specified augmentation scored as being more OOD than those with the row-specified augmen- lation. e.g. the top left-most, entry denotes that 70% of the time the Max-Softmax OOD metric was greater for ShupSS-ngmenttd vidcos than for vidcos with no augmentation. All but the Mahalanobis met- ric consider the composed augmentations (Sh&PS and Fr&BW) to be more OOD than the others.
To provide evidence for the above, we design the following experiment. Using the popular SlowFast 8x8 R50 architecture (Feichtenhofer et al., 2019) pretrained on Kinetics400 (Kay et al., 2017), we generate representations on æ16k videos in the Kinetics400 validation set. We then repeat this procedure on six augmented versions of this set: PS, Sh, Sh&PS, BW, Fr, and Fr&BW. PS and BW correspond to visual augmentations (PS swaps patches in individual frames, BW makes the video greyscale), while Sh and Fr correspond to augmentations of dynamics (Sh shuffles video frames, Fr freezes a subset of them). Sh&PP and Fr&BW are then both compositions of visual and dynamic augmentations.
We compute four OOD metries for each video: Max-Softmax (Hendrycks & Gimpel, 2017), Mahalanobis (Lee metric we call NN (L22,, and a metric similar to Mahalanobis but using von Mises-Fisher distribution in place of the Gaussian. Except for the Max-Softmax metrie, these metrics require additional dataset of videos to be used in-domain examples. As the above model was trained on Kinetics400, we use the Kinetics400 training set for this purpose. See App. C for more metric details.
et al., 2018), a nearest-ncighhor--asse
Table 1 shows the frequency with which, for each OOD metric, various augmented validation videos are considered more OOD than one another. If an OOD metric is additive, we expect to see that the Sh&PS augmented videos are more OOD than the PS and Sh augmented vidcos (and similarly for the Fr&BW, BW, and Fr augmentations). This is the trend we observe for all metrics except for the Mahalanobis metric. This provides evidence that OOD metrics can be fruitfully applied in INFLEVEL as proxy of surprise.
9

---

Published in Transactions on Machine Learning Research (10/2022)
For simplicity, and as the Mahalanobis metric did not consistently show the additive property, we will compute three of the above four metrics across all models in our experiments: the Max-Softmax, NN (T.2), and von Mises-Fisher metrics. Moreover, in what follows it will be convemient to be able to aggregate the results from those three metrics in a single number. To this end, we define a "majority vote" (MV) score which, recalling Eq. 1, we compute as
1 EED E E
1mmajority of surprise metrics consider x less surprising than y)
(2)
4 4.n P =E(VV.II) DE (IV.VT)
4.2 Baselines.
We consider a large collection of uideo-understanding and embodied Al models. These include: TimeS- Former (TF) (Bertasius et al., 2021) and Motionformer (MF) (Patrick et al., 2021), two recent transformer architectures attaining SoTA performance al action recognition; Channel-Separated Convolutional Networks (CSN) (Tran et al,, 2019) and X3D (Feichtenhofer, 2020), two efficient 3D- -DDooovolutionll video networks achieving high performance across many tasks; Slow (S) SlowFast (SF) networks (Feichtenhofer et al., 2019), two popular 3D convolution architectures; Two-Stream Inflated 3D ConvNet (I3D) (Carreira & Zis- serman, 2017), an early 3D-conv. action recognition model; Contrastive Video Representation Learning (CVRL) network (Qian et al., 2021), a 3D-ResNet-550 backbone trained using a self-supervised contrastive approach achicving impressive performance in the lincar-ovaluation setting: and finally two embodicd AI models trained for the ArmPoint.Nav (Ehsani et al., 2021) and Robo ObjectNav (Deitke al., 2020) tasks set in the A12-THOR simiulator (Kolve et al., 2019), these two models process input images using CNN backbones before passing features to GRU (Conv2GRU).
4.3 Results.
We now describe our results when evaluating the above bascline models INFLEVEL-LAB and INFLEVEL- SIM. Several of the OOD metrics require reference to set of videos for, e.g., nearest- com- putations. For fair comparison across models, we create a fixed query set we call JOINTTRAIN of ap- proximately 25k videos comprised of roughly 5k videos from each of the training splits of the Charades (Ch), HowTo100M (HT100M), Kinetics400 (K400), Something-SSmetting v2 (SSv2), and EpicKitchens100 (FK100) datasets (Sigurdsson et al., 2016; Miech et al., 2019; Kay et al., 2017; Goyal et al., 2017; Damen et al., 2018). In App. E we show that similar results hold when using different query sets.
INFLEVEI-LAB. Excopt for the embodied Conv2GRU models which we train ourselves using (Weihs et al., 2020), see App. F, we use publicly available pretrained models of all baselines. Several of these were available from the PyTorchVidee model zoo (Fan et al., 2021) while others were taken from authors' open-source code. As training dataset may be of importance to success INFLEVEL, we consider variants of these basclines pretrained on the datasets mentioned above.
Table 2 displays the results for INFLEVEL-LAB using the "majority vote" (MV) score described in Sec. 4.1. See App. Table F. for results using individual surprise metrics. For the Solidity and Gravity trials, we see no evidence that these models detect these principles are being violated. While one model (SF-100 pretrained on K400) attains a performance of 0.55 (random performance being 0.50), on the Solidity trials this value is not statistically significant at a 0.01 level. In the Continuity trials, there a stronger signal: many models score above chance albeit their performance is still quite poor (no model scores above 0.6). This suggests that select models may have begun form weak understanding of Continuity. Surprisingly there seems to be little relationship between networks' performance on their training dataset and their performance INFLEVEL: the Slow-50 models are perhaps the most successful architecture despite consistently under- performing other models. This may suggest that better-performing models overfit to their target domain and learn to rely on domain-specific signals that do not generalize to INFLEVEL. Interestingly, several models have scores that are well below chance implying that they find physically implausible videos less surprising than plausible ones. It is tempting to simply negate these model's surprise scores (thus giving above-random performance), we discuss these results further in App. D. In Appendix E.6 we present additional results showing that several models consistently able detect differences between trials in INFLEVBI.- LEBB but
10

---

Published in Transactions on Machine Learning Research (10/2022)
that these models' notion of surprise seems to focus on simple high-level visual attributes (e.g. the number of objects visible at the start and end of video) rather than on the more subtle notion of physical plausibility.
INFLEVEL-SIM. Results on INFLEVEL-SIM using the MV metric can be found in Table 2 under the INFLEVEI-SIM heading. See App. Table E.3 for per-surprise-meticc results. In-depth discussion is de- ferred to App. E but we highlight that no models perform at a statistically significant level above chance at Contimuity, suggesting that there may be some characteristic difference in how models interpret Contimuity in simulation. Several models attain a tiny (maximally 0.51) but statistically siguificant performance above random chance at Gravity; this is likely an artifact of the number of tests we run simultancously.
What violations can these models detect? The above results suggest that existing models do not have infants' highly general reysical-reasoning capabilities. This null result raises two questions: "are our surprise metrics meaningful for the large collection of models under consideration?" and "can we better understand the types of violations that these models can detect?". To better answer these questions, we run an additional collection of experiments similar to those done in Sec. 4.1. In particular, we collect JOINTEVAL, dataset of 5400 videos comprised of approximately 1k videos from evaluation splits of the Ch, HT100M, K400, SSv2, and EK100 datascts. We then create two variants of JOINTEVAL by independently applying the Sh (frame shuflling) and PS (patch swapping) augmentations to all videos in JOINTEVAL. Finally, we compule surprise scores across all three datasets (original and two variants) and calculate the frequency with which, using the MV score, the various models considered the augmented videos to be more surprising than the originals. The results of these evaluations are given in the last two columns of Table 2.
The results show several surprising trends: (a) Amazingly, the CVRL model is nearly perfect detecting Sh augmentations suggesting that self-surprised contrastive learning builds a robust understanding of the flow of time; (b) The transformer-basee architectures do not consistently consider Sh augmented vidcos to be surprising, perhaps suggesting that the lack of explicit ordering of inputs in transformer models makes these models less capable in reasoning about time. Interestingly, the TF models generally consider the PS videos to be more surprising than their unaugmented counterparts while the reverse is true for the MF models. (c) Even the untrained SF-50 model performs above chance in detecting the Sh and PS augmented videos demonstrating the power of the inductive biases inherent in 3D convolutional networks. Together, these results provide evidence that our surprise metrics are sufficiently sensitive to give insight into what violations models can detect and allow for meaningful comparisons across models.
5 Discussion and Limitations
Our results show that modern AI models lag well behind human infants when reasoning about three core physical concepts. Despite extensive visual experience, existing models struggle to identify violations of these principles despite their ability to accomplish other complex tasks (e.g. action recog.). This failure does not imply that the enterprise of building intelligent machines is fruitless. Instead, it implies that new research will be required to build systems that can adequately interpret novel scenarios. Pragmatically, this raises a question, given the growing competencies of artificial agents: should we care whether our models are capable of physical reasoning? In the short term, for some "internet AI" settings, the answer is likely no: for model designed to automate a task that is purely virtual (e.g., recommending videos), capacity for physical reasoning is of little importance. In the medium to long term, however, we identify two primary reasons why benchmarks like INFLEVEL are worth taking seriously.
First,, while increasingly capable, modern models suffer from a lack of trust due to their reputation as black- box mattern-matchers. INFLEVEL allows the community to track progress towards physical rcasoning and provide evidence that new models form meaningful expectations about the world. If model cannot reason about basic Continuity violations, we should not trust it to drive car. Second, physical reasoning appears to be one of the fundamental capabilities (along with e.g., geometry, space, statistical intuition, etc.) infants use to bootstrap their growth into general intelligence. Either this reasoning is essential to producing artificial general intelligence (AGT), in which case INFLEVEL will allow us to track models' progress towards allaining this foundation, or INFLEVEL will help us illuminate that AGI does not require such reasoning.
We see several promising approaches toward building models that can reason about physical events in novel domains. First, Ha & Schmidhuber (2018) emphasizes the construction of explicit models of the world
11

---

Published in Transactions on Machine Learning Research (10/2022)
INFLEVEL-LAB
LEVELVEL
JOINTEVAL
Continuity
Solidity
Gravity
Continuity
Solidity
Gravity
Arch.
Train Set
CSN
K400
0.47
0.44
0.49
0.50
0.49
0.50
0.81
0.61
CVRL
K600
0.38'
0.52
0.52
0.50
0.50
0.51
0.97
0.88
ArmPN
0.57*
0.52
0.49
0.50
0.50
0.50
0.42"
0.42"
COIV2GRU
ObjNav
0.48
0.50
0.50
0.50
0.50
0.51
0.42"
0.43*
13D
K400
0.50
0.50
0.51
0.50
0.50
0.51
0.69
0.56
EK100
0.47
0.49
0,50
0.50
0.50
0.50
0,52"
0.36"
MF
K00
0.44
0.52
0.49
0.50
0.49
0.50
0.40*
0.32*
SSv2
0.37
0.47
0.48
0.50
0.49
0.50
0.26*
0.27
Ch
0.58*
0.54
0.50
0.50
0.49
0.51
0.67
0.62
S-50
K400
0.50
0.50
0.49
0.50
0.50
0.51
0.69"
0.58
SSv2
0.53
0.51
0.52
0.51
0.50
0.51
0,55
0.51
SF-101
K400
0.46*
0.55
0.48
0.50
0.50
0.50
0.87
0.58
Ch
0.44
0.50
0.48
0.50
0.50
0.51
0.75*
0.57*
K 400
0.49
0.51
0.51
0.50
0.50
0.51
0.87*
0.57*
SF-50
None
0.46*
0.47
0.52
0.50
0.50
0.49
0.70"
0.63"
SSv2
0.49
0.50
0.48
0.50
0.50
0.50
0.61
0.52
HTTO0M
0.45*
0.48
0.49
0.50
0.50
0.50
0.51
0.78*
TF
K400
0.45*
0.50
0.51
0.50
0.50
0.50
0.53*
0.68*
SSv2
0.58*
0.49
0.51
0.51
0.51
0.49"
0.46"
0.67*
X3D
K400
0.47
0.49
0.51
0.50
0.50
0.50
0.73"
0.58*
Table 2: Results on INFLEVEL and JOINTEVAL. Here we display the aggregated "majority vote" score when evaluating various model architectures (Arch.), training datascts (Train Sct), and dataset splits (Continuity, Solidity, and Gravily for INFLEVEL and Sh/PS for JOINTEVAL). Several models score above chance (> 0.5) on Continuity violations in INFLEVEL- but performance on Solidity and Gravity violations is near-random across the board (= 0.5). High performance of some models on JOINTEvAL shows that these models detect less subtle violations which do not require fine-grained object reasoning. denotes significant difference from 0.5 at a 0.01 level using two-sided permutation test.
(e.g., 3D locations of entities and their relationships); reasoning is then performed using these models rather than "model-froe" directly from pixels. Evidence suggests that learning generalizable reasoning abilities "model-free" is challenging (Razeghi et al., 2022). A second direction is to work toward incorporating explicit knowledge, such as physical laws, into Al systems (Raissi et al., 2017), possibly using neuro-symbolic techniques (Mao et al., 2019; d' Avila Garcez & Lamb, 2020; Marcus, 2020). A third direction might focus on using richer inductive biases encoding space, time, objects, and causality (Baillargoon & DcJong, 2017: Spelke et al., 1992; Pearl & Mackenzie, 2018)-a00 even in advance of learning.
Limitations INFLEVEL is, by design, set in a controlled environment and tests carefully defined components of physical reasoning. Just as infants can become distracted overwhelmed, even model that achieves perfect score on INFLEVEL may struggle to reason in face of diverse, real-world, settings. INFLEVEL provides a valuable diagnostic for physical reasoning and is intended to be used alongside existing benchmarks to provide a holistic perspective on models' fundamental capabilities. As INFLEVEL is evaluation-only, INFLEVEL can only test what a given, frozen, model represents regarding core physical reasoning principles. This means that two models with the same architecture but trained on different data may obtain dramatically different results on INFLEVEL. Thus, when reporting results on INFLEVEL, researchers must take carefully describe their training protocol and avoid attempting to subtly tweak their training dataset to "beat" INFLEVEL; described in Section the system" and obtain high performance but doing so is akin to giving model the answers to test: performance is high without understanding, It is also important to emphasize that INF LEVEL is set in the context of an infant-coguition lab and involves household objects. This context may be very different from those used by models during training, e.g., an autonomous driving model will have very little experience with toy ducks but may understand Continuity perfectly when it involves the roadway environment. As above, this suggests need for broad spectrum of benchmarks.
videos in INFLEVEL
tightly controlled, heuristics can be used to "game
12

---

Published in Transactions on Machine Learning Research (10/2022)
References
A. Aguiar and R. Baillargeon. 2.5-month-old infants' reasoning about when objects should and should not be occluded. Cogn Psychol, 39( 1999. 3, 5
R. Baillargeon. Representing the existence and the location of hidden objects: object permanence in 6- and
8-month-old infants. Cognition, 23(1), 1986. 2, 4
R. Baillargeon. Innate Ideas Revisited: For a Principle of Persistence in Infants' Physical Reasoning. Perspect
Psychol Sei, 3(1):2 13, Jan 2008. 2
Baillargeon and G. F. DeJong. Explanation- learning in infancy. Psychon Bull Rev, 24(5):1511 1526, 10 2017. 12
R Baillargeon and DeVos. Object permanence in young infants: further evidence. Child Dev, 62(6):
1227-1246, December 1991. 6
R. Baillargeon, E. S. Spelke, and S. Wasserman. Object permanence in five-month-old infants. Cognition,
20(3), 1985. 2,
Renée Baillargeon. Object permanence in 3 1/2- and 4 1/2- moo-mooth-oll infants. 23(5), 1987. 4,
Developmental Psychology.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video under- standing? In ICML, 2021. 10
João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset.
In CVPR, 2017. 10
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Yee Kenneth Wong, Joshua B. Tenenbaum, and Chuang Gan. Grounding physical concepts objects and events through dynamic visual reasoning. 9th In- ternational Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nel, 2021. URL https::/openrrvvee. wttp://wwwww..eeeeeeneeeeeeeeeee
cEGCz.
Junyoung Chung, Çaglar Gülgehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, albs/1412.3555, 2014. 25
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. Scaling egocentric vision: The epic-kitchens dataset. In ECCV, 2018. 3,
Arijit Dasgupta, Jiafei Duan, Marcelo H. Ang, Yi Lin, Su-hua Wang, Renée Baillargeon, and Cheston Tan. A benchmark for modeling violation-of-cxeecatioo physical reasoning across event categories, 2021. 4
Artur d'Avila Garcez and Luis C. Lamb. Neurosymbolic Al: the 3rd wave. CoRR,, abs hbs//201..08776 2020. 12
Ernest Davis, Gary Marcus, and Angelica Chen. Reasoning from radically incomplete information: The case of containers. Advances in Cognitive Systems, 2:1-18, 2013. 4
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. Robothor: An open simulation-to-real embodied AI platform. In CVPR, 2020. 10
Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Manipulathor: A framework for visual object manipulation. Tn CVPR, 2021. 3,
8, 10, 20, 23
Haoqi Fan, Tullie Murrell, Heng Wang, Kalyan Vasudev Alwala, Yanghao Li, Yilei Li, Bo Xiong, Nikhila Ravi, Meng Li, Haichuan Yang, Jitendra Malik, Ross Girshick, Matt Feiszli, Aaron Adcock, Wan-Yen Lo, and Christoph Feichtenhofer. PyTorchVideo: deep learning library for video understanding. In ACM Multimedia, 2021. 10
13

---

Published in Transactions on Machine Learning Research (10/2022)
Christoph Feichtenhofer. X3D: expanding architectures for efficient video recognition. In CVPR, 2020. 10
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recogui- tion. In ICCV, 2019. 9, 10
Stanislav Fort, Jie Ren, and Balaji Iakshminarayanan. Exploring the limits of out-of-distribution detection. arXiv, 2021. 9, 21
Lex Fridman and Jitendra Malik. #110 jitendra malik: Computer vision, July 2020. 8
Chuang Gan, Jeremy Schwartz, Seth Altor, Martin Schrimpf, James Tracr, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Damian Mrowca, Michael Lingelbach, Aidan Curtis, Kevin T. Feigelis, Daniel M. Bear, Dan Gutfreund, David D. Cox, James J. DiCarlo, Josh Н. McDermott, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Threedworkd: A platform for interactive multi-modal physical simulation. arXiu, 2020. 4
Kanishk Gandhi, Gala Stojnic, Brenden M. Lake, and Moira R. Dillon. Baby intuitions benchmark (BIB): discerning the goals, preferences, and actions of others. CoRR, 2//22122..19998 2021.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westppall Heuna Kim, Valentin Haenel, Ingo Fründ, Peter N. Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017. 3, 10, 26
Frank E. Grubbs. Procedures for detecting outlying observations in samples. Technometrics, 11(1), 1969. 9
Xiaoyi Gu, Leman Akoglu, and Alessandro Rinaldo. Statistical analysis nearest neighbor methods for anomaly detection. 22
David Ha and Jürgen Schmidhuber. World models. CoRR, abs/1803.1011 2018. 11
Md. Abul Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, and Liming Chen. von mises-fisher mixture model-based deep learning: Application to face verification. CoRR, abs/1706.04264, 2017. 22
Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017. 9, 21
Susan J. Hespos and Renée Baillargeon. Reasoning about containment events in very young infants. tion, 7833 2001. ISSN 0010-0277. 6
Cogni-
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning.
CVPR, 2017. 4
Will Kay, João Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinctics human action video dataset. arXiv, 2017. 3, 9, 10
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: an interactive 3d environment. for visual AT. arXiv, 2019. 3, 5, 8, 10, 20
David Kremelberg, Embodiment necessary priori of general intelligence. In Artificial General Intelli- gence, 2019. 8
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017. 1,
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436 444, 2015.
14

---

Published in Transactions on Machine Learning Research (10/2022)
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Tn Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), NewrIPS, 2018. 9, 21
Chengshu Li, Fei Xia, Roberto Martin-Martin, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, Andrey Kurenkov, C. Karen Liu, Hyowon Gweon, Jiajun Wu, Li Fei-Fei, and Silvio Savarese. igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. In CoRL, 2021. 5, 8
Wei Liang, Yibiao Zhao, Yixin Zhu, and Song-Chun Zhu. What is where: Inferring containment rela- tions from videos. In Subbarao Kambhampati (ed.), Proceedings of the Twenty-Fifth International Joint Conference Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016, pp. 3418-3424. LICAI/AAAI Press, 2016, 4
Wei Liang, Yixin Zhu, and Song-Chun Chun Zhu. Tracking occluded objects and recovering incomplete trajectories by reasoning about containment. relations and human actions. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. 4
Yi Lin, Maayan Stavans, and Renée Baillargeon. Infants' Physical Reasoning and the Cognitive Architecture that Supports It. In Cambridge handbook of cognitive development. 2022. to appear. 2, 4, 5, 6
Y. Luo, L. Kaufman, and R. Baillargeon. Young infants' reasoning about physical events involving inert and self- proppelee objects. Cogn Psychol, 58 4844):4414486 Jun 2009. 2, 7
Jiayuan Mao, Chnang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The neuro-yymbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe-
view.net, 2019. 12
Gary Marcus. The next decade in Al: four steps lowards robust artificial intelligence. CoRR, abs 2b172002.0 2020. 12
K. V. Mardia. Distribution theory for the von mises-fisher distribution and its application. In G. P. Patil, S. Kotz, and J. K. Ord (eds.), A Modern Course on Statistical Distributions in Scientific Work, pp. 113-130, Dordrecht, 1975. Springer Netherlands. 22
Antoine Micch, Dimitri Zhukov, Jean- Bran-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howtol00m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV, 2019. 3, 10
A. Needham and R. Baillargeon. Intuitions about support in 4.5-month-ole infants. Cognition, 47(2), 1993.
2, 3, 7
Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, and Joshua B. Tenenbaum. PHASE: physically- grounded abstract social events for machinc social perception. In Thirty-Fifth AAAI Conference on Ar- lificial Intelligence, AAAI 2021, Thirly-Third Conference on Innovalive Applications of Arlificial Intel- ligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 845 853. AAAI Press, 2021. 4
Mandela Patrick, Dylan Campbell, Yuki Markus Asano, Ishan Misra, Florian Metze, Christoph Feichten- hofer, Andrea Vedaldi, and João F. Henriques. Keeping your eye on the ball: Trajectory attention in video transformers. In NewrIPS, 2021. 10
Judea Pearl and Dana Mackenzie. The Book of Why: The New Science of Cause and Effect.. Basic Books,
Tnc., USA, 1st edition, 2018. 12
Jean Piaget. The origins of intelligence in children. W W Norton & Co, New York, NY, US, 1952. 3
Jean Piaget. The construction of realily in the child. Basic Books, New York, 1954. 3
15

---

Published in Transactions on Machine Learning Research (10/2022)
Luis Piloto, Ari Weinstein, Dhruva TB, Arum Ahuja, Mehdi Mirza, Greg Wayne, David Amos, Chia-Chun Hung, and Matthew Botvinick. Probing physics knowledge using tools from developmental psychology. arXiv, 2018.
Luis Piloto, Ari Weinstein, Peter Battaglia, and Matthew Botvinick. Intuitive physics learning a deep-learning model inspired by developmental psychology. Nature Human Behaviour, Jul 2022. ISSN 2397-3374. 4
Rui Qian, Tianjian Meng, Boqing Gong, Ming Hsuan Yang, Huisheng Wang, Serge J. Belongie, and Yin Cui. Spatiotemporal contrastive video reprosentation learning. In CVPR, 2021. 10
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 11X:::11111100000 2017.
12
Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. CoRR, abs/2202.07206, 2022. 12
Ronan Riochet, Mario Ynocente Castro, Mathicu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. arxiv, 2018. 4, 8
Michael S. Ryoo, A. J. Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Tokenlearner: What can 8 learned tokens do for images and videos? In NewrIPS, 2021. 1
Thomas Serre. Deep learning: The good, the bad, and the ugly. Annual Review of Vision Science 5(1), 2019.
Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin A. Smith, Shari Liu, Dan Gutfreund, Elizabet S. Spelke, Joshua B, Tenenbaum, and Tomer Ullman. AGENT: benchmark for core psychological reasoning, Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 9614-9625, PMLR, 2021. 4
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), ECCV, 2016. 3, 10
Elizabeth S. Spelke, Karen Breinlinger, Janet Macomber, and Kristen Jacobson. Origins of knowledge. Psychologicol Review, 99( 4):605 632, 1992. 2, 12
Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Singh Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojeiech Galuba, Angel Chang. Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra. Habitat 2.0: Training home assistants to rearrange their habitat..
In NewrIPS, 2021. 5, 8
Du Tran, Heng Wang. Matt Feiszli, and Lorenzo Torresani. Vidco classification with channcl-soparated convolutional networks. In ICCV, 2019. 10
S. H. Wang, R. Baillargeon, and S. Paterson. Detecting continuity violations in infancy: new account and new evidence from covering and tube events. Cognition, 95(2):129-173, Mar 2005. 6
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. aree 2021.
Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. Allenact: A framework for embodied AT research. arXiu, 2020. 10, 26
16

---

Published in Transactions on Machine Learning Research (10/2022)
Luca Weihs, Aniruddha Kembhavi, Kiana Ehsani, Sarah M. Pratt, Winson Han, Alvaro Herrasti, Eric Kolve, Dustin Schwenk, Roozbeh Mottaghi, and Ali Farhadi. Learning generalizable visual representations via interactive gameplay. In ICLR, 2021. 8
Lawson L. S. Wong, Leslie Pack Kaelbling, and Tomás Lozano-Pérez. Manipulation-based active search for occluded objects. In 2013 IEEE International Conference Robotics and Automation, Karlsruhe, Germany, May 6-10, 2013, pp.
IEEE, 2013.
Jiajun Wu, Joseph J. Lim, Hongyi Zhang, Joshua B. Tenenbaum, and William T. Freeman. Physics 101: Learning physical object properties from unlabeled videos. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), BMVC, 2016. 4
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual do-animation. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett. (eds.), Advances in Neural Informalion Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL attps://proceedings neurips...... http:/n....n.rp...peppee/e/e//ee cc/paperr/011//tile/ 4
pdf. 4
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In CVPR, 2020. 5
Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-end semi-supervised object detection with soft. teacher. In ICCV, 2021.
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B. Tenenbaum, and Shuran Song. Densephysnet: Learning dense physical object representations via multi-step dynamic interactions. In Antonio Biechi, Hadas Gazis, and Seth Hutchinson (cds.), Robotics: Science and Systems XV, University of Freibury, Freibury im Breisgau, Germany, June 22-20, 2019, 2019. doi: //doi org/oo///// 106111667//RS /RSS.7//SS..0 2019. XV.046.
2019. URL https:
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. In NewrIPS, 2018. 1
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenen- baum. CLEVRER: collision events for vidco representation and reasoning. In ICLR, 2020. 4
Tao Yuan, Hangxin Liu, Lifeng Fan, Zilong Zheng, Tao Gao, Yixin Zhu, and Song- Zhu. Joint inference of states, robot knowledge, and human (false-)beliefss. In 2020 IEEE International Conference on Robotics and Automation, ICRA 2020, Paris, France, May 31 August 31, 2020, pp. 5978. IEEE, 2020. 4
Alan L. Yuille and Chenxi Liu. Deep nets: What have they ever done for vision? Int. J. Compul. Vis., 129 (3):781-800 802, 2021.
Xiaohna Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer Scaling vision transformers. arXiv, 2021.
17

---

Published in Transactions on Machine Learning Research (10/2022)
Appendices for Benchmarking Progress to Infant-Level Physical Reasoning in Al
Our appendices follow below and are organized into the following sections:
A Additional details regarding our INFLEVEL-LAD dataset.
B Additional details regarding our INFLEVEL-SIM dataset.
C Descriptions of the OOD metrics have used proxies for surprise during evaluation.
D A discussion of why negating OOD metric values is not recommended.
E Additional results and discussion on our evaluation procedure and when varying the query set used
during OOD metric computation.
F Further details on the baseline embodicd-AT models we evaluated.
G Descriptions of assets used this work (with their associated licenses) and miscellaneous information for INFLEVEL.
A INFLEVEL-LAB Details
camera angles
Diverse objects
Plausible outcome
Implausible outcome
Figure A.1: Continuity trial. One Continuity trial from INFLEVEL-LAB (familiarization phases not shown). As illustrated in the first and third columns, achieve diversity by varying camera angles and using many distinct covers and objects.
Here provide additional general information about INFLEVEI-LAB videos well as specific information regarding the Continuity, Solidity, and Gravity trials.
A.1 General information
As noted in the main paper, all videos were filmed from three camera angles (right, center, and left) 29.97 FPS in 1280 720 resolution. As is common in infant studies, metronome was used that timing across trials was approximately consistent. Primary and secondary objects were sourced from amazon. Fig. 5a for a listing of all secondary object types used during filming. See Fig. A.1 for an example of some of the ways in which we include variety across our vidcos. Several object types correspond to perishable items (e.g. pear, apple, corn, elc.), for these items we used non-perishable imitations. We used a single human operator across all vidcos (an author on this paper). Filming was conducted across multiple days and, introduce some additional variability into the videos, the operator wore different clothing across days. As trials were filmed over several days and, by their design, have differing image statistics (e.g. implausible trials might always show secondary object while plausible trials only show an secondary object half the time), we apply small video augmentations (jitters of crops, brightness, contrast, saturation, and hue) to all videos.
see
We film primary and secondary familiarization vidoos only once and reuse these vidcos across all relevant trials. For instance, the Continuity videos involving a pepper shaker as the secondary object and blue cup the primary object will have the same two familiarization videos as their first. two "parts" and will have
18

---

Published in Transactions on Machine Learning Research (10/2022)
unique final third part (recall Section 3.1 for description of these parts). As described in the main paper, we concatenate these parts together with short cuts (0.58). This reuse of familiarization videos has two advantages. First, reusing familiarization videos results in additional data and annotator efficiency. Second, and more importantly, as we use precisely the same habituation videos for the plausible and implausible trials, we remove any polential systematic bias how the operator handles objects before these trials. We took these shortcuts correspond similarly to when, infant trials, a curtain is drawn between the infants and the experimental apparatus.
A.2 Continuity Data Collection
Recall that we provide two separate physically implausible Continuity trials (see Sec. 3.1). To film these apparently impossible events without advanced visual effects, we employ the following two strategics.
VI trial- Recall that, in this trial, the operator covers a secondary object with a primary object, slides the primary object across the table, and then lifts the primary object to reveal that the secondary object has disappeared.
To film this trial in practice we discrctely attach magnets to the primary and secondary objects. When the primary object is lowered over the secondary object, the magnets engage, and thus, when the primary object is later lifted, the secondary object moves with it and seemss to disappear.
IV trial The operator lowers the primary object onto the table with nothing inside of it, slides the primary object across the table, and then lifts the primary object to reveal that the secondary object has suddenly appeared.
Filming this trial appears easy initially: can simply use magnets above but "in reverse," " simply have the object attached by magnets initially and then disengage the magnets when lifting the primary object so that the secondary object seems to magically appear. In practice, we found this approach challenging: disengaging the magnet resulted in using hand motions which made the IV trials systematically different than the other trials.
Instead, used another approach: note that VI trial played in reverse is an IV trial but with the event moving in the opposite direction (i.e., if the operator slid the primary object from left to right, then the reversed video shows an IV trial moving from right to left). So that we did not have a systematic difference between the directions in which the IV and VI trials occur, we filmed two VI trials for each object pair, one in each direction. By reversing these trials, we had a corresponding set of IV trials.
Note that reversing VV and II trials results in new, physically plausible, VV and II trials that, above, go in the opposite direction (e.g. slide from right-to-left if they were originally filmed left-to-right). To match the fact that we include physically implausible videos in both directions, we included these reversed plausible videos in our trials as well.
Now, for every primary/secondary object pair, our videos include 3 camera angles of VV, II, IV, and VI events filmed with the primary object moving left-to-right and with moving right-to-loft. When we compute metrics, we always condition on the object pair well as camera angle and direction of movement that surprise score comparisons are fair.
A.3 Solidity Data Collection
Unlike for the Continuity events above, creating physically implansible vidcos for our Solidity trials is sig- nificantly easier. For every object pair, we film two events, one in which we use the clipped cover (so the secondary object doesn't move with the primary object) and one in which we use the unclipped cover (where the secondary object does move with the primary object). As the unclipped and clipped covers are visually indistinguishable during these trials (the clipped portion, if any, is not visible to the camera) we simply match each of the above two videos with the two possible primary object familiarization videos, i.c. one in
19

---

Published in Transactions on Machine Learning Research (10/2022)
which the clipped cover is shown and one in which the unclipped cover is shown. In this way, generate four videos (2 familiarizations x 2 events) to generate the four (UT, UV, CI, and CV) trials.
For diversity, the direction (left-to-right or right-to-left) that the primary object is moved is varied across videos. Unlike for the Contimuity events, however, we do not film left-to-right and right-to-left videos for each object pair.
A.4 Gravity Data Collection
Our Gravity trials are filmed in a fashion similar to the Solidity trials. As clipped and unclipped containers are, effectively, indistinguishable outside of the familiarization period, we obtain the four trials (UI, UV, CL, and CV) by pairing familiarizations with events (2 familiarizations events) concatenated with short cuts.
B INFLEVEL-SIM Details
Here we provide additional information regarding our simulated INFLEVEL-SIM dataset.
As discussed in the main paper, INFLEVEL-SIM was collected within the A12-THOR environment (Kolve et al., 2019) using the robotic ManipulaTHOR agent (Rhsani et. al., 2021). Just as for INFLEVRL-LAB, we captured videos at 29.97 FPS in 1280x720 resolution from three camera angles. As INFLEVEL-SIM is filmed in simulation we used the fast approximate anti-aliasing algorithm (FXAA) available in the Unity game engine to smooth pixel-based artifacts. Recall Fig. 2 for example of four frames from Gravity trial INFLEVEI-SIM. In order to be able to capture our videos of interest, we had to make several changes to AT2- THOR and the ManipulaTHOR agent. In particular, we added actions allowing for additional freedom in arm movement (e.g., rotating the "wrist" and "elbow" of the agent's 's arm), added a second arm to the agent for use in our Gravity trials (where two objects must be manipulated by the operator simultancously), and wrote scripts to programmatically "clip" the various AI2RTHOR assets (e.g. removing the slides or bottoms of cups for the Solidily and Gravity trials). Note that, unlike in App. A, INFLEVEL-SIM being recorded in simulated environment means that simply need to change how physics behaves in this environment in order to produce physically implausible videos.
A12-THOR provides a large collection of object instances from a diverse set of object categories. Figure 5b shows the frequency with which objects various instances were used across our trials. Moreover, the use of a simulated environment makes casy to vary the visual aspects of the room in which our vidcos are set, increasing visual diversity in the dataset. In particular, we randomize (from a curated set of possibilities) the table color/texture (upon which the trial is conducted), background wall color color/textun texturre, and the lighting (see Fig. B.2). Note that these augmentations object pair will have the same randomizations.
done in a consistent manner so that all videos using the
same
While we have collected INFLRVEL-SIM as a fixed collection of RGB videos, A12-THOR allows for the easy generation of additional rich data (e.g. instance segmentations, depth maps, normal maps, etc.). As we will make our data generation scripts publicly available, it will be easy to build upon INFLEVEL-SIM add any such information as needed. Moreover, we suspect that future models, especially those that are embodied, will perform better when they are allowed to direct their gaze around their environment rather than being forced lo view fixed perspective. This is easy to accomplish in A12-THOR, and future models that would benefit from such directed gaze will be able to do so in INFLEVEL-SIM.
C OOD Metrics
As discussed in the main paper, we consider four metrics for out-of-domain (OOD) detection. We describe these metrics below assuming that we are given a model fe (e.g. a video-understandiin model) which takes vidcos v as input. Morcover, we assume that there some query set of vidcos Q (ug...., ..) of "in-domain" videos that can be used to compute these metries (when required).
20

---

Published in Transactions on Machine Learning Research (10/2022)
Figure B.2: Visual diversity in INFLEVEL-SIM The first frame of 8 randomly selected videos, using the center camera. position, in the INFLEVEL-SIM Continuity dataset.
Max-softmax (Hendrycks & Gimpel, 2017). The max-softmax (MSM) mcasure, generally used for clas- sification models, can be applied to any model whose output a collection of probabilities fo(u) (po(e)..... (u)) which sum one (E P(())( 1). The max-softmax measure for such a model and video then equals
Max-Softmax(fo. = max (po(
The intuition behind this measure simple: if a video v is in-domain for the model fe, then the model should be confident about its predictions and thus the probabilities (ı()) should be peaked, i.e. there should be some (v) near one. Alternatively, if model is given an OOD video v, then should be less confident about its predictions and the (v(v) should be near uniform. This simple idea has been shown to be quite effective when paired with deep networks (Fort et al., 2021).
Mahalanobis (Lee ct al., 2018). Assume now that fo outputs real-valued d-dimensional vector representations of input videos v. Additionally, assume that we can have some method for assigning every video v E Q to a single class (or cluster) C (co...., ce) so that cl(v) E C (e.g-, in action recognition, this might be the video's category; if categories don't exist, we can instead use clustering method such as k-means). The Mahalanobis (MH) OOD measure for model fo and video is then computed
MH(Ja, v) = max ((f((v) A5) ( (..)
cEC
where, for every e E C,
4. mcan I vs EQ and cl(v;) c} and
② 1
Σ (fo(v:) (f(()) Me((..))
Q 3
To understand this intuitively, note that füe is a flass-condiiinnll mean and is a dxd covariance matrix. Thus, up to an additive constant that's fixed across all c, the value
((f(()) pc) ( Ac)
equals the estimated, c-conditional, log-pro probbabbyilt of fe(u) when modeling this class-conditional distribution as multivariate Gaussian with an unknown mean and covariance (this unknown covariance shared across all c). Hence MH(Ja, v) is the max of all these log-probabilities and measures the maximum likelihood of v across all classes c. With this perspective, the MH metric is quite sensible: videos are considered in-domain when they have a large estimated class-conditional likelihood, and they are considerod more out-of-domain when these consiconn likelihoods are all small.
21

---

Published in Transactions on Machine Learning Research (10/2022)
von Mises-Fisher While the MH mcasure above models the c-conditional distributions Gaussians, there is no reason, priori, to assume that the Gaussian distribution is a good fit for this data. We instead propose to model these conditional distributions using the von Mises-Fisher (VMF) distribution, a classical distribution for modeling points on the unit-sphere in high-dimensional spaces (Mardia, 1975). Although we are far from the first to use the VMF distribution in the context of deep-learning (Hasnat al., 2017), we are, to the best of our knowledge, the first to use the VMF distribution for OOD detection. Our motivation for using the VMF distribution is simple. Many video-understanding models use single linear layer to map from their representation of video to their output (e.g., a set of logits used for classification). This linear layer can be interpreted as performing a collection of dot-products between learned vectors and the representation. Up to scaling, this is equivalent to measuring the (cosine of the) angle between the learned vectors and the representation. The VMF distribution uses precisely this type of angle-based reasoning to compute likeli- hood values, suggesting it may be a more appropriate fit than the standard Gaussian which uses L2 distances.
Nearest-nniighbor-bess metric (NN (L2)). The use of l nearest-neighhors (kNN) for outlier detection is well-studied (Gu et al.). The idea is quite intuitive: video is considered in-domain if it is close to (using some mcasure of distance D) other vidcos in the query set. We compute our NN-based OOD metrics as:
NNn(Ja, min D(v,
uEEQ
where we take D to be the usual L2 distance. Many different candidate metrics can be formed by taking D to be different measures of similarity, e.g. the cosine similarity.
Negating Surprise Scores
As mentioned in the main paper, several models attain performance significantly worse than random upon the Continuity subset of INPLEVEI-LAB, recall Table 2. For instance, the SSy2-pretrainee Motionformerr model obtains score of 0.32 using the NN (CS) measure. It is quite tempting to negate the surprise scores associated with these models, indeed doing would result in that Motionformer model achieving score of 0.68, value 10 percentage points larger than the current best score. We recommend against this approach for two reasons:
1. Tack of interprelability We have used OOD measures as proxies for surprise as they have been well-validated in past work and can be casily interpreted. Indeed, model that has the capacity for physical reasoning and has trained with physically plausible data should, intuitively, see physically implausible events as out-of-domain. Negated variants of OOD measures have no such interpretation: models should not be rewarded for viewing physically implausible events as more in-domain than physically plausible ones.
2. Overfilling- Fair evaluation on INFLEVEL requires that practitioners do not fit. their surprise measures to INFLEVEL. Choosing or modifying surprise measures after observing model's performance on LEVELVEL is form of training on the test set. Suppose, for instance, that we define a surprise measure as >T fo()) where y E Rd is some vector, So is a video-understanddin model that outpuls d-dimensional representations, and v is an input video. Then it is easy to see that, by varying and evaluating the model on INFL.VEL, we can perform zero-order optimization to find an optimal 7 maximizing the score. This optimization procedure is, effectively, equivalent to performing logistic regression to discriminate between physically plausible and implausible videos; we describe, in Section 3.4, why this should not be allowed even when using separate training and testing splits.
E
Additional Results and Discussion
E.1 Why Shouldn't Linear Decoders Be Used with INFLEVEL?
Recall the linear-probe evaluation procedure proposed in Section 3.4. Formally, we assume that we are given trained model fo with parameters 0 that. maps RGB video of dimensions H x W and length T' to
22

---

Published in Transactions on Machine Learning Research (10/2022)
d-dimensional vector representation. A common strategy used to measure the quality of representations produced by such model So is to freese the parameters 0 and train a decoder network Jn so that the composed network 9, o fe can accomplish some auxiliary task. This evaluation assumess that representations are "good" if they can be used as the foundation for accomplishing many relovant tasks. As noted in the main paper, this evaluation paradigm captures a common use-case for representation learning methods: transfer learning in low-sample regimes. Unfortunately, as the following example shows, using a lincar probe to study physical reasoning can result in conflating physical understanding with other related quantities.
Example (Conflating features with physical understanding). Consider two functions S,X [ -R where S is perfect physical-reasoning model (i.e. S(v) equals 1 if and only if v is a physically implausible video and S(v) = 0 otherwise) and X is a "secondary object tracker", i.e. X(v)= 1 if it sees no secondary objects or if it sees the same secondary object at the start and end of v (otherwise equals 0). Now, in our Continuity dataset, training a linear decoder to discriminate plausible and implausible videos will result in high accuracy using either fo = or fo X despite the fact that only S has the desired physical understanding. This does not mean that linear evaluation on INFLEVEL is meaningless, the X function described above is a kind of phesical-reasoning precursor: identifying and tracking objects is necessary skill for physical reasoning. Identifying this, however, is not our primary interest.
While this example is somewhat artificial, it demonstrates that allowing training on INFLEVEL can have surprising and unwanted side effects, For this reason, INFLEVEL is an evaluation-onf, benchmark and we introduce, see Section 3.4, an out-of-domain evaluation procedure that does not require explicit training on INFLEVEL.
E.2 There is No Universally Applicable Surprise Proxy
We will contimue to use the notation introduced in App. E.1. In practice, there is no single surprise proxy which can be used to detect whether or not any given (black box) model contains a well-hidden understanding of core physical reasoning principles. To see this, consider the following pathological example. Suppose that a model that takes a video and then: (1) computes a hash of the video as (v), (2) computes the true 'plausible or not" boolean S(u), and then (3) finally encrypts the concatenation ((v)) using some private key to produce the encrypted representation E(v). Clearly, the model in this pathological example has an understanding of physical plausibility but, unless the surprise proxy is chosen with special knowledge of the private key, this fixed proxy will not be able to detect this from E'(v). This suggests that every surprise metric will make some compromises and will not be well suited to all potential models.
E.3 Per-OOD-Metric Results
Table 2 from the main paper shows results using the "majority vote" score, which aggregates together the three surprise metrics discussed in Section 4.1. See Tables E.1 and E.3 for fine-grained results for each individual surprise metric for INFLEVEL-LAD and INFLEVEL-SIM respectively.
E.4 Other Query Sets
As described in the main paper, and referenced in App. C, many of our OOD metrics make reference to query-set Q of in-domain videas. To better compare models, our main paper shows results when using a fixed query set of s=25k videos from multiple data sources: approximately 5k videos from the training sets from the Charades (Ch), How (HT100M), Kinetics400 (K400), Something Shiiinnnttttiinggggggggg.. v2 (SSv2), and EpicKitchens100 (EK100) datasets. We will call this query dataset the joint query set.
We also consider pairing each model with a query datasot consisting only of vidcos from its training datasot, e.g., model trained on Kinetics400 would have query set Q consisting only of videos from the Kinetics400 training set. We call these query sets the train query sets." The performance of models on INFLEVEL-SIN.
For embodicd-AI model (trained for the ObjectNav and Naw tasks in A12-THOR) there is no static datasct of videos used during training: these modcls are trained interactivcly. For this reason, we collect static datasct of approximately 5k videos in A12-THOR in which we record the pretrained ArmPointNav model from Ehsani al. (Ehsani al., 2021) from thiro-person perspective as it, in training scenes, completes the Par task. We use this collection as our train query
23

---

Published in Transactions on Machine Learning Research (10/2022)
Event Cat.
Continuity NN (L2)
Solidity NN (L2)
Gravity
Metric Train Set
MSM
VMF
MSM
VMF
MSM
NN (L2)
VMF
Arch.
CSN
K400
0.47
0.53
0.42*
0.43*
0.48
0.48
0.52
0.46*
0.50
CVRL
K600
0.36"
0.41
0.32
0.53
0.51
0.53
ArmPN
0.58
0.56*
0.50
0.53
0.49
0.49
CoRV2GRU
ObjNav
0.49
0.47*
0.49
0.50
0.51
0.49
13D
K400
0.49
0.52
0.47*
0.49
0.49
0.52
0.52
0.50
0.50
10000
0.53
0.48
0.46*
0.52
0.51
0.48
0.53
0.50
0.49
MF
K400
0.47
0.46*
0.45*
0,50
0.30
0.50
0.50
0.49
0.49
SSv2
0.55*
0.38*
0.40"
0.51
0.50
0.44"
0.50
0.49
0.48
Ch
0.49
0.56
0.60
0.52
0.32
0.52
0.50
0.50
0.49
S-50
K400
0.46*
0.53
0.51
0.52
0.47
0.49
0.52
0.49
0.49
SSv2
0.49
0.58*
0.49
0,52
0.49
0.51
0.49
0.52
0.52
SF-101
K400
0.50
0.48
0.45*
0,52
0.53
0.55
0.47
0.52
0.49
Ch
0.46*
0.12*
0.45*
0.49
0.49
0.51
0.50
0.48
0.49
K400
0.45*
0.49
0.54
0.51
0.50
0.50
0.51
0.52
0.49
SF-50
None
0.46*
0.52
0.47*
0.49
0.46
0.48
0.47
0.52
0.51
SSv2
0.43*
0.55*
0.53
0.53
0.51
0.49
0.50
0.50
0.47
HT
0.50
0.47
0.44"
0,52
0.50
0.47
0,49
0.50
0.51
TF
K400
0.51
0.45*
0.44"
0.52
0.49
0.51
0,50
0.51
0.51
SSv2
0.55*
0.52
0.59*
0.50
0.48
0.51
0.51
0.50
0.50
X3D
K400
0.48
0.46"
0.51
0.49
0.32
0.49
0.51
0.52
0.49
Table F.1: Evaluation results on INFLEVEI-LAB using the joint query set. Compare with F.2.
when using the train query sets is shown in Table E.2. The results are quite similar to those obtained when using the joint query set.
E.5 Additional Discussion of INFLEVEL-SIM Results
Results of evaluating our various baseline models on INFLEVEL-SIM can be found in Table 2 in the main paper; fine-grained per-surprise-score results are also shown in Tables E.3 (using the joint query set) and Table E.4 (using train query sets). As noted in the main paper, there are main trends. Firstly, unlike in INFLEVEI-LAB, no models perform significantly above chance at recognizing Continuity violations. This is largely unsurprising for the video-underse understanding models trained real videos, as the domain gap between simulated and real videos is quite large. The ObjectNav and ArmPointNav models are, however, trained in simulation and so one might expect these models to do well in this setting. Secondly, several models at attain very small (maximally 0.51 with MV and 0.53 with VMF metrics) but statistically significant performances above random chance at detecting Gravity violations. As these values are so small, we hesitate to suggest that this means these models have any understanding of Gravity and are simply latching onto some statistical correlation present in our data collection methodology.
E.6 Exploratory Analysis of INFLEVEL-LAB Results
Table E.5 shows an exhaustive breakdown of how frequently each model considers each trial type more surprising than another trial type in INFLEVEL-LAB (using the joint query and the majority score). This fine-grained breakdown allows us to further analyze the main results presented in Table 2, to better understand the poor performance of models and, in particular, what features of these trials may contribute to model surprise. As these analyses are being done post-hoc, e.g. we have already seen model performance from Table E.5 and are now generating intuitively plausible hypotheses by examining the data, this analysis should be seen as exploratory. One such possible hypothesis is that these models simply judge some configurations
for the embodied models. We will make this dataset open-source and publicly available along with the other components of INFLeVRI.
24

---

Published in Transactions on Machine Learning Research (10/2022)
Event Cat.
Continuity NN (L2)
Solidity NN (L2)
Gravity NN (L2)
Metric
MSM
VMF
MSM
VMF
MSM
VMF
Arch.
Train Set
CSN
K400
0.47
0.52
0.42
0.43*
0.47
0.49
0.52
0.49
0.51
CVRL
K600
0.42
0.35*
0.53
0.52
0.51
0.53
ArmPN
0.56"
0.49
0.51
0.51
0.49
0.49
CoRV2GRU
ObjNav
0.50
0.47*
0.49
0.50
0.52
0.49
I3D
K400
0.49
0.52
0.43*
0.49
0.49
0.51
0.52
0.50
0.49
EK100
0.53
0.47
0.47
0.52
0.51
0.52
0.53
0.49
0.49
MF
K400
0.47
0.45*
0.41*
0.50
0.49
0.49
0.50
0.49
0.50
SSv2
0.55*
0.40
0.35*
0.51
0.52
0.49
0.50
0.48
0.50
Ch
0.49
0.52
0.59
0.52
0.50
0.52
0.50
0.50
0.49
S-50
K400
0.46*
0.54
0.47
0.52
0.48
0.49
0.52
0.50
0.50
SSv2
0.49
0.57*
0.54"
0.52
0.50
0.53
0.49
0.52
0.53*
SF-101
K400
0.50
0.49
0.48
0.52
0.53
0.53
0.47
0.52
0.50
Ch
0.46*
0.47
0.43*
0.49
0.51
0.50
0.50
0.48
0.48
K400
0.45*
0.48
0.42"
0.51
0.51
0.52
0.51
0.50
0.47
SF-50
None
0.46*
0.51
0.45*
0.49
0.52
0.50
0.47
0.54
0.49
SSv2
0.43*
0.52
0.46*
0.53
0.48
0.50
0.50
0.50
0.50
HT100M
0.50
0.46"
0.47
0.52
0.50
0.51
0.49
0.51
0.48
TF
K400
0.51
0.53
0.40*
0.52
0.51
0.51
0.50
0.50
0.50
SSv2
0.55*
0.46"
0.62
0.50
0.52
0.52
0.51
0.49
0.50
X3D
K400
0.48
0.40"
0.49
0.49
0.52
0.49
0.51
0.50
0.50
Table E.2: Evaluation results on INFLEVEL-LAB using train query sets. to those using the joint query set from Table E.1.
Results are quite similar
of objects to be surprising without considcration the physical plausibility these configurations. Table E.6, we show the subset trial comparisons where we compare only those trials that start and end with both the primary and secondary objects visible (S2E2 trials) against those that start with both objects visible but end with only visible (S2E1 trials). There are 9 such trial pairs.
Surprisingly, some models are remarkable consistent in finding the S2E2 trials less (or more) surprising than the S2E1 trials on average. For instance, the TF model trained on SSv2 finds the S2E2 trials less surprising than the S2E1 trial across all trial pairs; similar results hold across of 9 pairs for the CVRL and MF models. In contrast, the SF-50 model trained on SSv2 shows opposite results and finds the S2E2 trials more surprising than the S2F1 trials across all 9 pairs. This consistency remarkable and emphasizes that these models are able systematically represent differences between INFLEVEL-LAS trials. However, up to our ability to detect, they do not do in way that mimics infants' surprise at physical implausibility.
F Embodied Models
As noted in the main paper, except for the embodied Conv2GRU models trained for the ObjectNav and ArmPointNav tasks, all baseline models have evaluated on INFLEVEL come pretrained from external sources. While pretrained models exist for both the ObjectNav? and ArmPointNay tasks, we found these baseline models quite challenging to use with INFLEVEL as they were not designed with video-understandinng in mind. In particular, the pretrained embodied models have architectures in which, al every timestep, goal information (e.g. "you are looking for a cup"), previous action information (e.g. "your last action was to move forward"), and egocentric visual inputs (e.g. RGB images and depth maps), are each embodded and then passed through an RNN (namely 1-layer GRU (Chung al., 2014)) to produce a representation of the environment. Naively attempting to use these models in INFLEVEL presents several challenges: (1)
inne
25

---

Published in Transactions on Machine Learning Research (10/2022)
Event Cat.
Continuity NN (1.2)
Solidity NN (1.2)
Gravity
Metric
MSM
VMF
MSM
VMF
MSM
NN (1.2)
VMF
Arch.
Train Set
CSN
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.49
0.50
CVRL
K600
0.50
0.50
0.50
0.50
0.51
0.51
ArmPN
0.50
0.50
0.50
0.50
0.50
0.51
Conv2GRU
ObjNav
0.50
0.50
0.50
0.50
0.51
0.51
13D
K400
0.50
0.50
0.50
0.51
0.50
0.51
0.51
0.48"
0.52*
EK100
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
MF
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
SSv2
0.50
0.50
0.49
0.50
0.49
0,50
0.50
0.50
0,50
Ch
0.50
0.50
0.50
0.50
0.50
0.49
0.50
0.51
0.52*
S-50
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.49*
0.52*
SSv2
0.51
0.50
0.50
0.49
0.51
0.50
0.50
0.49
0.53*
SF-101
K400
0.50
0.50
0.50
0.51
0.50
0.50
0.50
0.49
0.50
Ch
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.51
0.51
K400
0.50
0.50
0,50
0.50
0.50
0,50
0.51
0.48*
0.52*
SF-50
None
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.49
SSv2
0.50
0.50
0,50
0.50
0.50
0,50
0.50
0.47*
0.53*
HTT00M
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
TF
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
SSv2
0.51
0.51
0.50
0.50
0.51
0.50
0.49
0.49*
0.50
X3D
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.49
0.50
Table E.3: Evaluation results INFLEVEL-SIM using the joint query set. Compare with E.4.
there is no obvious "goal" in INFLEVEL give to these modcls, (2) it is the operator who takes actions INFLEVEL and not the agent hence there is no last action to embed, and (3) while A12-THOR can provide ground truth depth maps, INFLEVEL is primarily designed to test models' ability to understand RGB videos without ideal geometric information.
For the above reasons, we have modified the baseline architectures provided for the ObjectNav and Arm- PointNav tasks and retrained these models using the AllenAct (Weihs et al., 2020) framework. Our new architectures use only egocentric RGB images and integrate goal / last action information only after the embedding of the RGB image has been passed through the GRU. In this way, the GRU acts as passive, goal-independent, memory of prior observations, and the representation coming from this GRU can be used for our evaluations. For further details please see code base, our pretrained models will be made publicly available.
G
Asset Licenses and Other Details
G.1 Asset Licenses
We have used multiple assets in this work, list relevant licenses below.
Charades 9 Licensed for non-commercial use.
MowTo100M dataset Apache License 2.0.
Kinetics400 dataset || Creative Commons Attribution 4.0 International License.
Something Something v2 dataset (Goyal et al., 2017) Proprietary dataset the Twenty Billion Neurons company with non-commercial academic licenses available on request.
Ohttps://prior alwemai.
10nttps://www.ck http://www......t..o////eeacaeaa
unttpss///aeppindd http:///awpaaaaae////////////oaa http://h///pppoooooorrrcieeeteie
26

---

Published in Transactions on Machine Learning Research (10/2022)
Event Cat.
Continuity NN (1.2)
Solidity NN (1.2)
Gravity
Metric
MSM
VMF
MSM
VMF
MSM
NN (1.2)
VMF
Arch.
Train Set
CSN
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
CVRL
K600
0.50
0.50
0.50
0.50
0.51
0.51
ArmPN
0.50
0.50
0.50
0.49
0.50
0.50
Conv2GRU
ObjNav
0.50
0.50
0.50
0.50
0.51
0.52*
13D
K400
0.50
0.50
0.50
0.51
0.50
0.50
0.51
0.48"
0.53*
EK100
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
MF
K400
0.50
0.50
0.50
0.50
0.49
0.50
0.50
0.50
0.50
SSv2
0.50
0.50
0.50
0.50
0.50
0,50
0.50
0.50
0,50
Ch
0.50
0.50
0.50
0.50
0.50
0.49
0.50
0.50
0.51
S-50
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.49*
0.52*
SSv2
0.51
0.50
0.50
0.49
0.50
0.50
0.50
0.50
0,53*
SF-101
K400
0.50
0.50
0.50
0.51
0.50
0.50
0.50
0.50
0.52*
Ch
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.51
0.50
K400
0.50
0.50
0,50
0.50
0.50
0.49
0.51
0.48*
0.51
SF-50
None
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.49
SSv2
0.50
0.50
0,50
0.50
0.50
0,50
0.50
0.48*
0.52*
HTT00M
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.51
TF
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.49
SSv2
0.51
0.51
0.51
0.50
0.49
0.51
0.49
0.51
0.50
X3D
K400
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.50
0.51
Table E.4: Evaluation results on INFLEVEL-SIM using train query sets. Compare with E.3.
Solidity
UB-UV
Raal/Maage
M-M
B:M
OSN
0.51
0.55
0:42
0::1
0:42
0.98
0.48
0..00
CVEL
Ro:0
0.27
0.32
0.31
0.52
0.455
01350
0.24
10:29
00:1
0:45
0.01
0.27
0.38
0.022
01.30)
0.33
0.555
0.41
190
R300
0.64
0.54
0.68
0.49
0.46
0.46
0.40
0.30
0.00
0.47
0.54
0.42
0.58
0.50
0.54
0.61
0.40
0.52
EK100
0.:35
0.32
0.32
0.50
0.45
0.43
0.50
0.80
0.51
MF
0.45
23RR
DlgNNey
ArmPN
0.90
0.51
0.35
0.55
0.46
0:38
0.5c
0.95
0.59
0.43
0.42
0.59
C5
00:0
0357
0.39
0.55
11.33
11:04
01:00
11.05
00:1
0.35
0.32
01:41
03.00
0.51
0..33
0.33
0.79
0.:33
$:50
Koo0
0.45
0.98
0.339
0.51
0.53
01312
0.306
0.61
0.54
0.45
0.53
0:39
0.39
0.45
01.28
0.30
0.39
11:4
58v2
0.48
0.01
0.53
0.57
0.47
0.54
0.55
0.35
0.55
0.32
0.47
0.00
0.61
0,57
0.34
0,50
6,57
01.6
SP
0.57
0.61
0.53
0.51
0:47
0:48
0.36
0:31
0.52
01009
0.39
0.54
None
0.43
0.41
0.45
0.49
0.49
0.51
0.46
0.44
0.45
0.32
0.44
0.47
0.46
0.52
0.47
0.51
0.53
0.50
5SV2
0.54
0.57
0.43
0.48
0.62
0.61
0.32
0.60
0.63
0.44
0.52
0.26
0.94
0.54
0.35
0.52
0.50
0.46
TP
HT100M
0.40
0.42
0.42
0.49
0.49
0.30
0.51
0.45
0.30
0.43
0.55
0.63
0.19
0.00
0.32
0.21
6.78
0.38
K000
0.12
0.30
0.45
0.31
0.50
0.00
0.91
0.:3
0.99
Table E.5: Evaluation results on INFLEVEL-LAD using the joint query set showing all possible comparisons between trials. Here we show the frequency with which (using the majority score) each model considers one trial type more surprising than another. 6.g., in the first column (Contimuity, II-IV, R-M) the first value of 0.47 notes that the CSN model considered the II Continuity trial to be more surprising than the corresponding IV trial in 47% of cases. Here R-M, for ease of reference, is used to denote that the first trial type (II) is physically plausible (i.e., Real) whereas the second trial type (IV) is physically implausible (i.e., Magic).
EpicKitchens100 dataset Creative Commons Attribation 4.0 International License. A12-THOR si Apache License 2.0.
iPhttp:://epie--ttcheeee www.sttcheeenithhheeeee github. 1o/2021
/ htctchuub.omm//////ee con/allen www/allenai/aaaeeecccaeaaeaaeeee /ai2thor
27

---

Published in Transactions on Machine Learning Research (10/2022)
Continuity VV-VI
Solidity
Gravity
Total 0.5
UV-CI M-M
CV-CI
CV-UI R-R
UV-UI M-R
UV-CI M-M
CV-CI R-M
CV-UI
UV-UI
R-M
R-M
R-R
M-R
Arch.
Train Set
CSN
K400
0.47
0.45
0.42
0.47
0.55
0.54
0.45
0.48
0.48
2
CVRL
K600
0.50
0.76
0.79
0.64
0.73
0.62
0.62
0.59
13D
K400
0.46
0.60
0.59
0.60
0.58
0.42
0.50
0.60
0.48
5
EK100
0.52
0.47
0.52
0.50
0,50
0.57
0.57
0.32
0.57
7
MF
K400
0.48
0.52
0.48
0.46
0.44
0.51
0.53
0.58
0.55
SSv2
0.44
0.58
0.50
0.55
0.58
0.59
0.55
0.54
0.60
ObjNav
0.45
0.56
0.47
0.64
ConV2GRU
ArmPN
0.55
0.45
0.46
0.38
0.44
0.48
0.43
0.42
0.46
Ch
0.66
0.60
0.65
0.64
0.56
0.50
0.51
0.53
0.51
8
S-50
K400
0.52
0.64
0.61
0.54
0.61
0.41
0.45
0.58
0.46
6
SSv2
0.54
0.45
0.55
0.55
0.50
0.39
0.37
0.34
0.34
3
SF-101
K400
0.49
0.44
0.57
0.60
0.47
0.36
0.46
0.64
0.50
Ch
0.46
0.68
0. 0.63
0.46
0.62
0.45
0.52
0.59
Kç00
0.47
0.52
0.58
0.59
0.54
0.43
0.51
0.58
0.48
SF-50
SSv2
0.48
0.36
0.36
0.41
0.36
0.31
0.33
0.39
0.36
None
0.51
0.54
0.44
0.48
0.53
0.54
0.52
0.47
0.50
6
SSv2
0.61
0.68
0.69
0.65
0.72
0.56
0.54
0.55
0.55
TF
HT100M
0.50
0.49
0.39
0.47
0.81
0.60
0.32
0.62
K400
0.50
0.51
0.45
0.31
0.40
0.55
0.24
0.51
X3D
K400
0.47
0.45
0.44
0.41
0.47
0.55
0.57
0.51
0.54
Table E.6: Subset of Table E.5 showing only those comparisons between trials that start with objects visible and end with v.s. object visible. For example, the Continuity VV trial starts and ends with both the primary and secondary objects visible while the Continuity VI trial starts with both objects visible but ends with the primary longer present. Here, in comparison to Table E.5, we have inverted some scores (new score 1- the old score) that we are always asking if model finds the ends-with-two-oojeer video less surprising than the ends-witthonc-objeet video. We have also added one additional final column that tallies, for each model, the number of times the model obtained a score of >0.5 across the various comparisons. E.g. the TF model trained on SSv2 has all scores >0.5.
Py TorchVideo library and pretrained models Apache License 2.0.
AltenAct framework MIT License.
ManipulaTHOR project 16 MIT License.
Jacinle library MIT License.
Motionformer library and pretrained models Apache License 2.0.
SlowFast library 19 Apache License 2.0.
"Spatiotemporal Contrastive Video Representation Learning" component of Tensorflow library with as- sociated pretrained model weights Apache License 2.0.
PyTorch library2 BSD-style license, see here.
Faiss 22 MIT License.
Timesformer library and pretrained 2o Creative Commons Antributioo-NonCommmnn 4.0 In- ternational Public License.
28

---

Published in Transactions on Machine Learning Research (10/2022)
G.2 Other Details
Our INFLEVEL-SIM video dataset was collected by author on this paper, no other humans were recorded in the creation of any assets produced in this work.
29


Reviews:
[{"review_id": "wPOYFTJp4f", "reviewer": "Reviewer_EhCD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- **S.1** The idea of a task that's trivial to humans but currently unsolvable for machine learning systems is great. \n- **S.2** The writing and structure are great. It's very accessible.\n- **S.3** The supplemental video is long but a great introduction to the proposed dataset.\n- **S.4** The plots and graphics are good and illustrative.\n\n### Weaknesses\n\n- **W.1** Importance. Since this is TMLR, I'm evaluating based on if you correctly specified your contributions (you did) and if this might be useful to _someone_ in the ML community (it probably is). So technically, I have to accept the paper. But what I'm worried about is the complete lack of justification as to why these specific tasks are supposed to be great for assessing physical reasoning. You listed 6 other physical-reasoning benchmarks at the end of your related works section and you didn't really mention why continuity, gravity, and solidity are more important than what the others are doing. There's also the work \"A roadmap for cognitive development in humanoid robots\" by Vernon et al that details a whole range of other capabilities that are also necessary to develop (or test for) human infant-level core knowledge.\n- **W.2** Possible future dataset/code release. I generally dislike the statement that the authors will release the dataset/code **after** the paper has been accepted. If your core contribution is a dataset, then release the dataset anonymously (e.g. as link to google drive folder), so the reviewers can evaluate your work. \n- **W.3** InfLevel-Sim non-consequitur. You start section 3.3. by saying that there's ample research showing that embodiment and interaction are needed to learn \"good\" representations and then you go on to introduce a dataset that has none of that. (Also n my humble opinion you missed out on citing Gibson here). Since it's a simulator, what's preventing you from providing an interface to researchers where the simulated agent can poke the test objects on the table or pick them up themselves or even carry out experiments themselves? Just flip the camera to be in the robot's perspective, start with the objects in hand and give the agent the opportunity to move the arm along the x/y/z axes and release either object. Even if this isn't perfect and wouldn't work in all your settings, this would conform to your point that embodiment and interaction lead to a better understanding of a scene.\n- **W.4** Choice of objects. See Fig.5 - I would like to call into question that most video models have seen credit cards and soap bars being dropped through funnels. Why did you choose this object frequency? Is this matched to some other (real-world) dataset or is this just based on what was available in AI2-Thor?\n- **W.5** Length of the work. I personally find it taxing to go through 12 pages of work and I think nearly every section (and especially the introduction) could have benefitted from being trimmed by a few sentences to a few paragraphs to make them less verbose. I mentioned that the writing is accessible but I think it could be 10 pages and still very clear and enjoyable to read.\n\n### Nitpicks and Misc\n\n- **C.1** Section 3.2 - You can round 29.97FPS to 30 FPS unless the 0.03FPS are critical to your research.\n- **C.2** Section 3.4 - The hypothetical here on \"why not use a subset of InfLevel\" as training dataset was confusing on first read because it sounded like you were actually proposing that.\n- **C.3** Section 4.1 - I think the entirety of section 3.4 can be cut from the paper or at least moved to the appendix and this would make space for a brief explanation of the different methods tested in 4.1.\n- **C.4** Section 4.2 - I don't see any dedicated intuitive-physics models in your baselines, only video prediction methods. DensePhysNet would be an example or anything trained on the PHYRE benchmark like \"Causal World Models by Unsupervised Deconfounding of Physical Dynamics\" (Li et al.).\n- **C.5** Section 4.3 - I really, really appreciate that you included the significance of your results.\n- **C.6** Table.1 - The table appears before you mention what \"JointEval\" is - so I'd put the table one page later.\n- **C.7** Section 5 - Limitation section is great. Discussion is good and should also be cut down to max half its current size because it's way too verbose.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces 2 new datasets for the evaluation of infant-level physical reasoning in ML systems. It also proposes to use OOD as method to quantify surprise, and experiments with several models showing that their dataset is currently unsolved, despite being trivial to humans.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I'm currently recommending acceptance because I believe that the authors did indeed contribute their contributions and I think they'll be used by some members of the ML community.\nThe main thing's I'd recommend changing though, to make the paper stronger:\n- Add at least a comment on your reasoning (cf. W.1).\n- Trim the work to 10 pages or less.\n", "broader_impact_concerns": "no concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FyleU9JLpE", "reviewer": "Reviewer_QbhC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Pro:**\n * Translation of an established and well-studied paradigm in developmental psychology to a setting for testing artificial systems.\n * Novel dataset, consisting of real-world data and simulated data (which can itself enable interesting research on the reliability of pure-simulation evaluation)\n * Well-written paper\n\n**Con:**\n * The metric to assess whether an artificial system understands simple visual physics is to measure “surprise” in non-plausible visual scenes by estimating some distance in activation space and comparing it to the distance observed in plausible visual scenes. Currently no artificial system performs (much) better than chance, which can be due to at least three hypotheses (two of which need to be ruled out for the benchmark to be meaningful):\n   * Current ML systems do not have an intuitive understanding of simple visual physics\n   * ML systems have an intuitive understanding but the metric is not sensitive enough to pick up on potentially very small “distances”/differences between plausible and implausible activation/output patterns. There is an attempt to verify this in the paper (Sec. 4.1, Fig. 6), but the synthetic changes introduced to make scenes “physically implausible” correspond to large-scale changes to simple statistics which could lead to much more significant “distances” in activation patterns compared to the changes relevant for the benchmark, which might only lead to (small) differences in higher-order statistics (of a fairly high order). Importantly, this hypothesis is still consistent with the results in Table 1 (where the metric is clearly sensitive to the synthetic changes, see last two columns).\n   * ML systems have an intuitive understanding and the metric is fine / sensitive enough, but the reference dataset for computing distances (called query set in the paper, Sec. 4.3) is inappropriate. I personally think this is less likely, but nevertheless cannot be taken for granted and needs careful evaluation.\n\nOverall, the only major concern I currently have is w.r.t. the reliability of the proposed metric to actually measure the desired difference. Since the metric is essential to the benchmark, this needs to be strengthened for acceptance of the paper (unless the paper is rewritten as a dataset-only paper, but I personally would not recommend that). Below are some changes and suggestions for doing this - I want to strongly encourage the authors to try and harden the metric as much as possible, because that will ultimately lead to a very meaningful benchmark that will stand the test of time. If this can be achieved, then I think the paper has good potential to be a very strong piece of work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new benchmark for evaluating intuitive physical-reasoning capabilities of machine learning models. To this end, the paper focuses on 3 important factors of human physical understanding of short visual sequences - these factors (Continuity, Solidity, and Gravity) have been extensively studied in human infants, and many developmental psychologists even argue that they arise as part of innate core physical-reasoning principles that humans are born with. The benchmark consists of 7.5k clips of real-world recordings (using the same setup that is used to test human infants) from 3 camera angles, and 75k simulated clips representing the same scenes but with a larger variety of objects. The benchmark is intended for evaluation/diagnosis only (of models trained on arbitrary datasets), and is designed to be architecture agnostic. To achieve this, physical understanding of models is measured by quantifying “surprise” (or novelty) on non-plausible scenes compared to plausible scenes (inspired by how physical reasoning is assessed in human infants). Finally, a number of state-of-the-art neural architectures are evaluated on the benchmark, showing (almost) no implicit understanding of intuitive physics, according to the metric evaluated.\n\nTo me personally, the idea behind the benchmark is sound and interesting - staying as close as possible to paradigms used in developmental psychology can allow for fruitful exchange between researchers in quite different fields (which is also discussed towards the end of the paper). To me, the dataset seems well designed, and having both a real-world and a simulated part enables interesting research questions (can the simulated data be an adequate replacement for real-world recordings?) and easy scaling to larger and more diverse datasets. My main worry with the current manuscript is the metric that is proposed to assess physical understanding: “surprise” is measured by various distance measures in activation/output space (a technique borrowed from unsupervised outlier detection). A solid and reliable metric is essential to the benchmark, but currently I have some doubts whether the metric is actually capable of reliably measuring what it’s supposed to measure (see more details/discussion below). I currently think that the metric needs more work to be convincing (and I suggest some ideas below) - otherwise the whole paper is reduced to a novel dataset, rather than a benchmark. I am looking forward to hearing the other reviewers’ opinions and authors’ thoughts on this issue.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Major/important changes\n\nAs stated above, there are (at least) three hypotheses that would explain the results observed in Table 1, but only one of the hypotheses is in favor of the main claims of the benchmark results. The other two hypotheses need to be ruled out as much as possible. Below are some suggestions for control experiments, baselines, and ablations (some more, some less elaborate - perhaps not all suggestions are implementable in the relatively short rebuttal phase). I want to encourage the authors to think about additional / more appropriate versions and propose these as well if relevant.\n\n1. The synthetic changes to produce Fig. 1 (PS, Sh, BW, Fr), are quite crude and should lead to clearly distinguishable activation/output patterns because they lead to significant deviations from natural video statistics. The metrics might simply pick up on these deviations. But for the plausible and implausible videos shown used in the benchmark, the high-level video statistics do not change between the two classes. Detecting subtle differences might need different, more sensitive metrics.\n \n1a (simple). Repeat the experiment in Fig. 1 with an untrained network. If the untrained network leads essentially to the same results, then the metric only picks up on input statistics that “leak” into the network activations; which would make the metric less reliable for detecting “surprise” in a model that “understands” what these activation patterns are.\n\n1b (medium). Use the simulator to create two kinds of videos with changes similar to the ones between physically plausible and implausible (i.e. not interfering with natural video statistics), but designed such that FastSlow should be able to pick up on these changes. Repeat the experiment for Fig. 1 with this dataset and show that the metrics proposed work in this setting.\n\n1c (quite elaborate, but most reliable). Train some auto-encoder or other generative model with an explicit latent representation, or a likelihood that can be evaluated on the benchmark data using only the physically plausible part (generate more synthetic data if needed to train the model). Then, measure the likelihood (or latent-space uncertainty) on the non-plausible videos. There should be an increased latent-space uncertainty / decreased likelihood (of generating such a video), corresponding exactly to the desired “surprise” of implausible videos. If this works, then apply the metrics introduced in 4.1 to this model - the metrics should be able to clearly measure the surprise of this model.\n\n\n2. (simple) As a baseline, train a (linear) detector using the ground-truth data from the benchmark (as suggested at the beginning of Sec. 3.4). Reproduce (parts of) Table 1 with this detector. This is not intended for actual use in the benchmark, but to get an upper bound on the detectability (with all the caveats mentioned in the paper). If this experiment fails, or only delivers poor results, then there is little hope to have more significant results with the metrics proposed in the paper.\n\n2a. Repeat the experiment but with untrained networks - this helps estimate how much of the detector’s accuracy depends on the network actually having learned meaningful representations vs. simply picking up on information that leaks into the network.\n\n3. (simple) What is the impact of the reference dataset (query set) on the metrics? It would be nice to see one or two experiments where the reference dataset is change, just to see whether that has a significant impact on the results or not. This could also be done roughly (but perhaps more easily) via the experiment that leads to Fig. 1.\n\n4. (simple) Subsection “What violations can these models detect?”: the conclusions drawn must be taken with care. The results could equally be explained by having developed a metric that is sensitive to changes in representations caused by PS, Sn, BW, and Fr. To strengthen the results, a comparison against the same evaluation (last two columns in Table 1) but with untrained networks would be interesting.\n\n5. (simple) Briefly clarify in the main text how the metrics in Sec. 4.1 (Table 1) are roughly computed. Do they depend on network outputs only, or do they involve final-layer activations, etc.?\n\nMinor (not crucial for acceptance)\n\nWhy is the focus on a single scalar metric? Besides overall accuracy, it could be interesting to report both false positive rates and false negative rates. It seems that most current models would have high false negative rates (implausible video is characterized as not surprising) but low false positive rates (plausible video is characterized as surprising). This could then still be collapsed into something like an area-under-curve metric to get a single scalar.\n\nWhy are there short cuts (0.5s) between the phases of the videos? It seems to me that a continuous video would be preferable to assess surprise or intuitive physical understanding. If this is simply a consequence of more data efficiency for recording the videos, this should be noted as such. It would also be good to have at least some discussion why the short cuts are not problematic (after all, infants in the experiment do not get to see these short cuts). The strongest argument would be studies with essentially the same outcomes but with human infants who watch videos instead of a live experimenter and experience the same cuts. \n\nWhile I appreciate the focus on evaluation only, please discuss briefly that this means that the benchmark results (in the future) must be interpreted with care. The same model trained on different data might easily obtain different benchmark scores. Since the training dataset and protocol is not fixed, future authors reporting results need to take good care of reporting these details, and should avoid tweaking the training data (and perhaps even the reference dataset / query set) to make their model appear to “beat the benchmark”.\n", "broader_impact_concerns": "I have no concerns regarding the broader impact.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qC49Q9RczK", "reviewer": "Reviewer_S66y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- The paper is overall very rigorous and thorough in its argumentation. I was particularly impressed by the experiments in Section 4.1 to test the hypothesis that surprise is \"additive\". Many papers would not even articulate that assumption, let alone design an entire experiment to validate it. I was also impressed by the experiments with JointEval, which were helpful for supporting the argument that the main null results are due to weaknesses in existing methods, rather than a flaw in the benchmark itself.\n- I like the commitment to \"evaluation only\" in the benchmark, and found the justification compelling.\n- The \"Conflating features with physical understanding\" example was also convincing and clear. Using OOD instead of training a linear decoder seems like a generally good idea.\n- The writing is overall very clear. I only found a few small typos noted below.\n- Overall, the paper was a pleasure to read, and I expect that the benchmark will be a useful contribution to the field.\n\n### Weaknesses\n\n- The need for a training set to use with most of the OOD detection methods seems limiting. My main concern is that it could become difficult to disentangle whether the training dataset or the approach under evaluation should be blamed/credited for bad/good OOD detection.\n- Against the backdrop of existing work, the real-world version of the benchmark proposed here seems like a much more valuable contribution than the simulated version. I am not convinced that the simulated version is necessary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes InfLevel, a benchmark for physical reasoning in video. The benchmark specifically focuses on three kinds of core knowledge, continuity, solidity, and gravity, which have been studied extensively in humans. The benchmark consists of a large dataset of videos that have violations of expectations under those three principles. The dataset includes both real videos and simulated videos. The benchmark also stipulates that out-of-distribution metrics should be used to evaluate models. In experiments with off-the-shelf models, it is found that all models fail to distinguish the violations of expectation in nearly all cases. Further experiments are conducted to support the design choices.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- I see no reason why the benchmark proposed here should be limited to \"end-to-end learning with deep-learning architectures\" -- a benchmark should be approach-agnostic. I would not say that it's necessary to evaluate other kinds of approaches for the present paper, but I do think the manuscript would be strengthened if the mentions of deep learning were replaced with more general terms.\n- Relatedly, the introduction briefly mentions that \"there is an active community of researchers pursuing [AI models that build in intuitive physics]\" and cites Lake et al (2017) as an example. Given the topic of this paper, a much more thorough discussion of models that build in physical priors and take inspiration from cognitive science would be appropriate.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present their new evaluation benchmark dataset of videos that are intended to assess physical reasoning of vision-based AI systems.  The benchmark consists of thousands of real videos as well as tens of thousands of synthetic videos of simulated physical environments.  They also propose a set of metrics (and propose pooling them by a voting rule) for assessing vision-based AI systems based whether it correctly identifies videos with physically implausible activity as out-of-domain (conceptually similar to violation-of-expectation / surprise metrics).  These metrics support evaluation of models that have not been trained to output surprise. The authors show that many baselines fail to detect physical violations and they confirm their metrics are sufficiently sensitive under appropriate conditions.  

All reviewers believed the work was well motivated and well written.  Two reviewers expressed some concerns about the metrics, which Reviewer QbhC found to be the overall weakest part of the submission in that this reviewer initially expressed some doubts about whether the metric is capable of reliably measuring what it is supposed to measure.  

The authors responded thoughtfully and thoroughly to the various reviewer concerns. Prompted by the reviews, the authors shared anonymous code and part of the dataset for reviewer inspection.  
 
Subsequently, all reviewers felt the core contributions were sufficiently strong to accept the submission.  However, Reviewer QbhC continues to believe that the OOD metrics used, which rely on distance between typical plausible and typical implausible inputs, may be overwhelmed by major image statistics and less able to distinguish higher-order statistics associated with physical plausibility vs. implausibility.  The authors do acknowledge limitations of the current metrics in the discussion with the reviewers, though this comes across a little less in the actual paper.  I'd encourage a small clarification somewhere in the paper along the lines of what was communicated in the discussion with Reviewer QbhC.  For example, in the discussion with the reviewer, the authors state "Given the above, we do allow people to define their own surprise metrics when evaluating their models on our benchmark (so long as they do not train these metrics on our data of course) and hope that others will continue to improve and refine our model-agnostic surprise measures."  This open-mindedness could be emphasized in the main text.

typo in the paper: fig 2 "Physicall" 

My assessment aligns with the reviewers that this paper should be accepted.
Decision: Accept (poster)