ComPhy: Compositional Physical Reasoning of Objects and Events from Videos
Authors: Zhenfang Chen
Kexin Yi
Yunzhu Li
Mingyu Ding
Antonio Torralba
Joshua B. Tenenbaum
Chuang Gan
Venue: ICLR 2022 Poster (2022)
Keywords: Compositional
Intutive Physics
Video Reasoning
Neural-Symbolic

Abstract: Objects' motions in nature are governed by complex interactions and their properties. While some properties, such as shape and material, can be identified via the object's visual appearances, others like mass and electric charge are not directly visible. The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world, whereas humans can effortlessly infer them with limited observations. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interaction. In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset. For a given set of objects, ComPhy includes few videos of them moving and interacting under different initial conditions. The model is evaluated based on its capability to unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions posted on one of the videos. Evaluation results of several state-of-the-art video reasoning models on ComPhy show unsatisfactory performance as they fail to capture these hidden properties. We further propose an oracle neural-symbolic framework named Compositional Physics Learner (CPL), combining visual perception, physical property learning, dynamic prediction, and symbolic execution into a unified framework. CPL can effectively identify objects' physical properties from their interactions and predict their dynamics to answer questions. 

Published as a conference paper at ICLR 2022
COMPHY: COMPOSITIONAL PHYSICAL REASONING OF OBJECTS AND EVENTS FROM VIDEOS
Zhenfang Chen
Kexin Yi Harvard University
Yunzhu Li
Mingyu Ding
MIT-IBM Watson AI Lab
MIT
The University of Hong Kong
Antonio Torralba MIT
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
Chuang Gan MIT-IBM Watson Al Lab
ABSTRACT
Objects' motions nature are governed by complex interactions and their prop- erties. While some properties, such as shape and material, can be identified via the object's visual appearances, others like mass and electric charge are not di- rectly visible. The compositionality between the visible and hidden properties poses unique challenges for AI models to reason from the physical world, whereas humans can effortlessly infer them with limited observations. Existing studies on video reasoning mainly focus on visually observable elements such as object appearance, movement, and contact interaction. In this paper, we take an initial step to highlight the importance of inferring the hidden physical properties not directly observable from visual appearances, by introducing the Compositional Physical Reasoning (ComPhy) dataset 1, For a given set of objects, ComPhy includes few videos of them moving and interacting under different initial conditions. The model is evaluated based its capability unravel the compositional hidden properties, such as mass and charge, and use this knowledge to answer a set of questions posted on one of the videos. Evaluation results of several state-of-the-arr video reasoning models on ComPhy show unsatisfactory performance as they fail to capture these hidden properties. We further propose an oracle neural-symbolic framework named Compositional Physics Learner (CPL), combining visual per- ception, physical property learning, dynamic prediction, and symbolic execution into unified framework. CPL can effectively identify objects' physical properties from their interactions and predict their dynamics to answer questions.
INTRODUCTION
Why do apples float in water while bananas sink? Why do magnets attract each other on a certain side and repel on the other? Objects in nature often exhibit complex properties that define how they interact with the physical world. To humans, the unraveling of new intrinsic physical properties often marks important milestones towards deeper and more accurate understanding of nature. Most of these properties are intrinsic as they are not directly reflected the object's visual appearances or otherwise detectable without imposing interaction. Moreover, these properties affect object motion in a compositional fashion, and the causal dependency and mathematical law between different properties are often complex.
As shown in Fig. 1, different intrinsic physical properties, such as charge and inertia, often lcad to drastically different future evolutions. Objects carrying the same or opposite charge will exert repulsive or attractive force on each other. The resulting motion not only depends on the amount of charge each object carries, but also their signs (see Fig. 1-(a)). The inertia determines how sensitive an object's motion is to external forces. When a massive object interacts with a light object via attraction, repulsion, or collision, the lighter object will undergo larger changes in its motion compared with the massive object's 's trajectory (see Fig. 1-(b)).
Recent studies have established a series of benchmarks to evaluate and diagnose machine learning systems in various physics-related environments (Bakhtin et al., 2019; Yi et al., 2020; Baradel et al.,
Project page: https: / http://mmmmorrrcnnnnngg.g. wit.. thub.

---

Published as a conference paper at ICLR 2022
ccccccoc
(Same charge on 1, 2)
(Opposite charge on 2)
(4 heavier than 5)
(4 lighter tham 5)
time
(a)
(b)
Figure 1: Non-visual properties like mass and charge govern the interaction between objects and lead to different motion trajectories. a) Objects attract and repel each other according to the (sign of) charge they carry. b) Mass determines how much an object's 's trajectory is perturbed during an interaction. Heavier objects have more stable motion.
2020). These benchmarks introduce reasoning tasks over wide range of complex object motion and interactions, which poses enormous challenges to existing models since most tasks require them to fully capture the underlying physical dynamics and, in some cases, be able make predictions. However, the majority of complexity in the motion trajectories facilitated by these environments comes from changes or interventions in the initial conditions of the physical experiments. The effects of object intrinsic physical properties, as well as the unique set of their challenges, are therefore of great importance for further investigation.
It's non-trivial to build benchmark for compositional physical reasoning. Existing benchmarks (Yi et al., 2020; Ates et al., 2020) assume that there is no variance in objects' physical properties and ask models to learn physical reasoning from massive videos and question-answw pairs. A straightforward solution is to correlate object appearance with physical properties like making all red spheres to be heavy and then ask questions about their dynamics. However, such design may incur shortcuts for models by just memorizing the appearance prior rather than understanding coupled physical properties. In this work, we propose novel benchmark called ComPhy that focuses on understanding object-centric and relational physics properties hidden from visual appearances. ComPhy first provides few video examples with dynamic interactions among objects for models identify objects' physical properties and then asks questions about the physical properties and corresponding dynamics. As shown Fig. 2, ComPhy consists of meta-train sets and meta-test sets, where each data point contains 4 reference videos and 1 target video. Within each set, the objects share the same intrinsic physical properties across all videos. Reasoning on ComPhy requires the model to infer the intrinsic and compositional physical properties of the object set from the reference videos, and then answer questions about this query video. To make the task feasible, we systematically control each object in the query video that should appear at least in one of the reference videos.
We also introduce an oracle model to tackle this task. Inspired by recent work on neural-symbolic reasoning on images and videos (Yi et al., 2018; 2020; Chen al., 2021), our model consists of four disentangled components: perception, physical property learning, dynamics prediction, and symbolic reasoning. Our model is able infer objects' compositional and intrinsic physical properties, predict their future, make counterfactual imaginations, and answer questions.
To summarize, this paper makes the following contributions. First, we present a new physical reasoning benchmark ComPhy with physical properties (mass and charge), physical events (attraction, repulsion), and their composition with visual appearance and motions. Second, we decorrelate physical properties and visual appearance with a few-shot reasoning setting. It requires models to infer hidden physical properties from only a few examples and then make predictions about the system's 's evolution to help answer the questions. Third, we propose an oracle neural-symbolic framework, which the objects' movements. At the core of our model are graph neural networks that capture the commaroee c compossitiona
modularized model that can infer objects' physical properties and predict
nature of the underlying system.
2 RELATED WORK
Physical Reasoning. Our work is closely related to recent developments in physical reasoning benchmarks (Riochet et al., 2018; Girdhar Ramanan, 2020; Ates et al., 2020; Hong et al., 2021). PHYRE (Bakhtin al., 2019) and its variant ESPRIT (Rajani et al., 2020) defines eosiroommen
2

---

Published as a conference paper at ICLR 2022
Question Diagnostic
Few-shot
Physical
Counterfactual
Dataset
Video
Answering Annotation
Composition
Reasoning
Property Property Dynamics
CLEVR (Johnson et al., 2017)
MovieQA (Tapaswi et al. 2016)
TGIF-QA (Jang et al., 2017)
TVQA/TVQA+ (Lei et ul., 2019)
AGQA (Grunde Mcl.auhhinn et al., 2021)
IntPhys (Riochet et al, 2018)
PHYRE/ ESPRIT (Rajani et 2020)
2020)
Cater (Riochet et al., 2018)
CoPhy(Baradel 2020)
CRAFT (Ates al., 2020)
CLEVRER et al., 2020)
ComPhy (ours)
Table I:
Comparison between ComPhy and other visual reasoning benchmarks. ComPhy is the dataset with reasoning tasks for both physical property learning and corresponding dynamic prediction.
where objects can move within a vertical 2D plane under gravity. Each task is associated with goal state, and the model solves the task by specifying the initial condition that will lead to the goal state. CLEVRER (Yi et al., 2020) contains videos of multiple objects moving and colliding on a flat plane, posting natural language questions about description, explanation, prediction, and counterfactual reasoning on the collision events. CoPhy (Baradel et al., 2020) includes experiment trials of objects moving in 3D space under gravity. The task focuses on predicting object trajectories under counterfactual interventions reasoning benchmarks by focusing on physical events driven by object intrinsic properties (situations shown in Fig. 2). ComPhy requires models to identify intrinsic properties from only few video examples and make dynamic predictions based on the identified properties and their compositionality.
the initial conditions. Our dataset contributes to these physical
Dynamics Modeling. Dynamics modeling of physical systems has been a long-standing research direction. Some researchers have studied this problem via physical simulations, drawing inference on the important system- and object-level properties via statistical approaches such as MCMC (Battaglia al., 2013; Hamrick et al., 2016; Wu et al., 2015), while others propose to directly learn the forward dynamics via neural networks (Lerer et al., 2016). Graph neural networks (Kipf & Welling, 201 17), due their object- and relation-centric inductive biases and efficiency, have been widely applied in forwarding dynamics prediction on a wide variety of systems (Battaglia et al., 2016; Chang al., 2016; Sanchez-Gonzaler
al., 2020; Li et al., 2019a). Our work combines the best of the two approaches by first inferring the object-centric intrinsic physical properties and then predicting their dynamics based on the intrinsic properties. Recently, VRDP (Ding et al., 2021) performs object-centric dynamic reasoning by learning differentiable physics models.
Video Question Answering. Our work is also related to answering questions about visual content. Various benchmarks have been proposed handle the tasks of cross-modal learning (Lei et al., 2018; Chen et al., 2019; Li et al., 2020; Wu et al., 2021; Hong et al., 2022). However, they mainly focus on understanding human actions and activities rather than learning physical events and properties, which is essential for robot planning and control. Following CLEVRER, we summarize the difference between ComPhy and previous benchmarks in Table 1. ComPhy is the only dataset that requires the model to learn physical property from few video examples, make dynamic predictions based on the physical property, and finally answer corresponding questions.
Few-shot Learning. Our work is also related to few-shot learning, which typically learns to classify images from only few labelled examples ( (Vinyals et al., 2016; Snell et al., 2017; Han et al., 2019). ComPhy also requires models to identify objects' property labels from only a few video examples. Different from them, reference videos have no labels for objects' physical properties but more interaction among objects, providing information for models to identify objects' physical properties.
3 DATASET
ComPhy studies objects' intrinsic physical properties from objects' interactions and how these properties affect their motions in future and counterfactual scenes answer corresponding questions. We first introduce videos and task setup in Section 3.1. We then discuss question types in Section 3.2, and statisties and balancing in Section 3.3.
3.1 VIDEOS
Objects and Events. Following Johnson et al. (2017), objects in ComPhy contain compositional appearance attributes like color, shape, and material. Each object in videos can be uniquely identified
3

---

Published as a conference paper at ICLR 2022
time
time
Reference video Z
ubj1 ubj2 gel
obji obj2 repel
obj2 obj4 collide
ubj2 ubj4 change direction
time
time
Reference videoo
Reference video
obji and obj4 collide
ubj1 and obj3 collide
obji obj4 collide
obji and obji chunge direction
time
Query
obj1 and obj2 repel
abj2 and obj3 collide
obj2 changes direction
ohj2 and ohj4 close
Factual Quastion
Counterfactual Question
III. Predictiva
Q: many moving objects are uncharged?
event would happen the
Q: Which event happen
purple object were heavier?
The red sphere and the brown
A 2
a) The green cube would collide
cube collide
x
Q: Is the purple sphere than
with the metal sphere sphere would collide with the brown cube
b) purple sphere brown
the brown cube?
cube collide
A. No
Figure 2: Sample target video, reference videos and question-answer pairs from ComPhy-
by these three attributes for simplicity. There events, in, out, collision, attraction and repulsion. These object appearance attributes and events form the basic concepts of the questions in ComPhy.
Physical Property. Previous benchmarks (Riochet et al., 2018: Yi al., 2020) mainly study appearance concepts like color and collision that can be perceived in even a single frame. In ComPhy, we additionally study intrinsic physical properties, mass and charge, which can not be directly captured from objects' static appearance. As shown in Fig. (a), objects with same or opposite charge will repel or attract each other while objects without charge will not affect each other's motion without collision. As shown Fig. 1 (b), the object with larger mass (inertia) tends to maintain its original moving direction after collision while the light object changes much more in moving direction. Note that these intrinsic physical properties and can be combined with each other to generate more complicated and diverse dynamic scenes. For simplicity, ComPhy contains two mass values (heavy and light) and three charge types (positive charged, negative charged and uncharged). Theoretically, we can add more physical properties like bounciness coefficients and friction into ComPhy and make their values continuous. However, such design will make the dataset too complicated and even difficult for people to infer the properties.
orthogonal the appearance attributes
Video Generations. For each video, first physical engine (Coumans & Bai, 2016-2021) to simulate objects' motions and then adopt a graphs engine (Community, 2018) to render frame sequences. Each target video for question answering contains to 5 objects with random composition between their appearance attributes and physical properties. We set the length of the target video to be seconds and additionally simulate the 6-th and 7-th seconds of the target video for predictive question annotation. We provide more video generation details in the Appendix.
Task Setup. It is not trivial design task setup to evaluate models' ability for physical reasoning since physical properties are not observable in static frame. A straightforward design is to correlate object appearance with the physical property like "red object is heavy", "yellow object light" and then ask "what would happen if they collide". However, such setting imperfect since it can not evaluate whether a model really understands the physical properties or just memorize the visual appearance prior. An ideal setting should be able evaluate whether model is like human that can identify objects' properties from objects' motion and interactions with each other in the dynamic scenes and make the corresponding dynamic predictions.
To achieve this, we design a meta setting for physical reasoning, which provides few reference video samples along with the target video for models to infer objects' physical properties and then ask
4

---

Published as a conference paper at ICLR 2022
questions about the objects' physical properties and dynamics. We show sample of the dataset in Fig. 2. Each set contains a target video, 4 reference videos, and some questions about the visual attributes, physical properties, events, and dynamics of the target video. Objects in each set share the same visual attributes (color, shape, and material) and intrinsic physical property (mass and charge).
To provide abundant visual content for physical property inference, we addition- ally provide 4 reference videos for each target video. We sample 2-3 objects from the target video, provide them with different initial velocities and locations, and make them interact (attract, repel or collide) with each other. The generation of the reference video follows the same standard as the target video, but the length of the videos is set to 2 seconds for scaling up. The interaction among objects in reference videos helps models to inference objects' properties. For example, the repulsion in Reference video of Fig. 2 can help us identify that object and object 2 carrying the same charge.
Reference Videos.
3.2 QUESTIONS
Following Johnson et al. (2017) and Yi et al. (2020), we develop a question engine to generate questions for factual, predictive, and counterfactual reasoning. Each question is paired with a functional program, which provides a series of explicit reasoning steps. We set all factual questions to be "open-ended" that can be answered by single word or short phrase. We set predictive questions and counterfactual questions to be multiple-choice and require models to independently predict each option is true false. We provide question templates and examples in the Appendix.
Factual. Factual questions test models' ability to understand and reason about objects' physical properties, visual attributes, events, and their compositional relations. Besides the factual questions in existing benchmarks (Yi et al., 2020; Ates et al., 2020), as shown in the samples in Fig. 2 (I), ComPhy includes novel and challenging questions asking about objects' physical properties, charge and mass.
Predictive questions evaluate models' ability to predict and reason about the events happening after the observed target video ends. It requires a model to observe objects' location and velocity at the end, identify objects' physical property and predict what will or not happen next.
Predictive.
Counterfactual Charge and Mass. Counterfactual questions ask what would happen on a certain hypothetical condition. ComPhy targets at reasoning the dynamics under the hypothesis that a certain object carrying different physical property. Fig. 2 (II) shows typical question samples. Previous work (Yi et al., 2020; Riochet et al., 2018) also has counterfactual questions. However, they are merely based on the hypothesis that object is removed rather than an object is carrying a different physical property value, which has different or even opposite effects on object motion prediction.
3.3 BALANCING AND STATISTICS
Overall, ComPhy has 8,000 sets for training, 2,000 sets for validation, and 2,000 for testing. It contains 41,933 factual questions, 50,405 counterfactual questions and 7,506 predictive questions, occupying 42%, 50% and 8% of the dataset, respectively. For simplicity, we manually make sure that video set will only contain pair of charged objects if it contains charged objects. Similarly, a video will only contain heavy object or no heavy objects. A non-negligible issue of previous benchmarks like CLEVRER is its bias. As pointed out by Ding et al. (2020), about half of the counterfactual objects in CLEVRER have no collisions with other objects and the counterfactual questions can be solved merely based on the observed target videos. In ComPhy, we manually remove counterfactall questions on objects that have no interaction with other objects. When generating questions for comparing mass values and identifying charge relations between two objects, we systematically control that the two objects should have at least one interaction in one of the provided few video examples. We make sure that the few video examples are informative enough to answer questions based on the questions' programs and the video examples' property and interaction annotation.
4 EXPERIMENTS
In this section, We evaluate various baselines and analyze their results to study ComPhy thoroughly.
4.1 BASELINES
Following CLEVRER, We evaluate a range of baselines on ComPhy in Table 2. Such baselines can be divided into three groups bias-analysis models, video-question answering models, and compositional reasoning models. To provide extensive comparison, we also implement some variant models that make use of both the target video and reference videos.
5

---

Published as a conference paper at ICLR 2022
Predictive
Counterfactual
Methods
Factual
per opt.
per ques.
per opt.
per ques.
Random
29.7
51.9
22.6
49.7
9.1
Frequent
30.9
56.2
25.7
50.3
8.7
Blind-LSTM
39.0
57.9
28.7
55.7
12.5
CNN-LSTM (Antol et al., 2015)
46.6
59.5
29.8
58.6
14.6
HCRN (Le al., 2020)
47.3
62.7
32.7
58.6
14.2
MAC (Hudson & Manning, 2018)
68.6
60.2
32.2
60.2
16.0
ALOE (Ding el al., 2020)
54.3
65.9
35.2
65.4
20.8
CNN-LSTM (Ref) (Antol ct al., 2015)
41.9
59.6
29.4
57.2
12.8
MAC (Ref) (Hudson & Manning, 2018)
65.8
60.2
30.7
60.3
14.3
ALOE (Ref) (Ding al., 2020)
57.7
67.9
37.1
67.9
22.2
Human Performance
90.6
88.0
75.9
80.0
52.9
Table 2: Evaluation of physical reasoning on ComPhy. Human performance is based on sampled questions. See Section 4.2 for more details.
Baselines. The first group of models are bias analysis models. They analyze the language bias in ComPhy and answer questions without visual input. Specifically, Random randomly selects an answer based on its question type. Frequent chooses the most frequent answer for each question type. Blind-LSTM uses an LSTM encode the question and predict the answer without visual input. The second group of models are visual question answering Models. These models answers questions based on the input videos and questions. CNN-LSTM (Antol et al., 2015) is a basic question answering model. We use Nes-5t to extract frame-level features and average them over the time dimension. We encode questions with the last hidden state from an LSTM. The visual features and the question embedding are concatenated to predict answers. HCRN (Le et al., 2020) is popular model that hierarchically models visual and textual relationships. The third group of models are visual reasoning models. MAC (Hudson & Manning, 2018) decomposes visual question answering into multiple attention- based reasoning steps and predicts the answer based on the last step. ALÔE (Ding et al., 2020) achieves state-of-the-art performance CLEVRER with transformers (Vaswani et al., 2017).
Baselines with Reference Videos. We implement some variants of existing baseline models, which adopt both the target video and reference vidcos as input. We develop CNN-LSTM (Ref), MAC (Ref) and ALOE (Ref) from CNN-LSTM. MAC and ALOE by concatenating the features of both reference videos and the target video as visual input.
Evaluation.
We the standard accuracy metric evaluate the performance of different methods. For multiple-choice questions, we report both the per-option accuracy and per-question accuracy. We consider a question is correct if the model answers all the options of the question correctly.
4.2 EVALUATION ON PHYSICAL REASONING.
We summarize the question-answering results of different baseline models in Table 2. We also find that models have different relative performances on different kinds questions, indicating that different kinds of questions in ComPhy require different reasoning skills.
Factual Reasoning. Factual questions in ComPhy require models to recognize objects' visual attributes, analyze their moving trajectories and identify their physies property answer the questions. Based on the result, we have the following observation. First, we find that the "blind" models, Random, Frequent and Blind-LSTM achieve much worse performance than other video question answering models and reasoning models. This shows the importance of modeling both visual context and linguistic information on ComPhy. We also find that video question answering models CNN- LSTM and HCRN perform worse than visual reasoning models MAC and ALOE. We believe the reasons are that models like HCRN are mostly designed for human action video, which mainly focuses on temporal modeling of action sequences rather than spatial-temporal modeling of physical events. Among all the baselines, we find that MAC performs the best on factual questions, showing the effectiveness of its compositional attention mechanism and iterative reasoning processes.
Dynamcis Reasoning. An important feature ComPhy is that it requires models to make coun- terfactual and future dynamic predictions based on their identified physical property to answer the questions. Among all the baselines, we find that ALOE (Ref) achieves the best performance on coun- terfactual and future reasoning, We think the reason is that the self-attention mechanism and object masking self-supervised techniques provides the model with the ability to model visual context and imaging counterfactual scenes answer the questions.

---

Published as a conference paper at ICLR 2022
IV. Symbolic Execution
Reference Vidaos
Property Graph
Question Choice
prrcestion
Physical Properte
"What would happen purple hoavies?
C: "The green cube would collide with the wetaa sphere
Question
Turospiion
Parser
Physical Propeing ty
Learmee
Execution Enginee
Arswer "
Targel
Percepttio
Shysical Prope ty earner
[+1
Perception
Property Inference
IlI. Dynamics Prediction
Figure 3: The perception module detects objects' location and visual appearance attributes. The physical property learner learns objects' properties based on detected object trajectories. The dynamic predictor predicts objects' dynamics in the counterfactual scene based on objects' properties and locations. Finally, an execution engine runs the program parsed by the language parser on the predicted dynamic scene answer the question.
Reasoning with Reference Videos. We observe that CNN-LSTM (Ref) and MAC (Ref) achieve only comparable or even slightly worse performance comparing with their original models, CNN- LSTM and MAC. We also find that ALOE (Ref) achieves better performance than ALOE. However, the increment from ALOE to ALOE (Ref) is limited. All these variant models can not get much gain from their original models by concatenating the reference videos additional visual input. We think the reason is that these models are based on massive training videos and question-answer pairs and have difficulties adapting the scenario in ComPhy, which requires models learn the new compositional visible and hidden physical properties from only few examples.
Human Performance. To assess human performance ComPhy, 14 people participated in the study, We require all the participants to have basic physics knowledge and can speak fluent English. Participants were first shown with few demo videos and questions to that they understood the visual events, physical properties, and the text description. Participants are then asked to answer 25 question samples different kinds in ComPhy. And we record accuracy of 90.6% for factual questions, 88.0% for predictive per option, 80.0% counterfactual per option, 75.9% for predictive per question and 52.9% counterfactual per question. This shows that while our questions are difficult for machine models, humans can still well understand the physical properties and make dynamic predictions in ComPhy to answer the corresponding questions.
5
COMPOSITIONAL PHYSICS LEARNER
5.1
MODEL
We propose an oracle model that performs compositional dynamics reasoning based on a neural- symbolic framework. Inspired by the recent models (Yi al., 2018; 2020), factorize the process of physical reasoning into several stages. As shown in Fig. 3, our model consists of four major stages: perception (I), property inference (II), dynamic prediction (III), and symbolic dynamics reasoning (IV). Given target video, 4 reference videos, and query question, we first detect objects' location and appearance attributes with perception module. Objects' trajectories are fed into a physical property learner learn their physical properties. Given objects' properties and location, the dynamic predictor predicts objects' dynamics based on the physical properties. Finally, an exccution engine runs the program from the language parser the predicted objects' motion to answer the question.
Our model's ss core contribution lies in a physical property learner that infers the hidden per-object and pairwise properties from the videos and dynamics model that predicts dynamics based on the inferred physical properties. These modules enable our model to tackle challenging physics reasoning tasks that involve physical properties that not directly observable.
7

---

Published as a conference paper at ICLR 2022
Perception. Our perception module detects and recognizes all objects at every video frame. Given an input frame, our model first applies a Mask-RCNN (He et al., 2017) to detect all objects in the frame and extract corresponding visual attributes like color, shape, and material. The perception module outputs the static attributes of all objects as well as their full-motion trajectories during the video. Please refer (Yi et al., 2018) for further details.
Physical Property Inference. The physical property learner (PPL) lies at the core of our model's capability to tackle complex and compositional physical interactions. Given the object motion trajectories from all reference and target videos, the PPL predicts the mass and relative charge between each object pair in the video. Under the hood. PPL implemented using a graph neural network (Kipf et al., 2018) where the node features contain object-centric property (mass) and edge features encode pairwise properties (relative charge).. Given the input trajectories {x: objects in the video, PPL. performs the following message passing operations,
N of N
Sem)((x),
e/. Sta v)),
= Seno! Σ el..).
(I)
where I E [0. 1] denotes the message passing steps and x; denotes the concatenation of the normalized detected object coordinates {zi.e: Vi,t)-= over all T frames. f(...) are functions implemented by fully-connected layers. We then use two fully-connected layer to predict the output mass label sgred (v?) and edge charge label (e.ed respectively.
We further note that because the system is invariant under charge inversion, the charge property is described by the relative signs between each object pair, even though the charge carried by each individual object should be easy to recover given the pairwise relation plus the true sign of one object. The full physical property of a video set can be represented as a fully connected property graph. Each node represents object that appears in at least one of the videos from the set. And each edge represents if the two nodes the edge connects carry the same, opposite, or no relative charge (that is, one or both objects are charge-neutral). As shown in Fig. 3(I), for each reference video, PPL only predicts the properties of the objects it covers, revealing only parts of the property graph. We align the predictions of different nodes and edges by objects' static attributes predicted by the perception module. The full object properties are obtained by combining all the subgraphs generated by each video from the set via max- mox-pooiin over node and edge.
Dynamic Prediction based on Physical Property. Given the object trajectories at the t-th frame and their corresponding object properties (i.e., mass and charge). we need to predict objects' positions at the L 1 frame. We achieve this using heaph-neural-network--
dynamic predictor. We represent 1.0 = I| (af, , h4, my), which is a concatenation of the i-th object at the 1-th frame with o
the object location (z), vf), size (wi, ht) and the mass label (m,) over history window of 3. We present objects by the locations of a history rather than the only location at the i-th frame to encode the velocity and account for the perception error. Specifically, we have
Σ
se.. (o o 1.0)
L1+1
of + greet > (b
Oj
(2)
h hd+1 Σ (lo 441
411 D.
where k E (0,1,2) represents if the two connected nodes carry the same, opposite, no relative charge. 2j.j.k are the k-th element of the one-hot indication vector Z,,j. I E [0, 1] denotes the message functions implemented by fully-connected layers. We predict object location and size at the / + 1-th frame using a function that consists of one fully connected layer (0)2) 222 Given the target video, we predict the future by initializing dynamic predictor input with the last three frames of the target video and then iteratively predict the future by feeding the prediction back to our model. We get the physical property-basee prediction for the counterfactual scenes by using the first frames in the target video as input and updating their mass label (m,) and one-hot indicator vector z,.j correspondingly.
passing steps and, I(...)
Symbolic Execution. With the object visual attributes, physical property, and dynamics ready, we use question parser to transform the question and choices into functional programs and perform step-by-step symbolic execution to get the answer. We provide more details in the Appendix.
5.2 PERFORMANCE ANALYSTS
Effectiveness of Physical Property Learning. We report the performance of the proposed oracle CPL. in Table 3. We can see that CPL shows better performance on all kinds of questions compared
8

---

Published as a conference paper at ICLR 2022
Predictive
Counterfactual
ALOR AL-S PP4)
Methods Factual
CPL
per opt.
per ques. per opt. per ques.
CPI.
80.5
75.3
56.4
68.3
29.1
NS-DR+
73.3
50.8
61.1
16.6
CPL-Gt
100
87.6
74.0
74.0
35.3
Pactual
Prodeiive
Proditis qack
Counatiatiaal
Counerductel
pet wp..
pat spr.
per quos.
Table 3: Evaluation of CPL on ComPhy.
Figure 4: Generalization of physical reasoning.
to the baseline methods in Table 2. The high accuracy on factual questions shows that CPL is able to inference objects' physical properties from few video examples and combine them with other appearance properties and events for question answering. Specifically, we compare the mass and charge edge label prediction result with the ground-trui -truthhtuthhhuo labels and find that they achieve an accuracy of 90.4% and 90.8%. This shows that that the PPL in CPL are able learn physical properties from objects' trajectories and interactions in a supervised manner. We also implement a graph neural network baseline that only relies on the target video without reference videos. It achieves an accuracy of 59.4% and 70.2% on mass and charge edge labels. This indicates the importance of reference videos for physical property inference. For better analysis, we have reported a baseline with ground-truth object properties and visual static and dynamic attributes (CPT-Gt). Although it achieves constant gains, there is still much room for further improvement. This shows the bottleneck of ComPhy lies in predicting physical -based dynamics.
Dynamics Reasoning. The better performance in counterfactual and predictive questions indicates that CPL can image objects' motions in counterfactual and future scenes based on the identified physical properties to some extent. We also notice that there is still a large gap between CPL's performance and human performance shown in Section 4.2 especially We find that the dynamic predictor CPL still shows its limitation on long-term dynamic prediction. This suggests that a stronger dynamic predictor may further boost CPL's performance.
counterfactual reasoning.
Comparison between CPL and NS-DR+. We also implement a variant of the NS-DR (Yi et al., 2020) model for better analysis. NS-DR does not explicitly consider changes in physical properties like mass and charge; thus it cannot run the symbolic programs related to the physical properties in ComPhy. To run NS-DR successfully in ComPhy, we provide NS-DR with extra ground-truth physical property labels. The variant, NS-DR+ uses the PropNet (Li et al., 2019b) for dynamics predictions, which does not consider the mass and charge information of the objects. Compared to our model, NS-DR+ assumes extra information and can directly answer factual questions using the supplied ground-trute labels; thus we ignore the comparison on factual questions and focus on the evaluation on the models' ability for dynamic prediction. As shown in Table 3, CPL achieves much better performance than NS-DR+ on counterfactual questions and predictive questions, showing the importance of modeling mass and charges on nodes and edges of the graph neural networks.
Generalization to More Complex Scenes. To test the model's generalization ability, we additionally generate a new dataset that contains videos with more objects (6 to 8) and contains both an object with a large mass and pair of charged objects. We show the performance of the best baseline models ALOE, ALOE (Ref) and CPL in Fig. 4. Both ALOE models and CPL have large drop in question-ansee answering performance. We believe the reason is that the transformer layer in ALOE, the graph neural network based-physica. property learner and dynamics predictor in CPL show limitation on more complex visual dynamic scenes, which has a different statistic distribution as the scenes in the original training set. We leave the development of more powerful models with stronger generalization ability as future work.
6 CONCLUSIONS
In this paper, we present benchmark named ComPhy for physical reasoning in videos. Given few video examples, ComPhy requires models to identify objects' intrinsic physical properties and predict their dynamics in counterfactual and future scenes based on the properties to answer the questions. We evaluate various existing thee-atft models ComPhy, but none of them achieves satisfactory performance, showing that physical reasoning in videos still a challenging problem and needs further exploration. We also develop oracle model named CPL, using graph neural networks to infer physical properties and predict dynamics. We hope that ComPhy and CPL can attract more research attention in physical reasoning and building more powerful physical reasoning models.

---

Published as a conference paper at ICLR 2022
Acknowledgement This work was supported by MIT-IBM Watson Al Lab and its member company Nexplore, ONR MURI, DARPA Machine Common Sense program, ONR (N00014-18-1-2847), and Mitsubishi Electric.
10

---

Published as a conference paper at ICLR 2022
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 6
Tayfun Ates, Muhammed Samil Atesoglu, Cagatay Yigit, llker Kesen, Mert Kobas, Erkut Erdem, Aykut Erdem, Tilbe Goksun, and Deniz Yuret. Craft: A benchmark for causal reasoning about forces and interactions. arXiv preprint 2002:220 2020. 2, 14
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015. 14
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. In Advances in Neural Information Processing Systems, volume 32, 2019. 2
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counter- factual learning of physical dynamics. In International Conference Learning Representations, 2020. 1,3
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Advances in Neural Information Processing Systems, volume 29, 2016.
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding, Proceedings of the National Academy of Sciences, 110(45)11327-11333 2013. 3
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 1190, 2019. 15
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. compositional object-based approach to learning physical dynamics. arXiv preprint arXiv:/612.0034// 2016. 3
Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Yee Kenneth Wong. Weakly-supervised spatio-
temporally grounding natural sentence in video. In ACL, 2019. 3
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan- Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. In International Conference on Learning Representations, 2021.
Blender Online Community. Blender a 3d modelling and rendering package, 2018. URL ht:p::
//www www.bleeeeedee org. 4, 14
Erwin Coumans and Yunfci Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http::/pybullett org, 2016-2021. 4, 14
David Ding, Felix Hill, Adam Santoro, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning, arXiv, 2020. 5,
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Joshua B Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. In NeurlPS, 2021.
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. In ICLR, 2020. 2
Madeleine Grunde-Mcl Grunde-Mel.aughlin, Ranjay Krishna, and Maneesh Agrawala. Agqa: benchmark for compositional seatio-temporal reasoning. CVPR, 2021.
Jessica B Hamrick, Peter W Battaglia, Thomas L Griffiths, and Joshua Tenenbaum. Inferring mass in complex scenes by mental simulation. Cognition, 157:61176, 2016. 3
11

---

Published as a conference paper at ICLR 2022
Chi Han, Jiayuan Mao, Chuang Gan, Joshua B. Tenenbaum, and Jiajun Wu. Visual Concept
Metaconcept Learning. In NewrIPS, 2019. 3
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history 2d cnns and imagenet? In CVPR, 2018. 15
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 15
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In CVPR, 2017. 8, 14
Yining Hong, Li Yi, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan, Ptr: A benchmark for part-based conceptual, relational, and physical reasoning. In NeurlPS, 2021. 2
Yining Hong, Kaichun Mo, Li Yi, Guibas Leonidas, Joshua B Tenenbaum, Antonio Torralba, and Chuang Gan. Fixing malfunctional objects with learned physical simulation and functional prediction. In CVPR, 2022. 3
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. In ICLR, 2018.
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio- temporal reasoning in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. CVPR, 2017. 5,5
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018. 8
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations (ICLR), 2017.
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation
networks for video question answering. In CVPR, 2020. 6
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video question answering. In EMNLP, 2018. 3
Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. Tvqa+: Spatio-temporal grounding for video question answering. In Tech Report, arXiv, 2019. 3
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In International conference on machine learning, pp. 430-438. PMLR, 2016. 3
Qing Li, Siyuan Huang, Yining Hong, and Song-Chun Zhu. A competence-aware curriculum for visual concepts learning via question answering. In ECCV, 2020.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019a. 3
Yunzhu Li, Jiajun Wu, Jun- Yan Zhu, Joshua B Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. In ICRA, 2019b.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, W, 2017. 14
Nazneen Fatema Rajani, Rui Zhang. Yi Chern Tan, Stephan Zheng, Jeremy Weiss, Aadit Vyas, Abhijit Gupta, Caiming Xiong, Richard Socher, and Dragomir Radev. Esprit: explaining solutions to physical reasoning tasks. In ACI., 2020. 2, 3
12

---

Published as a conference paper at ICLR 2022
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer. Rob Fergus. Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. arXiv preprint arXiv:1803.07616. 2018. 2.3.4,5
Alvaro Sanzhez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Conference on Machine Learning. pp. 8459-8466 8468. PMLR, 2020.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurIPS, 2017.
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through ceasstionnanswernng. In Proceedings of the IEEE conference computer vision and pattern recognition, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurlPS, 2016."
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Neur/PS, 2021. 3
Jiajun Wu, Ilker Yildirim, Joseph Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. Advances in neural information processing systems, 28:1 18:15,,, 2015.
Saining Xie, Ross Girshick, Piotr Dollar, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017. 15
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In Advances in Neural Information Processing Systems (NIPS), 2018. 7,8
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020. 1, 2, 3, 4, 5, 7, 9, 14, 15
13

---

Published as a conference paper at ICLR 2022
A EXAMPLES FROM COMPHY
Here we provide more examples from ComPhy in Fig. 5 and Fig. 6. From these examples, we can see the following features of ComPhy. First, to answer the factual questions, models not only need to recognize objects' visual appearance attributes and events in the video but also identify their intrinsic physical properties from the given video set. Second, to answer counterfactual and predictive questions, models needs to predict objects' dynamics in counterfactual future scenes, which can be severely affected by intrinsic physical properties. We also show some typical question and choice samples as well as their underlying reasoning program logic in Fig. 7 and Fig. 8.
B VIDEO GENERATION
We provide more details for video generation. The generation of the videos in ComPhy can be decomposed into two steps. First, we adopt physical engine Bullet Coumans Bai (2016-2021) to simulate objects' motions and their interactions with each other. Since Bullet does not officially support the effect of electronic charges, we add external forces between charged objects, whose values are inversely proportional to the square of the objects' distance, to simulated Coulomb forces. We assign the light object with a mass value of and assign heavy object with a mass value of We manually make sure that each reference video at least contain interaction (collision, charge and mass) among objects to provide enough information for physical property inference. And each object should appear at least once in the reference videos. The simulated objects' motions sent to BlenderCommunity (2018) to render high-quality image sequences.
C QUESTION TEMPLATES
We show the question templates that have not been introduced before in table 4. As shown in the table, we can see that the new question templates have more symbolic operators related to the physical properties. Phrases like "heavy moving spheres" and "charged cubes" require models to infer objects' physical property values; phrases like " heavier than require models to compare the relative physical property values of two objects. For counterfactual questions, we have new conditions like "If the cyan object were unchargeds and "If the sphere were lighter". They aim to reason the dynamics under the hypothesis that certain object carrying different physical property. Such new features in language make ComPhy unique and challenging.
D CPL DETAILS
In this section, we provide more details for the proposed Compositional Physics Learner (CPL). Inspired by the NS-DR model in (Yi et al., 2020), CPL decomposes physical reasoning in ComPhy into four main components, visual perception, physical property learning, property based dynamic predictor and symbolic execution.
The symbolic execution component first adopts program parser parse the query question into functional program, containing a series of neural operations. The program parser is an attention-based seq2seq model Bahdanau et al. (2015), whose input the word sequence in the question/choice and output is the sequence of neural operations. The symbolic executor then executes the operations on the predicted dynamic scene get the answer to the question. We summarize all the symbolic operations in CPL in table 5. Compared with the previous benchmarks Yi et al. (2020); Ates et al. (2020), ComPhy has more operation on physical property identification, comparison and corresponding dynamic prediction.
We train the Mask-RCNY He et al. (2017) in the perception module with 4,000 frames randomly- selected from the training set of ComPhy. We train the program parser and the property concept learner with program and property labels using cross-entropy loss. We optimize the dynamic predictor with mean square error loss between the predicted objects' trajectories and the detected objects' trajectories by the perception module. We train the all the modules using Pytorch library Paszke et al. (2017) 17 Titan Nvidia GTX 1080-Ti GPUs.
14

---

Published as a conference paper at ICLR 2022
Question Type
Template and Example
If the _AA_ were _MP_. Q
Counterfact mass
If the sphere were lighter, which event would not happen?
If the _SA_ were _C__._Q_?
Counterfact charge
If the eyan object were uncharged, which event would happen?
What the _H_ of the _______ _SA__ that is _PA_?
Query
What is the color of the moving cylinder that is heavy?
Are there any LPA__DA________
Exist
Are there any charged stationary cube?
How many PA_ DA_ SA_ are there _I?
Count
How many heavy moving spheres are there when the video ends?
Is the DAI_ SAI_ heavier than the _A2__SA2_?
Mass compare
Is the blue sphere heavier than the gray cube?
Are the DAI SAI and the _DA2_ SA2_ oppositely charged?
Compare charge
Are the blue sphere and the purple sphere oppositely charged?
Are the DAI SA1 and the _D2___ SA2 with the same type of charge?
Compare charge 2
Are the blue cube and the brown cylinder with the same type of charge?
What are the Hs of the two objects that are charged? What are the colors of the two objects that are charged?
Query both
Table 4: Question templates and examples in ComPhy. _SA__ denotes static attributes like "red" and "sphere"; _DA__ denotes dynamic attributes, "moving" and "stationaa "stationary": __P__ __MP de- notes mass attributes, "1 "lighter" and "heavier"; Q denotes question phrases like "which of the fellowing would happen"; CP denotes charge attributes, "uncharged" and "opposited charged"; ___ denotes visible concepts, "color", "shape" and materiall _P__ denotes physical attributes, heavy, 1 light, charged and "uncharged"; _Tl_ denotes time indicators like "when the video ends".
E BASELINE IMPLEMENTATION DETAILS
In this section, we provide more details for baselines the experimental section. We implement baselines based on the publicly- source code. For multiple-choice questions, indepen- dently concatenate the words each option and the question as a binary classification question. Similar to CLEVRER (Yi et al., 2020), sequences for CNN+LSTM and MAC and variants with reference videos. We evenly sample 25 frames for each target video and 10 frames for each reference video. For HCRN, we use appearance feature from ResNet-101 (He et al., 2016) and motion feature from ResNetXt-101 (Xie et al., 2017; Hara et al., 2018) following the official implementation. For ALOE, we use MONet(Burgess et al., 2019) to extract visual representation and sample 25 frames for each target video. For ALOE (Ref). we sample 10 frames for each reference video and concatenate the reference frames and the target frames as visual representation. We train all the models until they are fully converged, select the best checkpoint on validation set and finally test on the testing set.
ResNet-50 (He et al., 2016) extract visual feature
15

---

Published as a conference paper at ICLR 2022
Type
Operation
Signature
Counter factual __s____sses__ss heavy Return all events after making the object heavy Counterfacts erfactual mass 1: ghht
(object) events
(object)
Counterfact Operation
Return all events after making the object light
Counter fraactual.. .ancaarged
(object)
Return all events after making the object uncharged Counter factual _opppopppiiiiieeeeeeee site charged
(object) events
Return all events after making the object oppositely charged
f tert_eeeay
(objects) objects
select all the heavy objects
filter_le berhigght select all the light objects
(objects) objects
Object Property Operations
i ter_ ttr_cha rged select all the churged objects
(objects) objects
f 1 te r_uncharged select all the uncharged objects
(objects) + objects
Object Appear- Operations
Filter wwt.ttttiii..........tt aattr
(objects, objects
Select objects from the input list with the input static attribute
Filter aetteerramaaa cintamic__aa attr
atgjectts,atr,fruainn objects
Selects objects in the input frame with the dynamic attribute
Fi Pilter_event
(events, events
Select all events that involve the input objects
pet_coo__partee_
(evect) - object
Return the collision partner of the input object
Filler_beeoor
(events, events) events
Event Operations
Select all events before the target event
Filter_afee
(evenzs, events) events
Select all events after the target event Fi Pilter_or ter__rrdrr Select the event at the specific time order Get frame
(events, order) + event
(event) + frame
Return the frame of the input event in the video
Others
Uni que
(events/ohjects) eventlobject
Return the only event/object in the input list
Start
Returns the special "start" event end
Returns the special "end" event Ol ject.s
Input Operations
objects
Returns all objects in the video Events
events
Returns all events happening in the video Inseer.Events
( - events
Returns all future events happening in the video
Query_ Serry_oohhhho booth_att ribute
(object,object) - atlr
Returns the attributes of the input two objects
Query _d1 rect on
(object.frame) attr
Returns the direction of the object at the imput frame
is_heavies
(objl, boul
Returns "yes" if obj/ / is heavier than obj2
Ts_ 1 gigtee
(obj1, + bool
Returns "yes" if obj/ is lighter than obj2
Output Operations
Query_ate aatbue
(object) - altr
Returns the attribute of the input objects like color Count
(objects) - int
Returns number the input objects/ events Exist
(events) - int
(objects) - bool
Returns "yes" if the input objects is not empty ong_to
(event, events) bool
Returns "yes" if the input event belongs the input
Negate
(bool) bool
Returns the negation of the input boolean
Table 5: Symbolic operations of CPL ComPhy. In this table, "order" denotes the chronological order of an event, e.g. "First" and "Last"; "static attribute" denotes object static concepts like "Red" and "Rubber" and "dynamic attribute" represents object dynamic concepts like "Moving".
16

---

Published as a conference paper at ICLR 2022
Reference video
time
Reference video
time
2
Reference video
time
Reference
time
Target video
time
Factual
Q1: the cyan cube heavier than the rubber cylinder? A: No.
Q2: Are there any blue cylinders that enter the scene? AeYes.
Counterfactual
Predictive
Q3: rubber cylinder lighter, which the following would happen?
Q4: What will happen next?
a) The rubber cylinder and the metal object collide
a) would collide with rubber eylinder
b) The rubber cylinder and the sphere collide
b) rubber cylinder sphere would collide v
<) The cube collides with the sphere x
<) object would collide with the sphere
Reference video
time
Reference video
time
Reference video 3
time
Reference video 4
time
Target
time
1. Factual
Q1: What the colors the objects that are charged? A1: Yellow and blue.
Q2: Are there metal cubes thut enter the scene? A2: No.
Q3: What is the direction of the blue cube when the video ends? A3: Left.
II. H.. Counterfactual
III. Predictive
Q3: If the blue sphere were oppositely charged. what would happen?
Q4: Which will happen next?
The blue cube the red collide v blue sphere collides with the metal cube
The yellow sphere and the rubber cube would collide vf
The yellow object and blue sphere would collide v
The blue cube and the metal cube would collide
The yellow object and the red object would collide
Figure Sample target video, reference videos and question-answe aassirr pairs from ComPhy.
17

---

Published as a conference paper at ICLR 2022
Reference video 1
time
Reference video 2
time
Reference video 3
Reference video
time
Target video
time
Factual
Q1: What color is the moving rubber object that is uncharged? A1: Rubber.
02: In which direction is the red cylinder moving when the yellow object enters the scene? A2: Left.
Q3: How muny moving cyan objects ure charged? A3:
II. Counterfactual
Q3: If the cyan object uncharged. which event would happen?
Q4: If the sphere oppositely charged, which of the following would happen?
a) The sphere and the eyan object would collide v cylinder would collide with the sphere / The cylinder would collide with the gray cube
The cylinder sphere would The cylinder would collide cyan cube
The sphere would collide with the yellow object x
Reference video
time
Reference video 2
time
Reference video
time
Reference video 4
time
Target video
1. Factual
Q1: Is the moving purple object lighter than the moving brown object? A1: No.
Q2: What is the color of the moving metal object that is light? A2: Red.
Q3: Are there any brown eylinders that scene after the sphere enters scene? A3: Yes.
II. Counterfactual
III. Predictive
Q3: If the sphere were lighter. which event would not happen?
Q4: Which event will happen next?
sphere collides with cube /
a) The red cylinder and the cube would collide
b) The metal cylinder and the brown cylinder collide
b) The sphere and the cube would collide /f
c) The red ey linder and the sphere would collide x d) The metal cylin linder and the rubber ey linder would
collidex
Figure 6: Sample target video, reference videos and question-answer pairs from ComPhy.
18

---

Published as a conference paper at ICLR 2022
Q1: How many heavy stutionary objects are there when the
Q2: What color is the moving rubber object that is unchurged?
video begins?
Filter
Filter Stationary
Filter Material
Filter
Filter Charge
Query Color
Objects
Count
Objects
mass
Moving
heavy
Rubber
Uncharged
Filter start
Get
Events
Frame
Q3: How many moving green objects are charged?
Q4: What shape is the moving metal object that is light?
Filter
Filter Moving
Filter
Filter
Filter
Filter
Query
Objects
Count
Objects
color
charge
Muterial
Moving
Shupe
Green
charged
Light
Metal
Figure 7: Sample of factual questions and their underlying functional programs in ComPhy.
Q1: the rubber cylinder heavier, which of the following would happen?
Which of following would happen if sphere uncharged?
Filter Material
Filter Shape
Counterfact
belong
Counterfact
belong
Objects
Objects
Not
Shape
Charge
to
Choice
Choice
Cylinder
Heavy
Sphere
Uncharged
Program
Program
C1: The cylinder the cube would collide
The blue object and metal object would collide
Filter Collision
Filter Collision
belong
Filter Collision
Filter
belong
Events
Events
Cylinder
Filter
Filter Shape
Cube
Blue
Filter
Filter Material
Metal
Shape
Color
Question
Question
Objects
Objects
Objects
Objects
Program
Program
Figure 8: Sample of counterfactual questions, choice options and their underlying functional programs ComPhy.
19


Reviews:
[{"review_id": "f6uYFVuJ99", "reviewer": "Reviewer_3cQE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is structured identically to CLEVRER (Yi et al., ICLR 2020): Figures 1 and 2 contains examples of scenes and questions from the dataset. Table 1 uses check marks to highlight features of existing datasets. Figure 3 shows the frequency of question types in a pie chart. Figure 4 presents sample programs representing the questions. Table 2 compares baseline methods (with almost the same column headers as the corresponding table in CLEVRER). And so on.\n\nBeyond the writing, the models presented (CPL versus NS-DR) are also broadly the same. The similarity of the two papers raises the concerning possibility of plagiarism, or more likely, a marginal contribution sold anew in recycled packaging. Nevertheless, I will attempt to assess the work on its own merits.\n\n**Strengths**:\n- the code is public\n- human baseline scores\n- some attempt to avoid mistakes in past datasets (e.g. the exclusion of counterfactual questions on objects which have no interaction with other objects)\n\n**Weaknesses**:\n- Dataset: there's no innovation in the type or structure of questions asked. The generation makes arbitrary choices e.g. a video can contain a pair of charged objects (or none), or up to one heavy object. The visuals themselves are highly similar to prior work (borrowing originally from CLEVR), contributing to collective overfitting over time.\n- The model: neither the CPL framework nor the implementation of any module is novel. It is entirely supervised, and exhibits little generalization.\n- Writing: \n  - there's several instances of overclaiming (e.g. \"we compare the mass and charge edge label prediction result with the ground-truth labels and find that they achieve an accuracy of 90.4% and 90.8%. This shows the effectiveness of CPL for property identification.\" Given everything is trained with supervision, these prediction accuracies are no surprise.) \n  - the paper is disorganized (e.g. human baselines are presented inline but not in Table 2, the generalization results in Figure 6 use a different method than in Table 3, etc). Several parts are unclear (e.g. \"To run NS-DR successfully in ComPhy, we provide NS-DR with extra ground-truth physical property labels. The variant, NS-DR+ uses the PropNet (Li et al., 2019b) for dynamics predictions, which does not consider the mass and charge information of the objects.\" This sounds contradictory to me.)\n  - it is also sparsely detailed (e.g. how exactly was the human data collected? Are the modules trained end to end or one by one?).\n- Empirical results: \n  - despite the numerous baselines in the paper, some important ones are missing. For instance: how would an oracle model perform if it understood all object properties but had no inkling of the latent properties (mass and charge)? That would help reveal the significance of latent properties over the remaining properties.\n  - there's no error bars for any of the results.\n\nMore philosophically: do we actually need models which can infer/understand physical properties like charge from visual observation? Or models which can figure out–to use the example in the paper–why \"apples float in water while bananas sink?\" It doesn't seem to me that this kind of knowledge emerges from direct observation or experience in humans.", "summary_of_the_paper": "This work extends prior datasets (chiefly, CLEVRER) for physical reasoning from visual input. It endows objects with two latent properties–mass and charge–thereby producing object dynamics like attraction or repulsion. The paper also presents VQA results on the dataset from a modular architecture (similar to NS-DR from CLEVRER) trained by supervision.", "summary_of_the_review": "Both the presented dataset and model framework are minor tweaks of existing work. There's no novel insight. The dataset will not enable any new research directions. The paper also needs more work. I can't recommend accepting at all.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "cJ_T6Wx_xBn", "reviewer": "Reviewer_VByS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is overall well written and executed. With a focus on inferring hidden attributes, this work tackles a crucial aspect of visual reasoning. Experiments are well-designed to illustrate the claimed contributions.\n\nMy main concerns are about the complexity that the proposed method can handle. Overall, the method is only demonstrated on a simple synthetic dataset with two attributes (mass and charge), while mass only has two values (heavy vs light). Despite it does demonstrate the claimed contributions, to what extent the proposed method would work and generalize is unknown.\n\nAdditionally, it seems that all objects in a video in the datasets are of interest; there are no irrelevant objects. Would this become a problem for the algorithm?\n\nFor a visual reasoning task, the proposed method seems to struggle in the following scenario: If A and B are attracted, and B and C are attracted, A and C should be repelled.", "summary_of_the_paper": "This paper proposes a dataset and a method to study the hidden properties of objects present in videos. The method can extract object attributes and use them for predicting subsequent scenes, with the additional capability to handle counterfactual scenarios.", "summary_of_the_review": "This is a good work that fills the gap of visual reasoning of hidden attributes, but experiments could be more comprehensive.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QlCzOpkszKk", "reviewer": "Reviewer_DJEq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\n- The paper explores a quite interesting direction. Reasoning about the physical properties of objects is a critical requirement for designing agents that interact with the world around them.\n\n- The proposed CPL approach outperforms a number of baseline approaches.\n\n**Weaknesses**\n\nThe paper has some shortcomings:\n\n(1) The work is incremental compared to Yi et al., 2020. The only difference is that this paper infers hidden properties instead of collisions. The questions and the way they are generated are very similar as well.\n\n(2) The dataset is not comprehensive enough. Only two properties are considered: mass and electricity charge. I don't think electricity charge is a useful property to infer since it can be used only in a limited set of applications. The dataset should include useful properties such as friction and material. Moreover, the synthetic videos are so simplistic (cubes and spheres moving on a plane). It is not clear if a system that solves this task can be useful in any scenario that is slightly more complex in terms of appearance and object interactions. I encourage the authors to create more realistic scenarios in simulators such as AI2-THOR and iGibson.\n\n(3) It is not clear how the train/val/test splits are created. It doesn't seem there is much variability in the videos so there might be a high degree of similarity between train and test scenarios. Some metrics should be reported to show train and test splits are different enough.\n\n**Questions and comments**\n\n- \"CNN-LSTM (Antol et al., 2015) is a basic video question-answering model.\": This paper is not a video question-answering model. It just uses single frames.\n\n- \"We think the reason is that these models are based on massive training videos and question-answer pairs and have difficulties adapting to the new scenario in ComPhy\": It might be simply due to the large networks and small amount of data.\n\n- There is an assumption that all objects are already known and (it seems) there is only one object per each category. It would be nice to see the results of generalization to unseen shapes. However, it is not clear how the graph can accommodate new objects. It seems it is a fixed graph.\n\n- Equation 2 is not clear. What is g_emb and g_enc? What does z represent?\n\n- It is not clear why CPL shows better performance on counterfactual questions. There is no component in the model to better handle that type of question.\n\n- Why is CPL called oracle?", "summary_of_the_paper": "The paper focuses on inferring two hidden physical properties (mass and electricity charge). A synthetic dataset, called ComPhy is created. The dataset consists of simple shapes (such as cubes and sphere) that move on a flat plane. For each query video, there are four reference videos that the models can use to learn the properties of objects. The dataset also includes three types of questions: factual, counterfactual and predictive. The paper also presents a pipeline approach called CPL that leverages graphs to model the relationship of objects and solve the QA tasks.", "summary_of_the_review": "The reasons for my low rating are: (1) lack of novelty with respect to previous works, (2) dataset not being comprehensive enough, (3) simplicity of the dataset and lack of generalization to slightly richer scenarios in terms of appearance and object variability (4) lack of clarity on creation of the dataset. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PXMwxxyGyfL", "reviewer": "Reviewer_8BUA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths:\n\n1. The task definition, formulation, and the proposed dataset Comphy are novel and thoughtful. Models working on other physics-based (such as Phyre, Cleverer, Cophy) datasets may or may not work on this dataset (Comphy).\n\n2. Multiple baselines are tested and the proposed model CPL shows significant gain over these.\n\n### Weakness:\n\n1. On Motivation: The paper doesn't motivate the problem, and I am struggling to see any direct application. Could the authors provide any examples where such a model would be useful?\n\n2. (Minor): In Table 1., the authors suggest CoPhy and Cleverer do not support counterfactual, but to my understanding they do? I could be missing something here.\n\n3. On Task and dataset setup:\n\n(i) For intrinsic properties, the authors chose mass and charge suggesting that inferring the two is challenging. However, this is not explicitly shown in experiments. Authors should show that just finding mass or just finding charge from reference videos is doable, but having both in properties together makes the task difficult (I believe this to be the case intuitively, but an explicit expt would be helpful).\n\n(ii) The reason for choosing 4-videos as reference is unclear as compared to directly providing the values as input. For instance, the input could itself be the target video + properties of the objects, followed by multiple questions. \n\n(iii) How is number \"4\" for reference videos chosen? Does the task become simpler for a particular target video if more reference videos are added (say 10 or 1000)?\n\n(iv) In the current setup, each target video has 4-reference videos, followed by a number of questions. A simple extension would be to use the same 4-reference video, but use a different target (same objects, same properties but different target videos). This could suggest if the model is is able to answer some questions by chance or did it actually infer the physical properties.\n\n4. On Model and Experiments:\n\n(i) Intermediate results: Baselines showing results with ground-truth object properties should be reported. For instance, NS-DR and CPL in Table2., but  they are given the ground-truth heavy/light information as well as the corresponding charge. (This is related to point 3.(ii) ).\n\n(ii) The task of deriving object properties from the 4-reference video is independent of the questions asked on target video. As such, the authors should report separate metrics for just this part. This would show if future research should focus on deriving physical properties or given physical properties work on the counterfactual/predictive types of question.\n\n(iii) The physical property inference is (to the best of my understanding) is applied on 4 reference videos + target video. This setup of being dependent on target video seems confusing to me. As such a model which has seen 4 reference videos should be able to work on a completely new target video as well (see related point 3.(iv)) Could the authors show results without using target video?", "summary_of_the_paper": "The authors propose a new task and corresponding dataset ComPhy, a video question-answering dataset for evaluating video reasoning capability. First, a model is given 4 reference videos (2-sec clips showing object interactions in isolation) which is used to deduce object properties. Then a new query video (which could have more than 2 objects) is given on which 3 types of questions are asked: factual (about what happens in the video), counterfactual (what would happen if some condition changed), and predictive (what would happen after), with later two being multiple-choice questions. A particular contribution is the focus is on latent properties of the objects (mass and charge) which needs to be deduced from the reference videos. Multiple video reasoning model baselines are reported along with a newly proposed oracle Composition Physics Learner (CPL) model which shows considerable improvement over the baselines.\n\n", "summary_of_the_review": "The authors provide a novel and well-thought task and corresponding dataset ComPhy which requires a model to use 4-reference videos to infer intrinsic properties of the objects and thereby apply these findings to a target video and answer corresponding factual/predictive and counterfactual questions. However, in its current form, the exact motivation of the paper is unclear, the reliance on using 4-reference videos compared to simply providing the information is not justified, and some dataset choice such as using only target video for given reference videos are not clear. More intermediate results such as finding object properties would make the paper stronger.\n\n---\nPost Rebuttal: The authors have substantially updated their paper with additional ablative studies and human evaluation results. In my opinion, this substantially improves the findings and takeaways of the paper. More experiments with physical properties in isolation, showing the effect of increasing the number of reference videos, and de-coupling the reference and target videos could further strengthen the paper.\n\nAs such, I am increasing my score to 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new dataset called ComPhy to evaluate the ability of models to infer physical properties of objects and to reason about their interactions given these physical properties. The paper also presents an oracle model (named oracle because it requires gold property labels at training time) that is modular and carefully hand designed, but shows considerable improvement over a series of baselines. The reviewers for this submission had several concerns including:
(a) [VByS] "concerns are about the complexity that the proposed method can handle"\
(b) [VByS] "the method is only demonstrated on a simple synthetic dataset"\
(c) [8BUA] "I am struggling to see any direct application"\
(d) [8BUA] "choosing 4-videos as reference" -- why use ref videos, why use 4\
(e) [8BUA] "Baselines showing results with ground-truth object properties should be reported"\
(f) [3cQE] "no innovation in the type or structure of questions asked"\
(g) [3cQE] "neither the CPL framework nor the implementation of any module is novel"\
(h) [DJEq] "The only difference is that this paper infers hidden properties instead of collisions"\
(i) [DJEq] "The dataset is not comprehensive enough" -- only 2 properties and simplistic and synthetic videos\

The authors have provided detailed responses to these concerns and I discuss these below.

The authors have addressed (c),(d) and (e) well in their rebuttal.

I don't think (a) is concerning. The proposed model is not expected to solve the dataset entirely inspite of having access to gold properties at training time. As the authors mention, this indicates the complexity of the task at hand.

The authors also address (f) well. I dont think there is any need for innovation in the structure of questions asked. QA is merely a mechanism to probe the model, and using CLVERER style questions seems appropriate.

I disagree with the sentiment behind (g). The proposed oracle model clearly inherits modules from past works and assembles them to suit the needs of the dataset. It is this assembly that differentiates it from past works. This is true of most papers in our field, including ones that are widely acknowledged to be important papers. The underlying modules in proposed networks are rarely novel, but their assembly can lead to improvements on benchmarks. Furthermore, the oracle model, isnt the central contribution of this work. The dataset is, and hence, the requirement for novelty is reduced. The oracle is meant to serve as a guideline to show what one may achieve given gold labels at training, and it serves that purpose well.

Re (h), my takeaway is that inferring properties based on their dynamics and without any link to their appearance is an important step, and past datasets do not exhibit this characteristic. And thus, in spite of being a limited differentiation from CLEVERER, I think this is interesting.

Re: (b) and (i) I do agree with some aspects of these, with the reviewers.
I think its still valuable to have a dataset with synthetic videos, given that models today are unable to solve this dataset. Moving to more realistic videos is a next step.
However, as the reviewer [DJEq] points out, it would be desirable to add more physical properties and add more complex scene elements like ramps. That would have added a lot more diversity to the dataset -- visually, with regards to physical properties and with regards to the types of reasoning required.

Having said that, I believe that the dataset in its present form is still valuable to the community, and hence I recommend acceptance.
I think adding more physical properties and scene elements will have made this a much stronger submission.
Decision: Accept (Poster)