ComPhy: Compositional Physical Reasoning of Objects and Events from Videos

1 INTRODUCTION . Why do apples float in water while bananas sink ? Why do magnets attract each other on a certain side and repel on the other ? Objects in nature often exhibit complex properties that define how they interact with the physical world . To humans , the unraveling of new intrinsic physical properties often marks important milestones towards a deeper and more accurate understanding of nature . Most of these properties are intrinsic as they are not directly reflected in the object ’ s visual appearances or otherwise detectable without imposing an interaction . Moreover , these properties affect object motion in a compositional fashion , and the causal dependency and mathematical law between different properties are often complex . As shown in Fig . 1 , different intrinsic physical properties , such as charge and inertia , often lead to drastically different future evolutions . Objects carrying the same or opposite charge will exert a repulsive or attractive force on each other . The resulting motion not only depends on the amount of charge each object carries , but also their signs ( see Fig . 1- ( a ) ) . The inertia determines how sensitive an object ’ s motion is to external forces . When a massive object interacts with a light object via attraction , repulsion , or collision , the lighter object will undergo larger changes in its motion compared with the massive object ’ s trajectory ( see Fig . 1- ( b ) ) . Recent studies have established a series of benchmarks to evaluate and diagnose machine learning systems in various physics-related environments ( Bakhtin et al. , 2019 ; Yi et al. , 2020 ; Baradel et al. , 2020 ) . These benchmarks introduce reasoning tasks over a wide range of complex object motion and interactions , which poses enormous challenges to existing models since most tasks require them to fully capture the underlying physical dynamics and , in some cases , be able to make predictions . However , the majority of complexity in the motion trajectories facilitated by these environments 1Project page : https : //comphyreasoning.github.io comes from changes or interventions in the initial conditions of the physical experiments . The effects of object intrinsic physical properties , as well as the unique set of their challenges , are therefore of great importance for further investigation . It ’ s non-trivial to build a benchmark for compositional physical reasoning . Existing benchmarks ( Yi et al. , 2020 ; Ates et al. , 2020 ) assume that there is no variance in objects ’ physical properties and ask models to learn physical reasoning from massive videos and question-answer pairs . A straightforward solution is to correlate object appearance with physical properties like making all red spheres to be heavy and then ask questions about their dynamics . However , such a design may incur shortcuts for models by just memorizing the appearance prior rather than understanding coupled physical properties . In this work , we propose a novel benchmark called ComPhy that focuses on understanding object-centric and relational physics properties hidden from visual appearances . ComPhy first provides few video examples with dynamic interactions among objects for models to identify objects ’ physical properties and then asks questions about the physical properties and corresponding dynamics . As shown in Fig . 2 , ComPhy consists of meta-train sets and meta-test sets , where each data point contains 4 reference videos and 1 target video . Within each set , the objects share the same intrinsic physical properties across all videos . Reasoning on ComPhy requires the model to infer the intrinsic and compositional physical properties of the object set from the reference videos , and then answer questions about this query video . To make the task feasible , we systematically control each object in the query video that should appear at least in one of the reference videos . We also introduce an oracle model to tackle this task . Inspired by recent work on neural-symbolic reasoning on images and videos ( Yi et al. , 2018 ; 2020 ; Chen et al. , 2021 ) , our model consists of four disentangled components : perception , physical property learning , dynamics prediction , and symbolic reasoning . Our model is able to infer objects ’ compositional and intrinsic physical properties , predict their future , make counterfactual imaginations , and answer questions . To summarize , this paper makes the following contributions . First , we present a new physical reasoning benchmark ComPhy with physical properties ( mass and charge ) , physical events ( attraction , repulsion ) , and their composition with visual appearance and motions . Second , we decorrelate physical properties and visual appearance with a few-shot reasoning setting . It requires models to infer hidden physical properties from only a few examples and then make predictions about the system ’ s evolution to help answer the questions . Third , we propose an oracle neural-symbolic framework , which is a modularized model that can infer objects ’ physical properties and predict the objects ’ movements . At the core of our model are graph neural networks that capture the compositional nature of the underlying system . 2 RELATED WORK . Physical Reasoning . Our work is closely related to recent developments in physical reasoning benchmarks ( Riochet et al. , 2018 ; Girdhar & Ramanan , 2020 ; Ates et al. , 2020 ) . PHYRE ( Bakhtin et al. , 2019 ) and its variant ESPRIT ( Rajani et al. , 2020 ) defines an environment where objects can move within a vertical 2D plane under gravity . Each task is associated with a goal state , and the model solves the task by specifying the initial condition that will lead to the goal state . CLEVRER ( Yi et al. , 2020 ) contains videos of multiple objects moving and colliding on a flat plane , posting natural language questions about description , explanation , prediction , and counterfactual reasoning on the collision events . CoPhy ( Baradel et al. , 2020 ) includes experiment trials of objects moving in 3D space under gravity . The task focuses on predicting object trajectories under counterfactual interventions on the initial conditions . Our dataset contributes to this line of physical reasoning benchmarks by focusing on physical events driven by object intrinsic properties ( situations shown in Fig . 2 ) . ComPhy requires models to identify intrinsic properties from only a few video examples and make dynamic predictions based on the identified properties and their compositionality . Dynamics Modeling . Dynamics modeling of physical systems has been a long-standing research direction . Some researchers have studied this problem via physical simulations , drawing inference on the important system- and object-level properties via statistical approaches such as MCMC ( Battaglia et al. , 2013 ; Hamrick et al. , 2016 ; Wu et al. , 2015 ) , while others propose to directly learn the forward dynamics via neural networks ( Lerer et al. , 2016 ) . Graph neural networks ( Kipf & Welling , 2017 ) , due to their object- and relation-centric inductive biases and efficiency , have been widely applied in forwarding dynamics prediction on a wide variety of systems ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Sanchez-Gonzalez et al. , 2020 ; Li et al. , 2019a ) . Our work combines the best of the two approaches by first inferring the object-centric intrinsic physical properties and then predicting their dynamics based on the intrinsic properties . Video Question Answering . Our work is also related to answering questions about visual content . Various benchmarks have been proposed to handle the task of video question answering , e.g . MarioQA ( Mun et al. , 2017 ) , TVQA ( Lei et al. , 2018 ) and AGQA ( Grunde-McLaughlin et al. , 2021 ) . However , they mainly focus on understanding human actions and activities rather than learning physical events and properties , which is essential for robot planning and control . Following CLEVRER , we summarize the difference between ComPhy and the previous related benchmarks in Table 1 . ComPhy is the only dataset that requires the model to learn physical property from few video examples , make dynamic predictions based on the physical property , and finally answer corresponding questions . Few-shot Learning . Our work is also related to few-shot learning , which typically learns to classify images from only a few labelled examples ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Han et al. , 2019 ) . ComPhy also requires models to identify objects ’ property labels from only a few video examples . Different from them , reference videos have no labels for objects ’ physical properties but more interaction among objects , providing more information for models to identify objects ’ physical properties . 3 DATASET . ComPhy studies objects ’ intrinsic physical properties from objects ’ interactions and how these properties affect their motions in future and counterfactual scenes to answer corresponding questions . We first introduce videos and task setup in Section 3.1 . We then discuss question types in Section 3.2 , and statistics and balancing in Section 3.3 . 3.1 VIDEOS . Objects and Events . Following Johnson et al . ( 2017 ) , objects in ComPhy contain compositional appearance attributes like color , shape , and material . Each object in videos can be uniquely identified by these three attributes for simplicity . There are events , in , out , collision , attraction and repulsion . These object appearance attributes and events form the basic concepts of the questions in ComPhy . Physical Property . Previous benchmarks ( Riochet et al. , 2018 ; Yi et al. , 2020 ) mainly study appearance concepts like color and collision that can be perceived in even a single frame . In ComPhy , we additionally study intrinsic physical properties , mass and charge , which can not be directly captured from objects ’ static appearance . As shown in Fig . 1 ( a ) , objects with same or opposite charge will repel or attract each other while objects without charge will not affect each other ’ s motion without collision . As shown Fig . 1 ( b ) , the object with larger mass ( inertia ) tends to maintain its original moving direction after the collision while the light object changes much more in its moving direction . Note that these intrinsic physical properties are orthogonal to the appearance attributes and can be combined with each other to generate more complicated and diverse dynamic scenes . For simplicity , ComPhy contains two mass values ( heavy and light ) and three charge types ( positive charged , negative charged and uncharged ) . Theoretically , we can add more physical properties like bounciness coefficients and friction into ComPhy and make their values continuous . However , such a design will make the dataset too complicated and even difficult for people to infer the properties . Video Generations . For each video , we first use a physical engine ( Coumans & Bai , 2016–2021 ) to simulate objects ’ motions and then adopt a graphs engine ( Community , 2018 ) to render frame sequences . Each target video for question answering contains 3 to 5 objects with random composition between their appearance attributes and physical properties . We set the length of the target video to be 5 seconds and additionally simulate the 6-th and 7-th seconds of the target video for predictive question annotation . We provide more video generation details in the Appendix . Task Setup . It is not trivial to design a task setup to evaluate models ’ ability for physical reasoning since physical properties are not observable in a static frame . A straightforward design is to correlate object appearance with the physical property like “ red object is heavy ” , “ yellow object is light ” and then ask “ what would happen if they collide ” . However , such a setting is imperfect since it can not evaluate whether a model really understands the physical properties or just memorize the visual appearance prior . An ideal setting should be able to evaluate whether a model is like a human that can identify objects ’ properties from objects ’ motion and interactions with each other in the dynamic scenes and make the corresponding dynamic predictions . To achieve this , we design a meta setting for physical reasoning , which provides few reference video samples along with the target video for models to infer objects ’ physical properties and then ask questions about the objects ’ physical properties and dynamics . Following CLEVRER , we show a sample of the dataset in Fig . 2 . Each set contains a target video , 4 reference videos , and some questions about the visual attributes , physical properties , events , and dynamics of the target video . Objects in each set share the same visual attributes ( color , shape , and material ) and intrinsic physical property ( mass and charge ) . Reference videos . To provide abundant visual content for physical property inference , we additionally provide 4 reference videos for each target video . We sample 2-3 objects from the target video , provide them with different initial velocities and locations , and make them interact ( attract , repel or collide ) with each other . The generation of the reference video follows the same standard as the target video , but the length of the videos is set to 2 seconds for scaling up . The interaction among objects in reference videos helps models to inference objects ’ properties . For example , the repulsion in Reference video 1 of Fig . 2 can help us identify that object 1 and object 2 carrying the same charge .

Summary: The authors propose a new task and corresponding dataset ComPhy, a video question-answering dataset for evaluating video reasoning capability. First, a model is given 4 reference videos (2-sec clips showing object interactions in isolation) which is used to deduce object properties. Then a new query video (which could have more than 2 objects) is given on which 3 types of questions are asked: factual (about what happens in the video), counterfactual (what would happen if some condition changed), and predictive (what would happen after), with later two being multiple-choice questions. A particular contribution is the focus is on latent properties of the objects (mass and charge) which needs to be deduced from the reference videos. Multiple video reasoning model baselines are reported along with a newly proposed oracle Composition Physics Learner (CPL) model which shows considerable improvement over the baselines.