ComPhy: Compositional Physical Reasoning of Objects and Events from Videos

1 INTRODUCTION . Why do apples float in water while bananas sink ? Why do magnets attract each other on a certain side and repel on the other ? Objects in nature often exhibit complex properties that define how they interact with the physical world . To humans , the unraveling of new intrinsic physical properties often marks important milestones towards a deeper and more accurate understanding of nature . Most of these properties are intrinsic as they are not directly reflected in the object ’ s visual appearances or otherwise detectable without imposing an interaction . Moreover , these properties affect object motion in a compositional fashion , and the causal dependency and mathematical law between different properties are often complex . As shown in Fig . 1 , different intrinsic physical properties , such as charge and inertia , often lead to drastically different future evolutions . Objects carrying the same or opposite charge will exert a repulsive or attractive force on each other . The resulting motion not only depends on the amount of charge each object carries , but also their signs ( see Fig . 1- ( a ) ) . The inertia determines how sensitive an object ’ s motion is to external forces . When a massive object interacts with a light object via attraction , repulsion , or collision , the lighter object will undergo larger changes in its motion compared with the massive object ’ s trajectory ( see Fig . 1- ( b ) ) . Recent studies have established a series of benchmarks to evaluate and diagnose machine learning systems in various physics-related environments ( Bakhtin et al. , 2019 ; Yi et al. , 2020 ; Baradel et al. , 2020 ) . These benchmarks introduce reasoning tasks over a wide range of complex object motion and interactions , which poses enormous challenges to existing models since most tasks require them to fully capture the underlying physical dynamics and , in some cases , be able to make predictions . However , the majority of complexity in the motion trajectories facilitated by these environments 1Project page : https : //comphyreasoning.github.io comes from changes or interventions in the initial conditions of the physical experiments . The effects of object intrinsic physical properties , as well as the unique set of their challenges , are therefore of great importance for further investigation . It ’ s non-trivial to build a benchmark for compositional physical reasoning . Existing benchmarks ( Yi et al. , 2020 ; Ates et al. , 2020 ) assume that there is no variance in objects ’ physical properties and ask models to learn physical reasoning from massive videos and question-answer pairs . A straightforward solution is to correlate object appearance with physical properties like making all red spheres to be heavy and then ask questions about their dynamics . However , such a design may incur shortcuts for models by just memorizing the appearance prior rather than understanding coupled physical properties . In this work , we propose a novel benchmark called ComPhy that focuses on understanding object-centric and relational physics properties hidden from visual appearances . ComPhy first provides few video examples with dynamic interactions among objects for models to identify objects ’ physical properties and then asks questions about the physical properties and corresponding dynamics . As shown in Fig . 2 , ComPhy consists of meta-train sets and meta-test sets , where each data point contains 4 reference videos and 1 target video . Within each set , the objects share the same intrinsic physical properties across all videos . Reasoning on ComPhy requires the model to infer the intrinsic and compositional physical properties of the object set from the reference videos , and then answer questions about this query video . To make the task feasible , we systematically control each object in the query video that should appear at least in one of the reference videos . We also introduce an oracle model to tackle this task . Inspired by recent work on neural-symbolic reasoning on images and videos ( Yi et al. , 2018 ; 2020 ; Chen et al. , 2021 ) , our model consists of four disentangled components : perception , physical property learning , dynamics prediction , and symbolic reasoning . Our model is able to infer objects ’ compositional and intrinsic physical properties , predict their future , make counterfactual imaginations , and answer questions . To summarize , this paper makes the following contributions . First , we present a new physical reasoning benchmark ComPhy with physical properties ( mass and charge ) , physical events ( attraction , repulsion ) , and their composition with visual appearance and motions . Second , we decorrelate physical properties and visual appearance with a few-shot reasoning setting . It requires models to infer hidden physical properties from only a few examples and then make predictions about the system ’ s evolution to help answer the questions . Third , we propose an oracle neural-symbolic framework , which is a modularized model that can infer objects ’ physical properties and predict the objects ’ movements . At the core of our model are graph neural networks that capture the compositional nature of the underlying system . 2 RELATED WORK . Physical Reasoning . Our work is closely related to recent developments in physical reasoning benchmarks ( Riochet et al. , 2018 ; Girdhar & Ramanan , 2020 ; Ates et al. , 2020 ) . PHYRE ( Bakhtin et al. , 2019 ) and its variant ESPRIT ( Rajani et al. , 2020 ) defines an environment where objects can move within a vertical 2D plane under gravity . Each task is associated with a goal state , and the model solves the task by specifying the initial condition that will lead to the goal state . CLEVRER ( Yi et al. , 2020 ) contains videos of multiple objects moving and colliding on a flat plane , posting natural language questions about description , explanation , prediction , and counterfactual reasoning on the collision events . CoPhy ( Baradel et al. , 2020 ) includes experiment trials of objects moving in 3D space under gravity . The task focuses on predicting object trajectories under counterfactual interventions on the initial conditions . Our dataset contributes to this line of physical reasoning benchmarks by focusing on physical events driven by object intrinsic properties ( situations shown in Fig . 2 ) . ComPhy requires models to identify intrinsic properties from only a few video examples and make dynamic predictions based on the identified properties and their compositionality . Dynamics Modeling . Dynamics modeling of physical systems has been a long-standing research direction . Some researchers have studied this problem via physical simulations , drawing inference on the important system- and object-level properties via statistical approaches such as MCMC ( Battaglia et al. , 2013 ; Hamrick et al. , 2016 ; Wu et al. , 2015 ) , while others propose to directly learn the forward dynamics via neural networks ( Lerer et al. , 2016 ) . Graph neural networks ( Kipf & Welling , 2017 ) , due to their object- and relation-centric inductive biases and efficiency , have been widely applied in forwarding dynamics prediction on a wide variety of systems ( Battaglia et al. , 2016 ; Chang et al. , 2016 ; Sanchez-Gonzalez et al. , 2020 ; Li et al. , 2019a ) . Our work combines the best of the two approaches by first inferring the object-centric intrinsic physical properties and then predicting their dynamics based on the intrinsic properties . Video Question Answering . Our work is also related to answering questions about visual content . Various benchmarks have been proposed to handle the task of video question answering , e.g . MarioQA ( Mun et al. , 2017 ) , TVQA ( Lei et al. , 2018 ) and AGQA ( Grunde-McLaughlin et al. , 2021 ) . However , they mainly focus on understanding human actions and activities rather than learning physical events and properties , which is essential for robot planning and control . Following CLEVRER , we summarize the difference between ComPhy and the previous related benchmarks in Table 1 . ComPhy is the only dataset that requires the model to learn physical property from few video examples , make dynamic predictions based on the physical property , and finally answer corresponding questions . Few-shot Learning . Our work is also related to few-shot learning , which typically learns to classify images from only a few labelled examples ( Vinyals et al. , 2016 ; Snell et al. , 2017 ; Sung et al. , 2018 ; Han et al. , 2019 ) . ComPhy also requires models to identify objects ’ property labels from only a few video examples . Different from them , reference videos have no labels for objects ’ physical properties but more interaction among objects , providing more information for models to identify objects ’ physical properties . 3 DATASET . ComPhy studies objects ’ intrinsic physical properties from objects ’ interactions and how these properties affect their motions in future and counterfactual scenes to answer corresponding questions . We first introduce videos and task setup in Section 3.1 . We then discuss question types in Section 3.2 , and statistics and balancing in Section 3.3 . 3.1 VIDEOS . Objects and Events . Following Johnson et al . ( 2017 ) , objects in ComPhy contain compositional appearance attributes like color , shape , and material . Each object in videos can be uniquely identified by these three attributes for simplicity . There are events , in , out , collision , attraction and repulsion . These object appearance attributes and events form the basic concepts of the questions in ComPhy . Physical Property . Previous benchmarks ( Riochet et al. , 2018 ; Yi et al. , 2020 ) mainly study appearance concepts like color and collision that can be perceived in even a single frame . In ComPhy , we additionally study intrinsic physical properties , mass and charge , which can not be directly captured from objects ’ static appearance . As shown in Fig . 1 ( a ) , objects with same or opposite charge will repel or attract each other while objects without charge will not affect each other ’ s motion without collision . As shown Fig . 1 ( b ) , the object with larger mass ( inertia ) tends to maintain its original moving direction after the collision while the light object changes much more in its moving direction . Note that these intrinsic physical properties are orthogonal to the appearance attributes and can be combined with each other to generate more complicated and diverse dynamic scenes . For simplicity , ComPhy contains two mass values ( heavy and light ) and three charge types ( positive charged , negative charged and uncharged ) . Theoretically , we can add more physical properties like bounciness coefficients and friction into ComPhy and make their values continuous . However , such a design will make the dataset too complicated and even difficult for people to infer the properties . Video Generations . For each video , we first use a physical engine ( Coumans & Bai , 2016–2021 ) to simulate objects ’ motions and then adopt a graphs engine ( Community , 2018 ) to render frame sequences . Each target video for question answering contains 3 to 5 objects with random composition between their appearance attributes and physical properties . We set the length of the target video to be 5 seconds and additionally simulate the 6-th and 7-th seconds of the target video for predictive question annotation . We provide more video generation details in the Appendix . Task Setup . It is not trivial to design a task setup to evaluate models ’ ability for physical reasoning since physical properties are not observable in a static frame . A straightforward design is to correlate object appearance with the physical property like “ red object is heavy ” , “ yellow object is light ” and then ask “ what would happen if they collide ” . However , such a setting is imperfect since it can not evaluate whether a model really understands the physical properties or just memorize the visual appearance prior . An ideal setting should be able to evaluate whether a model is like a human that can identify objects ’ properties from objects ’ motion and interactions with each other in the dynamic scenes and make the corresponding dynamic predictions . To achieve this , we design a meta setting for physical reasoning , which provides few reference video samples along with the target video for models to infer objects ’ physical properties and then ask questions about the objects ’ physical properties and dynamics . Following CLEVRER , we show a sample of the dataset in Fig . 2 . Each set contains a target video , 4 reference videos , and some questions about the visual attributes , physical properties , events , and dynamics of the target video . Objects in each set share the same visual attributes ( color , shape , and material ) and intrinsic physical property ( mass and charge ) . Reference videos . To provide abundant visual content for physical property inference , we additionally provide 4 reference videos for each target video . We sample 2-3 objects from the target video , provide them with different initial velocities and locations , and make them interact ( attract , repel or collide ) with each other . The generation of the reference video follows the same standard as the target video , but the length of the videos is set to 2 seconds for scaling up . The interaction among objects in reference videos helps models to inference objects ’ properties . For example , the repulsion in Reference video 1 of Fig . 2 can help us identify that object 1 and object 2 carrying the same charge .

Summary: The paper focuses on inferring two hidden physical properties (mass and electricity charge). A synthetic dataset, called ComPhy is created. The dataset consists of simple shapes (such as cubes and sphere) that move on a flat plane. For each query video, there are four reference videos that the models can use to learn the properties of objects. The dataset also includes three types of questions: factual, counterfactual and predictive. The paper also presents a pipeline approach called CPL that leverages graphs to model the relationship of objects and solve the QA tasks.