CoPhy: Counterfactual Learning of Physical Dynamics

Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world . This work poses a new problem of counterfactual learning of object mechanics from visual input . We develop the CoPhy benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting . Having observed a mechanical experiment that involves , for example , a falling tower of blocks , a set of bouncing balls or colliding objects , we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions , such as displacing one of the objects in the scene . The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision of confounders . We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment , which results in significantly more accurate predictions at the level of super human performance . 1 INTRODUCTION . Reasoning is an essential ability of intelligent agents that enables them to understand complex relationships between observations , detect affordances , interpret knowledge and beliefs , and to leverage this understanding to anticipate future events and act accordingly . The capacity for observational discovery of causal effects in physical reality and making sense of fundamental physical concepts , such as mass , velocity , friction , etc. , may be one of differentiating properties of human intelligence that ensures our ability to robustly generalize to new scenarios ( Martin-Ordas et al. , 2008 ) . One way to express causality is based on the concept of counterfactual reasoning , that deals with a problem containing an if statement , which is untrue or unrealized . Predicting the effect of the interventions based on the given observations without explicitly observing the effect of the intervention on data is a hard task and requires modeling of the causal relationships between the variable on which the intervention is performed and the variable whose alternative future should be predicted ( Balke & Pearl , 1994 ) . Using counterfactuals has been shown to be a way to perform reasoning over causal relationships between the variables of low dimensional spaces and has been an unexplored direction for high dimensional signals such as videos . In this work , we develop the Counterfactual Physics benchmark ( CoPhy ) and propose a framework for causal learning of dynamics in mechanical systems with multiple degrees of freedom , as illustrated in Fig . 1 . For a number of scenarios , such as tower of blocks falling , balls bouncing against walls or objects colliding , we are given the starting frame A = X0 and a sequence of following frames B = X1 : τ , where τ covers the range of 6 sec . The observed sequences B , conditioned on the initial state A , are direct effects of the physical principles ( such as inertia , gravity or friction ) applied to the closed system , that cause the objects change their positions and 3D poses over time . The task is formulated as follows : having observed the tuple ( A , B ) , we wish to predict positions and poses of all objects in the scene at time t=τ , if we had changed the initial frame X0 by performing an intervention . The intervention is formalized by the do-operator introduced by Pearl et al . ( Pearl , 2009 ; Pearl & McKenzie , 2018 ) for dealing with causal inference ( Spirtes , 2010 ) . In our case , it implies modification of the variable A to C , defined as C = do ( X0=X̄0 ) . Accordingly , for each experiment in the CoPhy benchmark , we provide pairs of original sequences X0 : τ and their modified counterparts X̄0 : τ sharing the same values of all confounders . We note the fundamental difference between this problem of counterfactual future forecasting and the conventional setup of feedforward future forecasting , like video prediction ( Mathieu et al. , 2016 ) . The latter involves learning spatio-temporal regularities and thereby predicting future frames X1 ... τ from one or several past frame ( s ) X0 ( the causal chain of this problem is shown in Fig . 2a ) . On the other hand , counterfactual forecasting benefits from additional observations in the form of the original outcome X1 : τ before the do-operator . This adds a confounder variable U into the causal chain ( Fig . 2b ) , which provides information not observable in frame X0 . For instance , in the case of the CoPhy benchmark , observing the pair ( A , B ) might give us information on the masses , velocities or friction coefficients of the objects in the scene , which otherwise can not be inferred from frame X̄0 alone . Therefore , predicting the alternative outcome after performing counterfactual intervention then involves using the estimate of the confounder U together with the modified past do ( X0=X̄0 ) . Overall , we employ the idea of counterfactual intervention in predictive models and argue that counterfactual reasoning is an important step towards human-like reasoning and general intelligence . More specifically , key contributions of this work include : • a new task of counterfactual prediction of physical dynamics from high-dimensional visual input , as a way to access capacity of intelligent agents for causal discovery ; • a large-scale CoPhy benchmark with three physical scenarios and 300k synthetic experiments including rendered sequences of frames , metadata ( object positions , angles , sizes ) and values of confounders ( masses , frictions , gravity ) . This benchmark was specifically designed in bias-free fashion to make the counter-factual reasoning task challenging by optimizing the impact of the confounders on the outcome of the experiment . The dataset is publicly available1 . • a counterfactual neural model predicting an alternative outcome of a physical experiment given an intervention , by estimating the latent representation of the confounders . The model outperforms state-of-the-art solutions implementing feedforward video prediction , successfully generalizes to unseen initial states and does not require supervision on the confounders . We provide extensive ablations on the effects of key design choices and compare results with human performance , that show that the task is hard for humans to solve . The code will be made publicly available . 2 RELATED WORK . This work is inspired by a significant number of prior studies from several subfields , including visual reasoning , learning intuitive physics and perceived causality . 1Project page : http : //projet.liris.cnrs.fr/cophy Visual reasoning . The most recent works on visual reasoning approach the problem in the setting of visual question answering ( Hu et al. , 2017 ; Hudson & Manning , 2018 ; Johnson et al. , 2017 ; Mao et al. , 2019 ; Perez et al. , 2018 ; Santoro et al. , 2017 ) , embodied AI ( Wijmans et al. , 2019 ) , as well as learning intuitive physics ( Lerer et al. , 2016 ; Riochet et al. , 2018 ) . ( Santoro et al. , 2017 ) introduced Relation Networks ( RN ) , a fully-differentiable trainable layer for reasoning in deep networks . Following the same trend , ( Baradel et al. , 2018 ) estimate object relations from semantically well defined entities using instance segmentation predictions for video understanding . ( Santoro et al. , 2018 ) build a challenging dataset for solving the problem of abstract reasoning on the visual domain with some tasks such as interpolation or extrapolation . External memory ( Graves et al. , 2016 ; Jaeger , 2016 ) extends known recurrent neural mechanisms by decoupling the size of a representation from the controller capacity and introduces the separation between long-term and short-term reasoning . ( Reed et al. , 2015 ) propose to learn analogies in a fully supervised way . Our work builds upon this literature and extends the idea of visual reasoning to the counterfactual setting . Intuitive physics . Fundamental studies on cognitive psychology have shown that humans perform poorly when asked to reason about expected outcomes of a physics based event , demonstrating striking deviations from Newtonian physics in their intuitions ( McCloskey & Kohl , 1983 ; McClooskey et al. , 1980 ; 1983 ; Kubricht et al. , 2017 ) . The questions of approximating these mechanisms , learning from noisy observed and non-observed physical quantities ( such as sizes or velocities vs masses or gravity ) , as well as justifying importance of explicit physical concepts vs cognitive constructs in intelligent agents have been raised and explored in recent works on deep learning ( Wu et al. , 2015 ) . ( Lerer et al. , 2016 ; Groth et al. , 2018 ) follow this direction by training networks to predict stability of block towers . ( Ye et al. , 2018 ) build an interpretable intuitive physical model from visual signals using full supervision on the physical properties of each object . On similar tasks , ( Wu et al. , 2017 ) propose to learn physics by interpreting and reconstructing the visual information stream leading to inverting physics or a graphical engine . ( Zheng et al. , 2018 ) propose to solve this task by first extracting a visual perception of the world state and then predict the future . ( Battaglia et al. , 2016 ) introduce a fully-differentiable network physics engine called Interaction Network ( IN ) , which learns to predict physical systems such as gravitational ones , rigid body dynamics , and mass-spring systems . Similarly , ( van Steenkiste et al. , 2018 ) discover objects and their interactions in a unsupervised manner from a virtual environment . In ( Veličković et al. , 2018 ) , attention and relational modules are combined on a graph structure . Recent approaches ( Chang et al. , 2017 ; Janner et al. , 2019 ; Battaglia et al. , 2018 ) based on Graph Convolution Networks ( Kipf & Welling , 2017 ) have shown promising results on learning physics but are restricted to setups where physical properties need to be fully observable , which is not the case of our approach . The most similar to ours is work by ( Ehrhardt et al. , 2019 ) on unsupervised learning of intuitive physics from unpaired past experiences . Other physics benchmarks and simulators . The main objective for the creation of our benchmark is ( a ) to focus specifically on evaluating capabilities of state of the art models for performing counterfactual reasoning , ( b ) to be unbiased in terms of distributions of parameters to be estimated and balanced with respect to possible outcomes , and ( c ) to have sufficient variety in terms of scenarios and latent physical characteristics of the scene that are not visually observed and therefore can act as confounders . To the best of our knowledge , none of existing intuitive physics benchmarks have these properties . IntPhys ( Riochet et al. , 2018 ) focuses on a high level task of estimating physical plausibility in a black box fashion and modeling out of distribution events at test time . CATER ( Girdhar & Ramanan , 2019 ) introduces a video dataset requiring spatiotemporal understanding in order to solve the tasks such as action recognition , compositional action recognition , and adversarial target tracking . Phyre ( Bakhtin et al. , 2019 ) is an environment for solving physics based puzzles , where achieving sample efficiency may implicitly require counterfactual reasoning , but this component is not explicitly evaluated , construction of parallel data with several alternative outcomes is not straightforward , and the trivial baseline performance levels are not easy to estimate . Adapting these benchmarks to counterfactual reasoning would require significant refactoring and changing the logic of the data sampling . CLEVRER ( Yi et al. , 2019 ) is a diagnostic video dataset for systematic evaluation of models on a wide range of reasoning tasks including counterfactual questions . They cast the task as a classification problem where the model has to choose between a set of possible answers , whereas our benchmark requires the predicting the dynamics of each object . Perceptual causality . In the ML community , causal reasoning gained mainstream attention relatively recently ( Lopez-Paz et al. , 2017 ; Lopez-Paz & Oquab , 2017 ; Kocaoglu et al. , 2018 ; Rojas-Carulla et al. , 2018 ; Mooij et al. , 2016 ; Schölkopf et al. , 2012 ) , due to limitations of statistical learning becoming increasingly apparent ( Pearl , 2018 ; Lake et al. , 2017 ) . The concept of perceived causality has been however explored in cognitive psychology ( Michotte , 1963 ) , where human subjects have be shown to consistently report causal impressions not aligned with underlying physical principles of the events ( Gerstenberg et al. , 2015 ; Kubricht et al. , 2017 ) . Exploiting the colliding objects scenario as a standard testbed for these studies led to discovery of a number of cognitive biases , e.g . Motor Object Bias ( i.e . false perceived association of object ’ s velocity with its mass ) . In this work , we bring the domains of visual reasoning , intuitive physics and perceived causality together in a single framework to tackle the new problem of counterfactual learning of physical dynamics . Following prior literature ( Battaglia et al. , 2013 ) , we also compare counterfactual learning with human performance and expect that , similarly to learning intuitive vs Newtonian physics , modeling perceived vs true causality will get more attention from the ML community in the future .

Summary: This paper studies counterfactual event prediction in physical simulation. The authors proposed a model that leverages object-centric scene representations and graph networks for modeling object interactions. The model also uses a recurrent network to encode and extract the confounder information for counterfactual prediction. In the experiments, the authors compared the proposed method, baselines, and human performance on pose estimation and counterfactual prediction.