Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space

1 INTRODUCTION . Reasoning on complex , multi-modal and high-dimensional data is a natural ability of humans and other intelligent agents ( Martin-Ordas et al. , 2008 ) , and one of the most important and difficult challenges of AI . While machine learning is well suited for capturing regularities in high-dimensional signals , in particular by using high-capacity deep networks , some applications also require an accurate modeling of causal relationships . This is particularly relevant in physics , where causation is considered as a fundamental axiom . In the context of machine learning , correctly capturing or modeling causal relationships can also lead to more robust predictions , in particular better generalization to out-of-distribution samples , indicating that a model has overcome the exploitation of biases and shortcuts in the training data . In recent literature on physics-inspired machine learning , causality has often been forced through the addition of prior knowledge about the physical laws that govern the studied phenomena , e.g . ( Yin et al. , 2021 ) . A similar idea lies behind structured causal models , widely used in the causal inference community , where domain experts model these relationships directly in a graphical notation . This particular line of work allows to perform predictions beyond statistical forecasting , for instance by predicting unobserved counterfactuals , the impact of unobserved interventions ( Balke & Pearl , 1994 ) — “ What alternative outcome would have happened , if the observed event X had been replaced with an event Y ( after an intervention ) ” . Counterfactuals are interesting , as causality intervenes through the effective modification of an outcome . As an example , taken from ( Schölkopf et al. , 2021 ) , an agent can identify the direction of a causal relationship between an umbrella and rain from the fact that removing an umbrella will not affect the weather . We focus on counterfactual reasoning on high-dimensional signals , in particular videos of complex physical processes . Learning such causal interactions from data is a challenging task , as spurious correlations are naturally and easily picked up by trained models . Previous work in this direction was restricted to discrete outcomes , as in CLEVRER ( Yi et al. , 2020 ) , or to the prediction of 3D trajectories , as in CoPhy ( Baradel et al. , 2020 ) , which also requires supervision of object positions . In this work , we address the hard problem of predicting the alternative ( counterfactual ) outcomes of physical processes in pixel space , i.e . we forecast sequences of 2D projective views of the 3D scene , requiring the prediction over long horizons ( 150 frames corresponding to ∼ 6 seconds ) . We conjecture that causal relationships can be modeled on a low dimensional manifold of the data , and propose a suitable latent representation for the causal model , in particular for the estimation of the confounders and the dynamic model itself . Similar to V-CDN ( Kulkarni et al. , 2019 ; Li et al. , 2020 ) , our latent representation is based on the unsupervised discovery of keypoints , complemented by additional information in our case . Indeed , while keypoint-based representations can easily be encoded from visual input , as stable mappings from images to points arise naturally , we claim that they are not the most suitable representation for dynamic models . We identified and addressed two principal problems : ( i ) the individual points of a given set are discriminated through their 2D positions only , therefore shape , geometry and relationships between multiple moving objects need to be encoded through the relative positions of points to each other , and ( ii ) the optimal representation for a physical dynamic model is not necessarily a 2D keypoint space , where the underlying object dynamics has also been subject to the imaging process ( projective geometry ) . We propose a new counterfactual model , which learns a sparse representation of visual input in the form of 2D keypoints coupled with a ( small ) set of coefficients per point modeling complementary shape and appearance information . Confounders ( object masses and initial velocities ) in the studied problem are extracted from this representation , and a learned dynamic model forecasts the entire trajectory of these keypoints from a single ( counterfactual ) observation . Building on recent work in data-driven analysis of dynamic systems ( Janny et al. , 2021 ; Peralez & Nadri , 2021 ) , the dynamic model is presented in a higher-dimensional state space , where dynamics are less complex . We show , that these design choices are key to the performance of our model , and that they significantly improve the capability to perform long-term predictions . Our proposed model outperforms strong baselines for physics-informed learning of video prediction . We introduce a new challenging dataset for this problem , which builds on CoPhy , a recent counterfactual physics benchmark ( Baradel et al. , 2020 ) . We go beyond the prediction of sequences of 3D positions and propose a counterfactual task for predictions in pixel space after interventions on initial conditions ( displacing , re-orienting or removing objects ) . In contrast to the literature , our benchmark also better controls for the identifiability of causal relationships and counterfactual variables and provides more accurate physics simulation . 2 RELATED WORK . Counterfactual ( CF ) reasoning — and learning of causal relationships in ML was made popular by works of J. Pearl , e.g . ( Pearl , 2000 ) , which motivate and introduce mathematical tools detailing the principles of do-calculus , i.e . study of unobserved interventions on data . A more recent survey links these concepts to the literature in ML ( Schölkopf et al. , 2021 ) . The last years have seen the emergence of several benchmarks for CF reasoning in physics . CLEVRER ( Yi et al. , 2020 ) is a visual question answering dataset , where an agent is required to answer a CF question after observing a video showing 3D objects moving and colliding . Li et al . ( 2020 ) introduce a CF benchmark with two tasks : a scenario where balls interact with each other according to unknown interaction laws ( such as gravity or elasticity ) , and a scenario where clothes are folded by the wind . The agent needs to identify CF variables and causal relationships between objects , and to predict future frames . CoPhy ( Baradel et al. , 2020 ) clearly dissociates the observed experiment from the CF one , and contains three complex 3D scenarios involving rigid body dynamics . However , the proposed method relies on the supervision of 3D object positions , while our work does not require any meta data . Physics-inspired ML — and learning visual dynamics has been dealt early on with recurrent models ( Srivastava et al. , 2015 ; Finn et al. , 2016 ; Lu et al. , 2017 ) , or GANs ( Vondrick et al. , 2016 ; Mathieu et al. , 2016 ) . Kwon & Park ( 2019 ) adopt a Cycle-GAN with two discriminator heads , in charge of identifying false images and false sequences in order to improve the temporal consistency of the model in long term prediction . Nonetheless , the integration of causal reasoning and prior knowledge in these models is not straightforward . Typical work in physics-informed models relies on disentanglement between physics-informed features and residual features ( Villegas et al. , 2017a ; Denton & Birodkar , 2017 ) and may incorporate additional information based on the available priors on the scene ( Villegas et al. , 2017b ; Walker et al. , 2017 ) . PhyDNet Le Guen & Thome ( 2020 ) explicitly disentangles visual features from dynamical features , which are supposed to follow a PDE . It achieves SOTA performance on Human3.6M ( Ionescu et al. , 2014 ) and Sea Surface Temperature ( de Bezenac et al. , 2018 ) , but we show that it fails on our challenging benchmark . Keypoint detection — is a well researched problem in vision with widely used handcrafted baselines ( Lowe , 1999 ) . New unsupervised variants emerged recently and have been shown to provide a suitable object-centric representation , close to attention models , which simplify the use of physical and/or geometric priors ( Locatello et al. , 2020 ; Veerapaneni et al. , 2020 ) . They are of interest in robotics and reinforcement learning , where a physical agent has to interact with objects ( Kulkarni et al. , 2019 ; Manuelli et al. , 2020 ; 2019 ) . KeypointsNet ( Suwajanakorn et al. , 2018 ) is a geometric reasoning framework , which discovers meaningful keypoints in 3D through spatial coherence between viewpoints . Close to our work , ( Minderer et al. , 2019 ) proposes to learn a keypoints-based stochastic dynamic model . However , the model is not suited for CF reasoning in physics and may suffer from inconsistency in the prediction of dynamics over long horizons . 3 THE FILTERED-COPHY BENCHMARK We build on CoPhy ( Baradel et al. , 2020 ) , retaining its strengths , but explicitly focusing on a counterfactual scenario in pixel space and eliminating the ill-posedness of tasks we identified in the existing work . Each data sample is called an experiment , represented as a pair of trajectories : an observed one with initial conditionX0 = A and outcomeXt=1 .. T = B ( a sequence ) , and a counterfactual one X̄0 = C and X̄t=1 .. T = D ( a sequence ) . Throughout this paper we will use the letters A , B , C and D to distinguish the different parts of each experiment . The initial conditions A and C are linked through a do-operator do ( X0 = C ) , which modifies the initial condition ( Pearl , 2018 ) . Experiments are parameterized by a set of intrinsic physical parameters z which are not observable from a single initial image A . We refer to these as confounders . As in CoPhy , in our benchmark the do-operator is observed during training , but confounders are not — they have been used to generate the data , but are not used during training or testing . Following ( Pearl , 2018 ) , the counterfactual task consists in inferring the counterfactual outcome D given the observed trajectory AB and the counterfactual initial state C , following a three-step process : À Abduction : use the observed data AB to compute the counterfactual variables , i.e . physical parameters , which are not affected by the do-operation . Á Action : update the causal model ; keep the same identified confounders and apply the dooperator , i.e . replace the initial state A by C. Â Prediction : Compute the counterfactual outcome D using the causal graph . The benchmark contains three scenarios involving rigid body dynamics . BlocktowerCF studies stable and unstable 3D cube towers , the confounders are masses . BallsCF focuses on 2D collisions between moving spheres ( confounders are masses and initial velocities ) . CollisionCF is about collisions between a sphere and a cylinder ( confounders are masses and initial velocities ) ( Fig . 1 ) . Unlike CoPhy , our benchmark involves predictions in RGB pixel space only . The do-operation consists in visually observable interventions on A , such as moving or removing an object . The confounders can not be identified from the single-frame observation A , identification requires the analysis of the entire AB trajectory . Identifiability of confounders — For an experiment ( AB , CD , z ) to be well-posed , the confounders z must be retrievable from AB . For example , since the masses of a stable cube tower can not be identified generally in all situations , it can be impossible to predict the counterfactual outcome of an unstable tower , as collisions are not resolvable without known masses . In contrast to CoPhy , we ensure that each experiment ψ : ( X0 , z ) 7→ Xt=1 .. T , given initial condition X0 and confounders z , is well posed and satisfies the following constraints : Definition 1 ( Identifiability , ( Pearl , 2018 ) ) The experiment ( AB , CD , z ) is identifiable if , for any set of confounders z′ : ψ ( A , z ) = ψ ( A , z′ ) ⇒ ψ ( C , z ) = ψ ( C , z′ ) . ( 1 ) In an identifiable experiment there is no pair ( z , z′ ) that gives the same trajectory AB but different counterfactual outcomes CD . Details on implementation and impact are in appendix A.1 . Counterfactuality — We enforce sufficient difficulty of the problem through the meaningfulness of confounders . We remove initial situations where the choice of confounder values has no significant impact on the final outcome : Definition 2 ( Counterfactuality ) . Let zk be the set of confounders z , where the kth value has been modified . The experiment ( AB , CD , z ) is counterfactual if and only if : ∃k : ψ ( C , zk ) 6= ψ ( C , z ) . ( 2 ) In other words , we impose the existence of an object of the scene for which the ( unobserved ) physical properties have a determining effect on the trajectory . Details on how this constraint was enforced are given in appendix A.2 . Temporal resolution — the physical laws we target involve highly non-linear phenomena , in particular collision and resting contacts . Collisions are difficult to learn because their actions are both intense , brief , and highly non-linear , depending on the geometry of the objects in 3D space . The temporal resolution of physical simulations is of prime importance . A parallel can be made with Nyquist-Shannon frequency , as a trajectory sampled with too low frequency can not be reconstructed with precision . We simulate and record trajectories at 25 FPS , compared to 5 FPS chosen in CoPhy , justified with two experiments . Firstly , Fig . 2 shows the trajectories of the center of masses of cubes in BlocktowerCF , colored dots are shown at 25 FPS and black dots at 5 FPS . We can see that collisions with the ground fall below the sampling rate of 5 FPS , making it hard to infer physical laws from regularities in data at this frequency . A second experiment involves learning a prediction model at different frequencies , confirming the choice 25 FPS — details are given in appendix A.3 .

Summary: This work extends CoPhy, in which the author(s) address the problem of predicting counterfactual outcomes of physics-based tasks from pixel space (CoPhy used ground truth object positions). To do this, they learn a keypoint representation of the scene and use them to extract the confounders (such as velocities of objects, masses, etc). The author(s) also propose a new benchmark (built upon CoPhy's benchmark) for counterfactual prediction (after intervening the initial set of objects in the scene) which satisfies the Identifiability and the Counterfactuality constraints of causality.