Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space

1 INTRODUCTION . Reasoning on complex , multi-modal and high-dimensional data is a natural ability of humans and other intelligent agents ( Martin-Ordas et al. , 2008 ) , and one of the most important and difficult challenges of AI . While machine learning is well suited for capturing regularities in high-dimensional signals , in particular by using high-capacity deep networks , some applications also require an accurate modeling of causal relationships . This is particularly relevant in physics , where causation is considered as a fundamental axiom . In the context of machine learning , correctly capturing or modeling causal relationships can also lead to more robust predictions , in particular better generalization to out-of-distribution samples , indicating that a model has overcome the exploitation of biases and shortcuts in the training data . In recent literature on physics-inspired machine learning , causality has often been forced through the addition of prior knowledge about the physical laws that govern the studied phenomena , e.g . ( Yin et al. , 2021 ) . A similar idea lies behind structured causal models , widely used in the causal inference community , where domain experts model these relationships directly in a graphical notation . This particular line of work allows to perform predictions beyond statistical forecasting , for instance by predicting unobserved counterfactuals , the impact of unobserved interventions ( Balke & Pearl , 1994 ) — “ What alternative outcome would have happened , if the observed event X had been replaced with an event Y ( after an intervention ) ” . Counterfactuals are interesting , as causality intervenes through the effective modification of an outcome . As an example , taken from ( Schölkopf et al. , 2021 ) , an agent can identify the direction of a causal relationship between an umbrella and rain from the fact that removing an umbrella will not affect the weather . We focus on counterfactual reasoning on high-dimensional signals , in particular videos of complex physical processes . Learning such causal interactions from data is a challenging task , as spurious correlations are naturally and easily picked up by trained models . Previous work in this direction was restricted to discrete outcomes , as in CLEVRER ( Yi et al. , 2020 ) , or to the prediction of 3D trajectories , as in CoPhy ( Baradel et al. , 2020 ) , which also requires supervision of object positions . In this work , we address the hard problem of predicting the alternative ( counterfactual ) outcomes of physical processes in pixel space , i.e . we forecast sequences of 2D projective views of the 3D scene , requiring the prediction over long horizons ( 150 frames corresponding to ∼ 6 seconds ) . We conjecture that causal relationships can be modeled on a low dimensional manifold of the data , and propose a suitable latent representation for the causal model , in particular for the estimation of the confounders and the dynamic model itself . Similar to V-CDN ( Kulkarni et al. , 2019 ; Li et al. , 2020 ) , our latent representation is based on the unsupervised discovery of keypoints , complemented by additional information in our case . Indeed , while keypoint-based representations can easily be encoded from visual input , as stable mappings from images to points arise naturally , we claim that they are not the most suitable representation for dynamic models . We identified and addressed two principal problems : ( i ) the individual points of a given set are discriminated through their 2D positions only , therefore shape , geometry and relationships between multiple moving objects need to be encoded through the relative positions of points to each other , and ( ii ) the optimal representation for a physical dynamic model is not necessarily a 2D keypoint space , where the underlying object dynamics has also been subject to the imaging process ( projective geometry ) . We propose a new counterfactual model , which learns a sparse representation of visual input in the form of 2D keypoints coupled with a ( small ) set of coefficients per point modeling complementary shape and appearance information . Confounders ( object masses and initial velocities ) in the studied problem are extracted from this representation , and a learned dynamic model forecasts the entire trajectory of these keypoints from a single ( counterfactual ) observation . Building on recent work in data-driven analysis of dynamic systems ( Janny et al. , 2021 ; Peralez & Nadri , 2021 ) , the dynamic model is presented in a higher-dimensional state space , where dynamics are less complex . We show , that these design choices are key to the performance of our model , and that they significantly improve the capability to perform long-term predictions . Our proposed model outperforms strong baselines for physics-informed learning of video prediction . We introduce a new challenging dataset for this problem , which builds on CoPhy , a recent counterfactual physics benchmark ( Baradel et al. , 2020 ) . We go beyond the prediction of sequences of 3D positions and propose a counterfactual task for predictions in pixel space after interventions on initial conditions ( displacing , re-orienting or removing objects ) . In contrast to the literature , our benchmark also better controls for the identifiability of causal relationships and counterfactual variables and provides more accurate physics simulation . 2 RELATED WORK . Counterfactual ( CF ) reasoning — and learning of causal relationships in ML was made popular by works of J. Pearl , e.g . ( Pearl , 2000 ) , which motivate and introduce mathematical tools detailing the principles of do-calculus , i.e . study of unobserved interventions on data . A more recent survey links these concepts to the literature in ML ( Schölkopf et al. , 2021 ) . The last years have seen the emergence of several benchmarks for CF reasoning in physics . CLEVRER ( Yi et al. , 2020 ) is a visual question answering dataset , where an agent is required to answer a CF question after observing a video showing 3D objects moving and colliding . Li et al . ( 2020 ) introduce a CF benchmark with two tasks : a scenario where balls interact with each other according to unknown interaction laws ( such as gravity or elasticity ) , and a scenario where clothes are folded by the wind . The agent needs to identify CF variables and causal relationships between objects , and to predict future frames . CoPhy ( Baradel et al. , 2020 ) clearly dissociates the observed experiment from the CF one , and contains three complex 3D scenarios involving rigid body dynamics . However , the proposed method relies on the supervision of 3D object positions , while our work does not require any meta data . Physics-inspired ML — and learning visual dynamics has been dealt early on with recurrent models ( Srivastava et al. , 2015 ; Finn et al. , 2016 ; Lu et al. , 2017 ) , or GANs ( Vondrick et al. , 2016 ; Mathieu et al. , 2016 ) . Kwon & Park ( 2019 ) adopt a Cycle-GAN with two discriminator heads , in charge of identifying false images and false sequences in order to improve the temporal consistency of the model in long term prediction . Nonetheless , the integration of causal reasoning and prior knowledge in these models is not straightforward . Typical work in physics-informed models relies on disentanglement between physics-informed features and residual features ( Villegas et al. , 2017a ; Denton & Birodkar , 2017 ) and may incorporate additional information based on the available priors on the scene ( Villegas et al. , 2017b ; Walker et al. , 2017 ) . PhyDNet Le Guen & Thome ( 2020 ) explicitly disentangles visual features from dynamical features , which are supposed to follow a PDE . It achieves SOTA performance on Human3.6M ( Ionescu et al. , 2014 ) and Sea Surface Temperature ( de Bezenac et al. , 2018 ) , but we show that it fails on our challenging benchmark . Keypoint detection — is a well researched problem in vision with widely used handcrafted baselines ( Lowe , 1999 ) . New unsupervised variants emerged recently and have been shown to provide a suitable object-centric representation , close to attention models , which simplify the use of physical and/or geometric priors ( Locatello et al. , 2020 ; Veerapaneni et al. , 2020 ) . They are of interest in robotics and reinforcement learning , where a physical agent has to interact with objects ( Kulkarni et al. , 2019 ; Manuelli et al. , 2020 ; 2019 ) . KeypointsNet ( Suwajanakorn et al. , 2018 ) is a geometric reasoning framework , which discovers meaningful keypoints in 3D through spatial coherence between viewpoints . Close to our work , ( Minderer et al. , 2019 ) proposes to learn a keypoints-based stochastic dynamic model . However , the model is not suited for CF reasoning in physics and may suffer from inconsistency in the prediction of dynamics over long horizons . 3 THE FILTERED-COPHY BENCHMARK We build on CoPhy ( Baradel et al. , 2020 ) , retaining its strengths , but explicitly focusing on a counterfactual scenario in pixel space and eliminating the ill-posedness of tasks we identified in the existing work . Each data sample is called an experiment , represented as a pair of trajectories : an observed one with initial conditionX0 = A and outcomeXt=1 .. T = B ( a sequence ) , and a counterfactual one X̄0 = C and X̄t=1 .. T = D ( a sequence ) . Throughout this paper we will use the letters A , B , C and D to distinguish the different parts of each experiment . The initial conditions A and C are linked through a do-operator do ( X0 = C ) , which modifies the initial condition ( Pearl , 2018 ) . Experiments are parameterized by a set of intrinsic physical parameters z which are not observable from a single initial image A . We refer to these as confounders . As in CoPhy , in our benchmark the do-operator is observed during training , but confounders are not — they have been used to generate the data , but are not used during training or testing . Following ( Pearl , 2018 ) , the counterfactual task consists in inferring the counterfactual outcome D given the observed trajectory AB and the counterfactual initial state C , following a three-step process : À Abduction : use the observed data AB to compute the counterfactual variables , i.e . physical parameters , which are not affected by the do-operation . Á Action : update the causal model ; keep the same identified confounders and apply the dooperator , i.e . replace the initial state A by C. Â Prediction : Compute the counterfactual outcome D using the causal graph . The benchmark contains three scenarios involving rigid body dynamics . BlocktowerCF studies stable and unstable 3D cube towers , the confounders are masses . BallsCF focuses on 2D collisions between moving spheres ( confounders are masses and initial velocities ) . CollisionCF is about collisions between a sphere and a cylinder ( confounders are masses and initial velocities ) ( Fig . 1 ) . Unlike CoPhy , our benchmark involves predictions in RGB pixel space only . The do-operation consists in visually observable interventions on A , such as moving or removing an object . The confounders can not be identified from the single-frame observation A , identification requires the analysis of the entire AB trajectory . Identifiability of confounders — For an experiment ( AB , CD , z ) to be well-posed , the confounders z must be retrievable from AB . For example , since the masses of a stable cube tower can not be identified generally in all situations , it can be impossible to predict the counterfactual outcome of an unstable tower , as collisions are not resolvable without known masses . In contrast to CoPhy , we ensure that each experiment ψ : ( X0 , z ) 7→ Xt=1 .. T , given initial condition X0 and confounders z , is well posed and satisfies the following constraints : Definition 1 ( Identifiability , ( Pearl , 2018 ) ) The experiment ( AB , CD , z ) is identifiable if , for any set of confounders z′ : ψ ( A , z ) = ψ ( A , z′ ) ⇒ ψ ( C , z ) = ψ ( C , z′ ) . ( 1 ) In an identifiable experiment there is no pair ( z , z′ ) that gives the same trajectory AB but different counterfactual outcomes CD . Details on implementation and impact are in appendix A.1 . Counterfactuality — We enforce sufficient difficulty of the problem through the meaningfulness of confounders . We remove initial situations where the choice of confounder values has no significant impact on the final outcome : Definition 2 ( Counterfactuality ) . Let zk be the set of confounders z , where the kth value has been modified . The experiment ( AB , CD , z ) is counterfactual if and only if : ∃k : ψ ( C , zk ) 6= ψ ( C , z ) . ( 2 ) In other words , we impose the existence of an object of the scene for which the ( unobserved ) physical properties have a determining effect on the trajectory . Details on how this constraint was enforced are given in appendix A.2 . Temporal resolution — the physical laws we target involve highly non-linear phenomena , in particular collision and resting contacts . Collisions are difficult to learn because their actions are both intense , brief , and highly non-linear , depending on the geometry of the objects in 3D space . The temporal resolution of physical simulations is of prime importance . A parallel can be made with Nyquist-Shannon frequency , as a trajectory sampled with too low frequency can not be reconstructed with precision . We simulate and record trajectories at 25 FPS , compared to 5 FPS chosen in CoPhy , justified with two experiments . Firstly , Fig . 2 shows the trajectories of the center of masses of cubes in BlocktowerCF , colored dots are shown at 25 FPS and black dots at 5 FPS . We can see that collisions with the ground fall below the sampling rate of 5 FPS , making it hard to infer physical laws from regularities in data at this frequency . A second experiment involves learning a prediction model at different frequencies , confirming the choice 25 FPS — details are given in appendix A.3 .

Summary: The paper introduces a testing approach, dataset, and method for counterfactual video predictions, using 3d physics simulation videos.  Importantly, the approach predicts directly from pixel space rather than requiring spoon-fed keypoints.  It employs a counterfactual approach to establish a network's capacity to learn causal relations.  Separating the problem into one of parsing the inputs into keypoints and additional coefficients, inferring object attributes, and sequential prediction from an input frame + object attributes, it proposes an architecture which is based on combining modules for each of these learning tasks (but where keypoints are learned in an unsupervised way).  Besides this key architectural innovation, it adds an inductive bias by applying directional Gaussian filters to the keypoint maps.  The paper checks that the network actually works as intended by empirically examining the effect of changing object coefficients.  Three other ablation analyses, one that combines two of the modules (rather than separating them via stop-grad) and one that removes coefficients, and one comparing the handcrafted filter bank with learnable ones, add confidence to the approach.  In the supplement, the paper also analyzes in detail how the method compares to a previous method (Transporter).  It uses sensible benchmarks including previous methods and common-sense baselines where either the original or counterfactual input is used as a prediction, and does relatively well on this challenging problem, although the video on the website shows that there is still plenty of room for improvement.