Year: 2022

Abstract:
Learning causal relationships in high-dimensional data (images, videos) is a hard task, as they are often defined on low-dimensional manifolds and must be extracted from complex signals dominated by appearance, lighting, textures and also spurious correlations in the data. We present a method for learning counterfactual reasoning of physical processes in pixel space, which requires the prediction of the impact of interventions on initial conditions. Going beyond the identification of structural relationships, we deal with the challenging problem of forecasting raw video over long horizons. Our method does not require the knowledge or supervision of any ground truth positions or other object or scene properties. Our model learns and acts on a suitable hybrid latent representation based on a combination of dense features, sets of 2D keypoints and an additional latent vector per keypoint. We show that this better captures the dynamics of physical processes than purely dense or sparse representations. We introduce a new challenging and carefully designed counterfactual benchmark for predictions in pixel space and outperform strong baselines in physics-inspired ML and video prediction.

heading: 1 INTRODUCTION
text: Reasoning on complex, multi-modal and high-dimensional data is a natural ability of humans and other intelligent agents (Martin-Ordas et al., 2008), and one of the most important and difficult challenges of AI. While machine learning is well suited for capturing regularities in high-dimensional signals, in particular by using high-capacity deep networks, some applications also require an accurate modeling of causal relationships. This is particularly relevant in physics, where causation is considered as a fundamental axiom. In the context of machine learning, correctly capturing or modeling causal relationships can also lead to more robust predictions, in particular better generalization to out-of-distribution samples, indicating that a model has overcome the exploitation of biases and shortcuts in the training data. In recent literature on physics-inspired machine learning, causality has often been forced through the addition of prior knowledge about the physical laws that govern the studied phenomena, e.g. (Yin et al., 2021). A similar idea lies behind structured causal models, widely used in the causal inference community, where domain experts model these relationships directly in a graphical notation. This particular line of work allows to perform predictions beyond statistical forecasting, for instance by predicting unobserved counterfactuals, the impact of unobserved interventions (Balke & Pearl, 1994) — “What alternative outcome would have happened, if the observed event X had been replaced with an event Y (after an intervention)”. Counterfactuals are interesting, as causality intervenes through the effective modification of an outcome. As an example, taken from (Schölkopf et al., 2021), an agent can identify the direction of a causal relationship between an umbrella and rain from the fact that removing an umbrella will not affect the weather.
We focus on counterfactual reasoning on high-dimensional signals, in particular videos of complex physical processes. Learning such causal interactions from data is a challenging task, as spurious correlations are naturally and easily picked up by trained models. Previous work in this direction
was restricted to discrete outcomes, as in CLEVRER (Yi et al., 2020), or to the prediction of 3D trajectories, as in CoPhy (Baradel et al., 2020), which also requires supervision of object positions. In this work, we address the hard problem of predicting the alternative (counterfactual) outcomes of physical processes in pixel space, i.e. we forecast sequences of 2D projective views of the 3D scene, requiring the prediction over long horizons (150 frames corresponding to ∼ 6 seconds). We conjecture that causal relationships can be modeled on a low dimensional manifold of the data, and propose a suitable latent representation for the causal model, in particular for the estimation of the confounders and the dynamic model itself. Similar to V-CDN (Kulkarni et al., 2019; Li et al., 2020), our latent representation is based on the unsupervised discovery of keypoints, complemented by additional information in our case. Indeed, while keypoint-based representations can easily be encoded from visual input, as stable mappings from images to points arise naturally, we claim that they are not the most suitable representation for dynamic models. We identified and addressed two principal problems: (i) the individual points of a given set are discriminated through their 2D positions only, therefore shape, geometry and relationships between multiple moving objects need to be encoded through the relative positions of points to each other, and (ii) the optimal representation for a physical dynamic model is not necessarily a 2D keypoint space, where the underlying object dynamics has also been subject to the imaging process (projective geometry).
We propose a new counterfactual model, which learns a sparse representation of visual input in the form of 2D keypoints coupled with a (small) set of coefficients per point modeling complementary shape and appearance information. Confounders (object masses and initial velocities) in the studied problem are extracted from this representation, and a learned dynamic model forecasts the entire trajectory of these keypoints from a single (counterfactual) observation. Building on recent work in data-driven analysis of dynamic systems (Janny et al., 2021; Peralez & Nadri, 2021), the dynamic model is presented in a higher-dimensional state space, where dynamics are less complex. We show, that these design choices are key to the performance of our model, and that they significantly improve the capability to perform long-term predictions. Our proposed model outperforms strong baselines for physics-informed learning of video prediction.
We introduce a new challenging dataset for this problem, which builds on CoPhy, a recent counterfactual physics benchmark (Baradel et al., 2020). We go beyond the prediction of sequences of 3D positions and propose a counterfactual task for predictions in pixel space after interventions on initial conditions (displacing, re-orienting or removing objects). In contrast to the literature, our benchmark also better controls for the identifiability of causal relationships and counterfactual variables and provides more accurate physics simulation.

heading: 2 RELATED WORK
text: Counterfactual (CF) reasoning — and learning of causal relationships in ML was made popular by works of J. Pearl, e.g. (Pearl, 2000), which motivate and introduce mathematical tools detailing the principles of do-calculus, i.e. study of unobserved interventions on data. A more recent survey links these concepts to the literature in ML (Schölkopf et al., 2021). The last years have seen the emergence of several benchmarks for CF reasoning in physics. CLEVRER (Yi et al., 2020) is a visual question answering dataset, where an agent is required to answer a CF question after observing a video showing 3D objects moving and colliding. Li et al. (2020) introduce a CF benchmark with two tasks: a scenario where balls interact with each other according to unknown interaction laws (such as gravity or elasticity), and a scenario where clothes are folded by the wind. The agent needs to identify CF variables and causal relationships between objects, and to predict future frames. CoPhy (Baradel et al., 2020) clearly dissociates the observed experiment from the CF one, and contains three complex 3D scenarios involving rigid body dynamics. However, the proposed method relies on the supervision of 3D object positions, while our work does not require any meta data.
Physics-inspired ML — and learning visual dynamics has been dealt early on with recurrent models (Srivastava et al., 2015; Finn et al., 2016; Lu et al., 2017), or GANs (Vondrick et al., 2016; Mathieu et al., 2016). Kwon & Park (2019) adopt a Cycle-GAN with two discriminator heads, in charge of identifying false images and false sequences in order to improve the temporal consistency of the model in long term prediction. Nonetheless, the integration of causal reasoning and prior knowledge in these models is not straightforward. Typical work in physics-informed models relies on disentanglement between physics-informed features and residual features (Villegas et al., 2017a;
Denton & Birodkar, 2017) and may incorporate additional information based on the available priors on the scene (Villegas et al., 2017b; Walker et al., 2017). PhyDNet Le Guen & Thome (2020) explicitly disentangles visual features from dynamical features, which are supposed to follow a PDE. It achieves SOTA performance on Human3.6M (Ionescu et al., 2014) and Sea Surface Temperature (de Bezenac et al., 2018), but we show that it fails on our challenging benchmark.
Keypoint detection — is a well researched problem in vision with widely used handcrafted baselines (Lowe, 1999). New unsupervised variants emerged recently and have been shown to provide a suitable object-centric representation, close to attention models, which simplify the use of physical and/or geometric priors (Locatello et al., 2020; Veerapaneni et al., 2020). They are of interest in robotics and reinforcement learning, where a physical agent has to interact with objects (Kulkarni et al., 2019; Manuelli et al., 2020; 2019). KeypointsNet (Suwajanakorn et al., 2018) is a geometric reasoning framework, which discovers meaningful keypoints in 3D through spatial coherence between viewpoints. Close to our work, (Minderer et al., 2019) proposes to learn a keypoints-based stochastic dynamic model. However, the model is not suited for CF reasoning in physics and may suffer from inconsistency in the prediction of dynamics over long horizons.
3 THE FILTERED-COPHY BENCHMARK
We build on CoPhy (Baradel et al., 2020), retaining its strengths, but explicitly focusing on a counterfactual scenario in pixel space and eliminating the ill-posedness of tasks we identified in the existing work. Each data sample is called an experiment, represented as a pair of trajectories: an observed one with initial conditionX0 = A and outcomeXt=1..T = B (a sequence), and a counterfactual one X̄0 = C and X̄t=1..T = D (a sequence). Throughout this paper we will use the letters A,B,C and D to distinguish the different parts of each experiment. The initial conditions A and C are linked through a do-operator do(X0 = C), which modifies the initial condition (Pearl, 2018). Experiments are parameterized by a set of intrinsic physical parameters z which are not observable from a single initial image A. We refer to these as confounders. As in CoPhy, in our benchmark the do-operator is observed during training, but confounders are not — they have been used to generate the data, but are not used during training or testing. Following (Pearl, 2018), the counterfactual task consists in inferring the counterfactual outcome D given the observed trajectory AB and the counterfactual initial state C, following a three-step process:
À Abduction: use the observed data AB to compute the counterfactual variables, i.e. physical parameters, which are not affected by the do-operation.
Á Action: update the causal model; keep the same identified confounders and apply the dooperator, i.e. replace the initial state A by C.
Â Prediction : Compute the counterfactual outcome D using the causal graph.
The benchmark contains three scenarios involving rigid body dynamics. BlocktowerCF studies stable and unstable 3D cube towers, the confounders are masses. BallsCF focuses on 2D collisions between moving spheres (confounders are masses and initial velocities). CollisionCF is about collisions between a sphere and a cylinder (confounders are masses and initial velocities) (Fig. 1).
Unlike CoPhy, our benchmark involves predictions in RGB pixel space only. The do-operation consists in visually observable interventions on A, such as moving or removing an object. The confounders cannot be identified from the single-frame observation A, identification requires the analysis of the entire AB trajectory.
Identifiability of confounders — For an experiment (AB,CD, z) to be well-posed, the confounders z must be retrievable from AB. For example, since the masses of a stable cube tower cannot be identified generally in all situations, it can be impossible to predict the counterfactual outcome of an unstable tower, as collisions are not resolvable without known masses. In contrast to CoPhy, we ensure that each experiment ψ : (X0, z) 7→ Xt=1..T , given initial condition X0 and confounders z, is well posed and satisfies the following constraints:
Definition 1 (Identifiability, (Pearl, 2018)) The experiment (AB,CD, z) is identifiable if, for any set of confounders z′:
ψ(A, z) = ψ(A, z′)⇒ ψ(C, z) = ψ(C, z′). (1)
In an identifiable experiment there is no pair (z, z′) that gives the same trajectory AB but different counterfactual outcomes CD. Details on implementation and impact are in appendix A.1.
Counterfactuality — We enforce sufficient difficulty of the problem through the meaningfulness of confounders. We remove initial situations where the choice of confounder values has no significant impact on the final outcome:
Definition 2 (Counterfactuality). Let zk be the set of confounders z, where the kth value has been modified. The experiment (AB,CD, z) is counterfactual if and only if:
∃k : ψ(C, zk) 6= ψ(C, z). (2)
In other words, we impose the existence of an object of the scene for which the (unobserved) physical properties have a determining effect on the trajectory. Details on how this constraint was enforced are given in appendix A.2.
Temporal resolution — the physical laws we target involve highly non-linear phenomena, in particular collision and resting contacts. Collisions are difficult to learn because their actions are both intense, brief, and highly non-linear, depending on the geometry of the objects in 3D space. The temporal resolution of physical simulations is of prime importance. A parallel can be made with Nyquist-Shannon frequency, as a
trajectory sampled with too low frequency cannot be reconstructed with precision. We simulate and record trajectories at 25 FPS, compared to 5 FPS chosen in CoPhy, justified with two experiments. Firstly, Fig. 2 shows the trajectories of the center of masses of cubes in BlocktowerCF, colored dots are shown at 25 FPS and black dots at 5 FPS. We can see that collisions with the ground fall below the sampling rate of 5 FPS, making it hard to infer physical laws from regularities in data at this frequency. A second experiment involves learning a prediction model at different frequencies, confirming the choice 25 FPS — details are given in appendix A.3.

heading: 4 UNSUPERVISED LEARNING OF COUNTERFACTUAL PHYSICS
text: We introduce a new model for counterfactual learning of physical processes capable of predicting visual sequences D in the image space over long horizons. The method does not require any supervision other than videos of observed and counterfactual experiences. The code is publicly available online at https://filteredcophy.github.io. The model consists of three parts, learning the latent representation and its (counterfactual) dynamics:
• The encoder (De-Rendering module) learns a hybrid representation of an image in the form of a (i) dense feature map and (ii) 2D keypoints combined with (iii) a low-dimensional vector of coef-
t+1 <latexit sha1_base64="UFjzFUjBZZFZA2zFyF3bMc02sQ4=">AAACz3icjVHLSsNAFD2Nr1pfVZdugkUQhJJUQZeiG5ct2Ae0pSTptA7Ni8xEkVJx6w+41b8S/0D/wjtjCmoRnZDkzLn3nJl7rxv7XEjLes0Zc/MLi0v55cLK6tr6RnFzqyGiNPFY3Yv8KGm5jmA+D1ldcumzVpwwJ3B91nRH5yrevGaJ4FF4KW9j1g2cYcgH3HMkUZ2O4MPA6Y3lgT3pFUtW2dLLnAV2BkrIVjUqvqCDPiJ4SBGAIYQk7MOBoKcNGxZi4roYE5cQ4jrOMEGBtCllMcpwiB3Rd0i7dsaGtFeeQqs9OsWnNyGliT3SRJSXEFanmTqeamfF/uY91p7qbrf0dzOvgFiJK2L/0k0z/6tTtUgMcKJr4FRTrBlVnZe5pLor6ubml6okOcTEKdyneELY08ppn02tEbp21VtHx990pmLV3styU7yrW9KA7Z/jnAWNStk+LFdqR6XTs2zUeexgF/s0z2Oc4gJV1Mk7xiOe8GzUjBvjzrj/TDVymWYb35bx8AHpg5QL</latexit> t <latexit sha1_base64="0xZGfCuLYwPeHjwEQMO6b9WJr9M=">AAACzXicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLoxp0V7ANtKcl0WofmRTIRSq1bf8Ct/pb4B/oX3hlTUIvohCRnzr3nzNx73cgTibSs15wxN7+wuJRfLqysrq1vFDe3GkmYxozXWeiFcct1Eu6JgNelkB5vRTF3fNfjTXd4quLNWx4nIgwu5SjiHd8ZBKIvmCOJumonYuA73bGcdIslq2zpZc4COwMlZKsWFl/QRg8hGFL44AggCXtwkNBzDRsWIuI6GBMXExI6zjFBgbQpZXHKcIgd0ndAu+uMDWivPBOtZnSKR29MShN7pAkpLyasTjN1PNXOiv3Ne6w91d1G9HczL59YiRti/9JNM/+rU7VI9HGsaxBUU6QZVR3LXFLdFXVz80tVkhwi4hTuUTwmzLRy2mdTaxJdu+qto+NvOlOxas+y3BTv6pY0YPvnOGdBo1K2D8qVi8NS9SQbdR472MU+zfMIVZyhhjp5B3jEE56NcyM17oz7z1Qjl2m28W0ZDx+7UZOb</latexit>
<latexit sha1_base64="FL+PDwiA9qZAVte/TaU/BhL7sc8=">AAACDXicbVDLSsNAFJ3UV42vqEs3wSp0VZIu1GXBjcsK9gFNKJPJTTt0MgkzE6GE/oAbf8WNC0Xcunfn3zhpg2jrgYHDOfdw554gZVQqx/kyKmvrG5tb1W1zZ3dv/8A6POrKJBMEOiRhiegHWAKjHDqKKgb9VACOAwa9YHJd+L17EJIm/E5NU/BjPOI0ogQrLQ2tMy+AEeU5Aa5AzEzgJAlBmB7w8EccWjWn4cxhrxK3JDVUoj20Pr0wIVms84RhKQeukyo/x0JRwmBmepmEFJMJHsFAU45jkH4+v2Zmn2sltKNE6MeVPVd/J3IcSzmNAz0ZYzWWy14h/ucNMhVd+Tnlaab0mYtFUcZsldhFNXZIBRDFpppgIqj+q03GWGCiS5BFCe7yyauk22y4F43mbbPWqpd1VNEJOkV15KJL1EI3qI06iKAH9IRe0KvxaDwbb8b7YrRilJlj9AfGxze65Jvi</latexit>
encoder
<latexit sha1_base64="FL+PDwiA9qZAVte/TaU/BhL7sc8=">AAACDXicbVDLSsNAFJ3UV42vqEs3wSp0VZIu1GXBjcsK9gFNKJPJTTt0MgkzE6GE/oAbf8WNC0Xcunfn3zhpg2jrgYHDOfdw554gZVQqx/kyKmvrG5tb1W1zZ3dv/8A6POrKJBMEOiRhiegHWAKjHDqKKgb9VACOAwa9YHJd+L17EJIm/E5NU/BjPOI0ogQrLQ2tMy+AEeU5Aa5AzEzgJAlBmB7w8EccWjWn4cxhrxK3JDVUoj20Pr0wIVms84RhKQeukyo/x0JRwmBmepmEFJMJHsFAU45jkH4+v2Zmn2sltKNE6MeVPVd/J3IcSzmNAz0ZYzWWy14h/ucNMhVd+Tnlaab0mYtFUcZsldhFNXZIBRDFpppgIqj+q03GWGCiS5BFCe7yyauk22y4F43mbbPWqpd1VNEJOkV15KJL1EI3qI06iKAH9IRe0KvxaDwbb8b7YrRilJlj9AfGxze65Jvi</latexit>
encoder
<latexit sha1_base64="FL+PDwiA9qZAVte/TaU/BhL7sc8=">AAACDXicbVDLSsNAFJ3UV42vqEs3wSp0VZIu1GXBjcsK9gFNKJPJTTt0MgkzE6GE/oAbf8WNC0Xcunfn3zhpg2jrgYHDOfdw554gZVQqx/kyKmvrG5tb1W1zZ3dv/8A6POrKJBMEOiRhiegHWAKjHDqKKgb9VACOAwa9YHJd+L17EJIm/E5NU/BjPOI0ogQrLQ2tMy+AEeU5Aa5AzEzgJAlBmB7w8EccWjWn4cxhrxK3JDVUoj20Pr0wIVms84RhKQeukyo/x0JRwmBmepmEFJMJHsFAU45jkH4+v2Zmn2sltKNE6MeVPVd/J3IcSzmNAz0ZYzWWy14h/ucNMhVd+Tnlaab0mYtFUcZsldhFNXZIBRDFpppgIqj+q03GWGCiS5BFCe7yyauk22y4F43mbbPWqpd1VNEJOkV15KJL1EI3qI06iKAH9IRe0KvxaDwbb8b7YrRilJlj9AfGxze65Jvi</latexit>
encoder
<latexit sha1_base64="Orjb44rrcAoaMOflHZ+pOpN066A=">AAACDXicbVDLSsNAFJ3UV42vqEs3wSp0VZIu1GXBjcsK9gFNKJPJTTt0MgkzE6GE/oAbf8WNC0Xcunfn3zhpg2jrgYHDOfdw554gZVQqx/kyKmvrG5tb1W1zZ3dv/8A6POrKJBMEOiRhiegHWAKjHDqKKgb9VACOAwa9YHJd+L17EJIm/E5NU/BjPOI0ogQrLQ2tMy+AEeU5Aa5AzMwQSBKCMD3g4Y84tGpOw5nDXiVuSWqoRHtofXphQrJY5wnDUg5cJ1V+joWihMHM9DIJKSYTPIKBphzHIP18fs3MPtdKaEeJ0I8re67+TuQ4lnIaB3oyxmosl71C/M8bZCq68nPK00wBJ4tFUcZsldhFNXZIBRDFpppgIqj+q03GWGCiS5BFCe7yyauk22y4F43mbbPWqpd1VNEJOkV15KJL1EI3qI06iKAH9IRe0KvxaDwbb8b7YrRilJlj9AfGxzeq/ZvY</latexit> decoder
<latexit sha1_base64="6R3Mri7PYTSnRZSyQWyao2eQFoQ=">AAACEXicbVC7TsMwFHXKq5RXgJHFokLqVCUdgLHAwlgk+pDaqHKcm9aq40S2g6ii/gILv8LCAEKsbGz8DW4bIWg5kqWjc+6xfY+fcKa043xZhZXVtfWN4mZpa3tnd8/eP2ipOJUUmjTmsez4RAFnApqaaQ6dRAKJfA5tf3Q19dt3IBWLxa0eJ+BFZCBYyCjRRurblZ4PAyYyCkKDnJR6Gu61H2YXl4aDCH6Mvl12qs4MeJm4OSmjHI2+/dkLYppGJk85UarrOon2MiI1oxzM7amChNARGUDXUEEiUF4222iCT4wS4DCW5giNZ+rvREYipcaRbyYjoodq0ZuK/3ndVIfnXsZEkmoQdP5QmHKsYzytBwdMAtV8bAihkpm/YjokklBTgpqW4C6uvExatap7Wq3d1Mr1Sl5HER2hY1RBLjpDdXSNGqiJKHpAT+gFvVqP1rP1Zr3PRwtWnjlEf2B9fAPXGp2O</latexit>
AB
<latexit sha1_base64="LQ1VmGeF/VhWqd7uuIq+2B7cKOE=">AAACEHicbVC7TsMwFHXKq5RXgJHFokJ0qpIOwFipC2OR6ENqq8pxblqrjhPZDqKK+gks/AoLAwixMrLxNzhthKDlSJaOzrnH9j1ezJnSjvNlFdbWNza3itulnd29/QP78KitokRSaNGIR7LrEQWcCWhppjl0Ywkk9Dh0vEkj8zt3IBWLxK2exjAIyUiwgFGijTS0z/sejJhIKQgNclbqa7jXXpA2DAXh/+hDu+xUnTnwKnFzUkY5mkP7s+9HNAlNnnKiVM91Yj1IidSMcjC3JwpiQidkBD1DBQlBDdL5QjN8ZhQfB5E0R2g8V38nUhIqNQ09MxkSPVbLXib+5/USHVwNUibiRIOgi4eChGMd4awd7DMJVPOpIYRKZv6K6ZhIQk0JKivBXV55lbRrVfeiWrupleuVvI4iOkGnqIJcdInq6Bo1UQtR9ICe0At6tR6tZ+vNel+MFqw8c4z+wPr4BkOCnUQ=</latexit> C
<latexit sha1_base64="cq4NG81Kyap3DwEKDNltOdfsGlQ=">AAACEHicbVC7TsMwFHXKq5RXgJHFokJ0qpIOwFgJBsYi0YfUVpXj3LRWHSeyHUQV9RNY+BUWBhBiZWTjb3DaCEHLkSwdnXOP7Xu8mDOlHefLKqysrq1vFDdLW9s7u3v2/kFLRYmk0KQRj2THIwo4E9DUTHPoxBJI6HFoe+PLzG/fgVQsErd6EkM/JEPBAkaJNtLAPu15MGQipSA0yGmpp+Fee0F6ZSgI/0cf2GWn6syAl4mbkzLK0RjYnz0/oklo8pQTpbquE+t+SqRmlIO5PVEQEzomQ+gaKkgIqp/OFpriE6P4OIikOULjmfo7kZJQqUnomcmQ6JFa9DLxP6+b6OCinzIRJxoEnT8UJBzrCGftYJ9JoJpPDCFUMvNXTEdEEmpKUFkJ7uLKy6RVq7pn1dpNrVyv5HUU0RE6RhXkonNUR9eogZqIogf0hF7Qq/VoPVtv1vt8tGDlmUP0B9bHN0UVnUU=</latexit>
D
<latexit sha1_base64="K5x681+1SSQCD1CiUVLXGsCnMTk=">AAACEnicbVDLSsNAFJ3UV42vqEs3g0Wom5J0oS4LBXFZwT6gDWUyuWmHTiZhZiKU0G9w46+4caGIW1fu/BunD0RbDwwczrmHufcEKWdKu+6XVVhb39jcKm7bO7t7+wfO4VFLJZmk0KQJT2QnIAo4E9DUTHPopBJIHHBoB6P61G/fg1QsEXd6nIIfk4FgEaNEG6nvnPcCGDCRUxAa5MSuX2NQmsVEJ9LugQh/nL5TcivuDHiVeAtSQgs0+s5nL0xoFps85USpruem2s+J1IxymNi9TEFK6IgMoGuoIDEoP5+dNMFnRglxlEjzhMYz9XciJ7FS4zgwk2bXoVr2puJ/XjfT0ZWfM5FmGgSdfxRlHOsET/vBIZNANR8bQqhkZldMh0QSakpQ0xK85ZNXSata8S4q1dtqqVZe1FFEJ+gUlZGHLlEN3aAGaiKKHtATekGv1qP1bL1Z7/PRgrXIHKM/sD6+AS9anbU=</latexit> CF estimator
<latexit sha1_base64="ASsVVHKCaJPs5TVCfA4HZIcyxDw=">AAACDnicbVDLSsNAFJ3UV62vqEs3wVLoqiRdqMuCG5cV7AOaUCaTm3boZBJmJkII/QI3/oobF4q4de3Ov3HSBtHWAwOHc+7hzj1+wqhUtv1lVDY2t7Z3qru1vf2DwyPz+KQv41QQ6JGYxWLoYwmMcugpqhgMEwE48hkM/Nl14Q/uQUga8zuVJeBFeMJpSAlWWhqbDdeHCeU5Aa5AzGtBxnFEiay5wIMfdWzW7Za9gLVOnJLUUYnu2Px0g5ikkc4ThqUcOXaivBwLRQmDec1NJSSYzPAERprqnSC9fHHO3GpoJbDCWOjHlbVQfydyHEmZRb6ejLCaylWvEP/zRqkKr7yc8iRVwMlyUZgyS8VW0Y0VUAFEsUwTTATVf7XIFAtMdAmyKMFZPXmd9Nst56LVvm3XO82yjio6Q+eoiRx0iTroBnVRDxH0gJ7QC3o1Ho1n4814X45WjDJziv7A+PgGpb6cZA==</latexit>
dynamics
<latexit sha1_base64="br3grPl3n8NADB8782iahWzTCdM=">AAACE3icbVC7TsMwFHV4lvIKMLJEVEgVQ5V0AMZKZWAsEn1IbVQ5zk1r1XEi20FEUf+BhV9hYQAhVhY2/ganjRC0XMnS8Tn3XPseL2ZUKtv+MlZW19Y3Nktb5e2d3b198+CwI6NEEGiTiEWi52EJjHJoK6oY9GIBOPQYdL1JM9e7dyAkjfitSmNwQzziNKAEK00NzbOBByPKMwJcgZiWBwrulRdkzegq1Tfg/o80NCt2zZ6VtQycAlRQUa2h+TnwI5KE2k8YlrLv2LFyMywUJQz09ERCjMkEj6CvIcchSDeb7TS1TjXjW0Ek9OHKmrG/HRkOpUxDT3eGWI3lopaT/2n9RAWXbkZ5nCjgZP5QkDBLRVYekOVTAUSxVANMBNV/tcgYC0x0CDIPwVlceRl06jXnvFa/qVca1SKOEjpGJ6iKHHSBGugatVAbEfSAntALejUejWfjzXift64YhecI/Snj4xupmp6O</latexit>
CoDy
<latexit sha1_base64="3GzDvZurdPxBKdxeDIGosntvvbM=">AAACCnicbVDLSsNAFJ3UV42vqEs30SJ0VZIu1GXBjcsK9gFNKJPJTTt0MgkzE6GErt34K25cKOLWL3Dn3zhpg2jrgYHDOfdw554gZVQqx/kyKmvrG5tb1W1zZ3dv/8A6POrKJBMEOiRhiegHWAKjHDqKKgb9VACOAwa9YHJd+L17EJIm/E5NU/BjPOI0ogQrLQ2tUy+AEeU5Aa5AzEwFUpke8PBHGVo1p+HMYa8StyQ1VKI9tD69MCFZrPOEYSkHrpMqP8dCUcJgZnqZhBSTCR7BQFOOY5B+Pj9lZp9rJbSjROjHlT1XfydyHEs5jQM9GWM1lsteIf7nDTIVXfk55WmmgJPFoihjtkrsohc7pAKIYlNNMBFU/9UmYyww0SXIogR3+eRV0m023ItG87ZZa9XLOqroBJ2hOnLRJWqhG9RGHUTQA3pCL+jVeDSejTfjfTFaMcrMMfoD4+MbbxqapA==</latexit>
test
<latexit sha1_base64="0bPwfA3v6EK6uBQ7Oy9bsfa3SyY=">AAACC3icbVDLSsNAFJ3UV42vqEs3oUXoqiRdqMuCG5cV7AOaUCaTm3boZBJmJkIJ3bvxV9y4UMStP+DOv3HSBtHWAwOHc+7hzj1ByqhUjvNlVDY2t7Z3qrvm3v7B4ZF1fNKTSSYIdEnCEjEIsARGOXQVVQwGqQAcBwz6wfS68Pv3ICRN+J2apeDHeMxpRAlWWhpZNS+AMeU5Aa5AzE0lMOWmBzz8kUZW3Wk6C9jrxC1JHZXojKxPL0xIFus8YVjKoeukys+xUJQwmJteJiHFZIrHMNSU4xikny9umdvnWgntKBH6cWUv1N+JHMdSzuJAT8ZYTeSqV4j/ecNMRVd+TnmaKeBkuSjKmK0SuyjGDqkAothME0wE1X+1yQQLTHQJsijBXT15nfRaTfei2bpt1duNso4qOkM11EAuukRtdIM6qIsIekBP6AW9Go/Gs/FmvC9HK0aZOUV/YHx8Ay+kmww=</latexit>
train
<latexit sha1_base64="2194DdG5/8bRXRO0lRS2SUgokE8=">AAACCXicdZDLSgMxFIYz9VbrbdSlm2ARuirTItrZFURwI1S0F+iUkklP29BMZkgyQhm6deOruHGhiFvfwJ1vY6Yt9YL+EPj5zjmcnN+POFPacT6szNLyyupadj23sbm1vWPv7jVUGEsKdRryULZ8ooAzAXXNNIdWJIEEPoemPzpL681bkIqF4kaPI+gEZCBYn1GiDera2PNhwERCQWiQk9zl9XnOA9FbgK6dd4qu61bcMi4VnanwjBx/kTyaq9a1371eSOPAzFNOlGqXnEh3EiI1oxwmOS9WEBE6IgNoGytIAKqTTC+Z4CNDergfSvOExlP6fSIhgVLjwDedAdFD9buWwr9q7Vj3K52EiSjWIOhsUT/mWIc4jQX3mASq+dgYQiUzf8V0SCShJgSVhrC4/X/TKBdLJ8XyVTlfLczjyKIDdIgKqIROURVdoBqqI4ru0AN6Qs/WvfVovVivs9aMNZ/ZRz9kvX0Cag2aFw==</latexit> MSE
<latexit sha1_base64="6R3Mri7PYTSnRZSyQWyao2eQFoQ=">AAACEXicbVC7TsMwFHXKq5RXgJHFokLqVCUdgLHAwlgk+pDaqHKcm9aq40S2g6ii/gILv8LCAEKsbGz8DW4bIWg5kqWjc+6xfY+fcKa043xZhZXVtfWN4mZpa3tnd8/eP2ipOJUUmjTmsez4RAFnApqaaQ6dRAKJfA5tf3Q19dt3IBWLxa0eJ+BFZCBYyCjRRurblZ4PAyYyCkKDnJR6Gu61H2YXl4aDCH6Mvl12qs4MeJm4OSmjHI2+/dkLYppGJk85UarrOon2MiI1oxzM7amChNARGUDXUEEiUF4222iCT4wS4DCW5giNZ+rvREYipcaRbyYjoodq0ZuK/3ndVIfnXsZEkmoQdP5QmHKsYzytBwdMAtV8bAihkpm/YjokklBTgpqW4C6uvExatap7Wq3d1Mr1Sl5HER2hY1RBLjpDdXSNGqiJKHpAT+gFvVqP1rP1Zr3PRwtWnjlEf2B9fAPXGp2O</latexit>
AB
<latexit sha1_base64="LQ1VmGeF/VhWqd7uuIq+2B7cKOE=">AAACEHicbVC7TsMwFHXKq5RXgJHFokJ0qpIOwFipC2OR6ENqq8pxblqrjhPZDqKK+gks/AoLAwixMrLxNzhthKDlSJaOzrnH9j1ezJnSjvNlFdbWNza3itulnd29/QP78KitokRSaNGIR7LrEQWcCWhppjl0Ywkk9Dh0vEkj8zt3IBWLxK2exjAIyUiwgFGijTS0z/sejJhIKQgNclbqa7jXXpA2DAXh/+hDu+xUnTnwKnFzUkY5mkP7s+9HNAlNnnKiVM91Yj1IidSMcjC3JwpiQidkBD1DBQlBDdL5QjN8ZhQfB5E0R2g8V38nUhIqNQ09MxkSPVbLXib+5/USHVwNUibiRIOgi4eChGMd4awd7DMJVPOpIYRKZv6K6ZhIQk0JKivBXV55lbRrVfeiWrupleuVvI4iOkGnqIJcdInq6Bo1UQtR9ICe0At6tR6tZ+vNel+MFqw8c4z+wPr4BkOCnUQ=</latexit> C
<latexit sha1_base64="cq4NG81Kyap3DwEKDNltOdfsGlQ=">AAACEHicbVC7TsMwFHXKq5RXgJHFokJ0qpIOwFgJBsYi0YfUVpXj3LRWHSeyHUQV9RNY+BUWBhBiZWTjb3DaCEHLkSwdnXOP7Xu8mDOlHefLKqysrq1vFDdLW9s7u3v2/kFLRYmk0KQRj2THIwo4E9DUTHPoxBJI6HFoe+PLzG/fgVQsErd6EkM/JEPBAkaJNtLAPu15MGQipSA0yGmpp+Fee0F6ZSgI/0cf2GWn6syAl4mbkzLK0RjYnz0/oklo8pQTpbquE+t+SqRmlIO5PVEQEzomQ+gaKkgIqp/OFpriE6P4OIikOULjmfo7kZJQqUnomcmQ6JFa9DLxP6+b6OCinzIRJxoEnT8UJBzrCGftYJ9JoJpPDCFUMvNXTEdEEmpKUFkJ7uLKy6RVq7pn1dpNrVyv5HUU0RE6RhXkonNUR9eogZqIogf0hF7Qq/VoPVtv1vt8tGDlmUP0B9bHN0UVnUU=</latexit>
D
<latexit sha1_base64="cq4NG81Kyap3DwEKDNltOdfsGlQ=">AAACEHicbVC7TsMwFHXKq5RXgJHFokJ0qpIOwFgJBsYi0YfUVpXj3LRWHSeyHUQV9RNY+BUWBhBiZWTjb3DaCEHLkSwdnXOP7Xu8mDOlHefLKqysrq1vFDdLW9s7u3v2/kFLRYmk0KQRj2THIwo4E9DUTHPoxBJI6HFoe+PLzG/fgVQsErd6EkM/JEPBAkaJNtLAPu15MGQipSA0yGmpp+Fee0F6ZSgI/0cf2GWn6syAl4mbkzLK0RjYnz0/oklo8pQTpbquE+t+SqRmlIO5PVEQEzomQ+gaKkgIqp/OFpriE6P4OIikOULjmfo7kZJQqUnomcmQ6JFa9DLxP6+b6OCinzIRJxoEnT8UJBzrCGftYJ9JoJpPDCFUMvNXTEdEEmpKUFkJ7uLKy6RVq7pn1dpNrVyv5HUU0RE6RhXkonNUR9eogZqIogf0hF7Qq/VoPVtv1vt8tGDlmUP0B9bHN0UVnUU=</latexit>
D
<latexit sha1_base64="RTAuI0/lGHg/ovhK8kP52smBKPs=">AAACFHicbVDLSsNAFJ3UV42vqEs3wSIUhJJ0oS4LblxWsA9oQplMbtqhk0mYmQgl9CPc+CtuXCji1oU7/8ZJG0RbDwwczrmHO/cEKaNSOc6XUVlb39jcqm6bO7t7+wfW4VFXJpkg0CEJS0Q/wBIY5dBRVDHopwJwHDDoBZPrwu/dg5A04XdqmoIf4xGnESVYaWlonXsBjCjPCXAFYmYKIAmXSmSk8E0PePjjDa2a03DmsFeJW5IaKtEeWp9emJAs1nnCsJQD10mVn2OhKGEwM71MQorJBI9goCnHMUg/nx81s8+0EtpRIvTjyp6rvxM5jqWcxoGejLEay2WvEP/zBpmKrvyc8jRTwMliUZQxWyV20ZAdUl2CYlNNMBFU/9UmYyww0SXIogR3+eRV0m023ItG87ZZa9XLOqroBJ2iOnLRJWqhG9RGHUTQA3pCL+jVeDSejTfjfTFaMcrMMfoD4+Mb81qfSg==</latexit>
reconstruction
<latexit sha1_base64="zyDwLF1Zzh8supV97s76j7Juw8s=">AAACGXicbVDLSgMxFM34rPU16tJNsAhdlZku1GWhG5cV7APaUjLpnTY0kxmSO2IZ+htu/BU3LhRxqSv/xvSBaOuBwOGcc5PcEyRSGPS8L2dtfWNzazu3k9/d2z84dI+OGyZONYc6j2WsWwEzIIWCOgqU0Eo0sCiQ0AxG1anfvANtRKxucZxAN2IDJULBGVqp53qdAAZCZRwUgp7kQ2CYajC0g3CPQZhVJ/kOqP5PoOcWvJI3A10l/oIUyAK1nvvR6cc8jew8l8yYtu8l2M2YRsEl2NtTAwnjIzaAtqWKRWC62WyzCT23Sp+GsbZHIZ2pvycyFhkzjgKbjBgOzbI3Ff/z2imGV91MqCRFUHz+UJhKijGd1kT7QgNHObaEcS3sXykfMs24LcFMS/CXV14ljXLJvyiVb8qFSnFRR46ckjNSJD65JBVyTWqkTjh5IE/khbw6j86z8+a8z6NrzmLmhPyB8/kNeEahHQ==</latexit>
features C<latexit sha1_base64="zyDwLF1Zzh8supV97s76j7Juw8s=">AAACGXicbVDLSgMxFM34rPU16tJNsAhdlZku1GWhG5cV7APaUjLpnTY0kxmSO2IZ+htu/BU3LhRxqSv/xvSBaOuBwOGcc5PcEyRSGPS8L2dtfWNzazu3k9/d2z84dI+OGyZONYc6j2WsWwEzIIWCOgqU0Eo0sCiQ0AxG1anfvANtRKxucZxAN2IDJULBGVqp53qdAAZCZRwUgp7kQ2CYajC0g3CPQZhVJ/kOqP5PoOcWvJI3A10l/oIUyAK1nvvR6cc8jew8l8yYtu8l2M2YRsEl2NtTAwnjIzaAtqWKRWC62WyzCT23Sp+GsbZHIZ2pvycyFhkzjgKbjBgOzbI3Ff/z2imGV91MqCRFUHz+UJhKijGd1kT7QgNHObaEcS3sXykfMs24LcFMS/CXV14ljXLJvyiVb8qFSnFRR46ckjNSJD65JBVyTWqkTjh5IE/khbw6j86z8+a8z6NrzmLmhPyB8/kNeEahHQ==</latexit> features C
<latexit sha1_base64="XwH2KonjIsArLOelrX/d1a6hDzE=">AAACGnicbVDLSgMxFM3UV62vqks3wSJ0VWa6UJdVNy4r2Ae0pWTSO21oJjMkd8Qy9Dvc+CtuXCjiTtz4N6YPRFsPBA7nnJvkHj+WwqDrfjmZldW19Y3sZm5re2d3L79/UDdRojnUeCQj3fSZASkU1FCghGasgYW+hIY/vJr4jTvQRkTqFkcxdELWVyIQnKGVunmv7UNfqJSDQtDjXAAMEw2GthHu0Q/Si8txrg2q95Po5gtuyZ2CLhNvTgpkjmo3/9HuRTwJ7TyXzJiW58bYSZlGwSXY2xMDMeND1oeWpYqFYDrpdLUxPbFKjwaRtkchnaq/J1IWGjMKfZsMGQ7MojcR//NaCQbnnVSoOEFQfPZQkEiKEZ30RHtCA0c5soRxLexfKR8wzbgtwUxK8BZXXib1csk7LZVvyoVKcV5HlhyRY1IkHjkjFXJNqqRGOHkgT+SFvDqPzrPz5rzPohlnPnNI/sD5/AYPfqFn</latexit>
features AB
<latexit sha1_base64="maLvmXGlyY5vqEh0Mp2cx+bAmjY=">AAACGXicbVDLSgMxFM34rPVVdekmWISuykwX6rKgC5cV7APaUjLpnTY0kxmSO2IZ+htu/BU3LhRxqSv/xkxbRFsPBA7nnJvkHj+WwqDrfjkrq2vrG5u5rfz2zu7efuHgsGGiRHOo80hGuuUzA1IoqKNACa1YAwt9CU1/dJn5zTvQRkTqFscxdEM2UCIQnKGVegW348NAqJSDQtCTfAAMEw2GdhDu0Q/Sq0m+A6r/E+gVim7ZnYIuE29OimSOWq/w0elHPAntPJfMmLbnxthNmUbBJdjbEwMx4yM2gLalioVguul0swk9tUqfBpG2RyGdqr8nUhYaMw59mwwZDs2il4n/ee0Eg4tuKlScICg+eyhIJMWIZjXRvtDAUY4tYVwL+1fKh0wzbkswWQne4srLpFEpe2flyk2lWC3N68iRY3JCSsQj56RKrkmN1AknD+SJvJBX59F5dt6c91l0xZnPHJE/cD6/AXnZoR4=</latexit>
features D
<latexit sha1_base64="L7asew2wQQa5o+amE+sbX9aAEgU=">AAACCnicbVDLSsNAFJ3UV42vqEs30SJ0VZIu1GXBjcsK9gFNKJPJTTt0MgkzE6GErt34K25cKOLWL3Dn3zhpg2jrgYHDOfdw554gZVQqx/kyKmvrG5tb1W1zZ3dv/8A6POrKJBMEOiRhiegHWAKjHDqKKgb9VACOAwa9YHJd+L17EJIm/E5NU/BjPOI0ogQrLQ2tUy+AEeU5Aa5AzEyWSGl6wMMfZWjVnIYzh71K3JLUUIn20Pr0woRksc4ThqUcuE6q/BwLRQmDmellElJMJngEA005jkH6+fyUmX2uldCOEqEfV/Zc/Z3IcSzlNA70ZIzVWC57hfifN8hUdOXnlKeZAk4Wi6KM2Sqxi17skAogik01wURQ/VebjLHARJcgixLc5ZNXSbfZcC8azdtmrVUv66iiE3SG6shFl6iFblAbdRBBD+gJvaBX49F4Nt6M98VoxSgzx+gPjI9vcKiapQ==</latexit>
loss <latexit sha1_base64="uQJl1HkK/GSm3eRQevwtYYn9juM=">AAACIHicbVDLSgMxFM3UVx1fVZdugkXoqsx0YV0W3LisYB/QDiWTudOGZpIhyQhl6Ke48VfcuFBEd/o1pg9EWw8EDuecy809YcqZNp736RQ2Nre2d4q77t7+weFR6fikrWWmKLSo5FJ1Q6KBMwEtwwyHbqqAJCGHTji+nvmde1CaSXFnJikECRkKFjNKjJUGpXo/hCETOQVhQE3dMUxSyYTRmIgIUwmxzTJrarcPIvrJDUplr+rNgdeJvyRltERzUProR5JmiZ2nnGjd873UBDlRhlEOU7efaUgJHZMh9CwVJAEd5PMDp/jCKhGOpbJPGDxXf0/kJNF6koQ2mRAz0qveTPzP62UmvgpyJtLMgKCLRXHGsZF41haOmAJq+MQSQhWzf8V0RBShtgQ9K8FfPXmdtGtV/7Jau62VG5VlHUV0hs5RBfmojhroBjVRC1H0gJ7QC3p1Hp1n5815X0QLznLmFP2B8/UND5ikCw==</latexit> keypoints and coe cients
<latexit sha1_base64="Hpf/zL23FA8DOGK0Itkxwy81mnk=">AAACGHicbVDLSgMxFM3UVx1foy7dBIvQVZ3pQsVVwY3LCrYVOkPJZO60oZnMkGSEMvQz3Pgrblwo4rY7/8b0gWjrgcDhnHuS3BNmnCntul9WaW19Y3OrvG3v7O7tHziHR22V5pJCi6Y8lQ8hUcCZgJZmmsNDJoEkIYdOOLyZ+p1HkIql4l6PMggS0hcsZpRoI/Wccz+EPhMFBaFBju35xZhkGRBJBIVr2wcR/fg9p+LW3BnwKvEWpIIWaPaciR+lNE9MnnKiVNdzMx0URGpGOYxtP1eQETokfegaKkgCKihmi43xmVEiHKfSHKHxTP2dKEii1CgJzWRC9EAte1PxP6+b6/gqKJjIcg2Czh+Kc451iqct4YhJoJqPDCFUMvNXTAemEGpKUNMSvOWVV0m7XvMuavW7eqVRXdRRRifoFFWRhy5RA92iJmohip7QC3pD79az9Wp9WJ/z0ZK1yByjP7Am3z3noGs=</latexit>
source appearance:
<latexit sha1_base64="tRKMOOH8EMjRMIulKRUKumYtkb8=">AAACDXicbVDLSsNAFJ34rPEVdekmWIWuStKFiquCG5cV7APaUCaTm3ToZBJmJkIJ/QE3/oobF4q4de/Ov3HSBtHWAwOHc+7hzj1+yqhUjvNlrKyurW9sVrbM7Z3dvX3r4LAjk0wQaJOEJaLnYwmMcmgrqhj0UgE49hl0/fF14XfvQUia8Ds1ScGLccRpSAlWWhpapwMfIspzAlyBmJoKiwjUlTkAHvyIQ6vq1J0Z7GXilqSKSrSG1ucgSEgW6zxhWMq+66TKy7FQlDCYmoNMQorJGEfQ15TjGKSXz66Z2mdaCewwEfpxZc/U34kcx1JOYl9PxliN5KJXiP95/UyFl15OeZop4GS+KMyYrRK7qMYOqACi2EQTTATVf7XJCAtMdAmyKMFdPHmZdBp197zeuG1Um7Wyjgo6Rieohlx0gZroBrVQGxH0gJ7QC3o1Ho1n4814n4+uGGXmCP2B8fENioSbww==</latexit>
target:
<latexit sha1_base64="+aeiwbn8kJg8LWYtvouWc47+GTA=">AAACHXicbVDLSgMxFM3UV62vUZduBovQVZkpoi4LdeGygn1Ip5RM5k4bmkmGJCOWoT/ixl9x40IRF27EvzF9INp6IHA4515uzgkSRpV23S8rt7K6tr6R3yxsbe/s7tn7B00lUkmgQQQTsh1gBYxyaGiqGbQTCTgOGLSCYW3it+5AKir4jR4l0I1xn9OIEqyN1LNP/QD6lGcEuAY5Lvga7nUQZTVxeTv2/QJwIkKQBR94+DPUs4tu2Z3CWSbenBTRHPWe/eGHgqSx2ScMK9Xx3ER3Myw1JQzM1VRBgskQ96FjKMcxqG42TTd2TowSOpGQ5nHtTNXfGxmOlRrFgZmMsR6oRW8i/ud1Uh1ddDPKk1SbmLNDUcocLZxJVU5IJRDNRoZgIqn5q0MGWGJiSlCTErzFyMukWSl7Z+XKdaVYLc3ryKMjdIxKyEPnqIquUB01EEEP6Am9oFfr0Xq23qz32WjOmu8coj+wPr8BH3aidA==</latexit>
CoDY encoder
<latexit sha1_base64="j+zk/qYHBOzhDjjdTzhO8D4S6Xc=">AAACHXicbVDLSgMxFM3UV62vUZduBovQVZkpoi4LdeGygn1Ip5RM5k4bmkmGJCOWoT/ixl9x40IRF27EvzF9INp6IHA451xu7gkSRpV23S8rt7K6tr6R3yxsbe/s7tn7B00lUkmgQQQTsh1gBYxyaGiqGbQTCTgOGLSCYW3it+5AKir4jR4l0I1xn9OIEqyN1LNP/QD6lGcEuAY5Lvga7nUQZTVxeTv2/UIIRIQgCz7w8CfUs4tu2Z3CWSbenBTRHPWe/eGHgqSxmScMK9Xx3ER3Myw1JQzM1lRBgskQ96FjKMcxqG42vW7snBgldCIhzePamaq/JzIcKzWKA5OMsR6oRW8i/ud1Uh1ddDPKk1QDJ7NFUcocLZxJVU5IJRDNRoZgIqn5q0MGWGJiSlCTErzFk5dJs1L2zsqV60qxWprXkUdH6BiVkIfOURVdoTpqIIIe0BN6Qa/Wo/VsvVnvs2jOms8coj+wPr8BD4+iag==</latexit>
CoDY
decoder
E <latexit sha1_base64="3kNOWTAFvBUVsIVywMumQR8pU80=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LIogssK9gFtkcl02obmRTIplFLc+gNu9bPEP9C/8M6YglpEJyQ5c+45d+be60Sem0jLes0ZS8srq2v59cLG5tb2TnF3r5GEacxFnYdeGLcclgjPDURdutITrSgWzHc80XRGlyreHIs4ccPgVk4i0fXZIHD7LmeSqHbHZ3LImTe9mt0VS1bZ0stcBHYGSshWLSy+oIMeQnCk8CEQQBL2wJDQ04YNCxFxXUyJiwm5Oi4wQ4G8KakEKRixI/oOaNfO2ID2Kmei3ZxO8eiNyWniiDwh6WLC6jRTx1OdWbG/5Z7qnOpuE/o7WS6fWIkhsX/55sr/+lQtEn2c6xpcqinSjKqOZ1lS3RV1c/NLVZIyRMQp3KN4TJhr57zPpvYkunbVW6bjb1qpWLXnmTbFu7olDdj+Oc5F0KiU7ZNy5ea0VL3IRp3HAQ5xTPM8QxXXqKGuO/6IJzwbNWNszIz7T6mRyzz7+LaMhw9xDJPg</latexit>
<latexit sha1_base64="z9DMvvhNeK7HTsX+YTC7OdTorFQ=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LKoC8FNBfuAtkiSTuvYycNkItbiyh9wqz8m/oH+hXfGCGoRnZDkzLn3nJl7rxsJnkjLeskZU9Mzs3P5+cLC4tLySnF1rZGEaeyxuheKMG65TsIED1hdcilYK4qZ47uCNd3hoYo3r1mc8DA4k6OIdX1nEPA+9xxJVKNzxIR0zoslq2zpZU4COwMlZKsWFp/RQQ8hPKTwwRBAEhZwkNDThg0LEXFdjImLCXEdZ7hDgbQpZTHKcIgd0ndAu3bGBrRXnolWe3SKoDcmpYkt0oSUFxNWp5k6nmpnxf7mPdae6m4j+ruZl0+sxAWxf+k+M/+rU7VI9LGva+BUU6QZVZ2XuaS6K+rm5peqJDlExCnco3hM2NPKzz6bWpPo2lVvHR1/1ZmKVXsvy03xpm5JA7Z/jnMSNCple6dcOd0tVQ+yUeexgU1s0zz3UMUxaqiT9yUe8Ign48S4Mm6M249UI5dp1vFtGffveCCRgQ==</latexit>
<latexit sha1_base64="2194DdG5/8bRXRO0lRS2SUgokE8=">AAACCXicdZDLSgMxFIYz9VbrbdSlm2ARuirTItrZFURwI1S0F+iUkklP29BMZkgyQhm6deOruHGhiFvfwJ1vY6Yt9YL+EPj5zjmcnN+POFPacT6szNLyyupadj23sbm1vWPv7jVUGEsKdRryULZ8ooAzAXXNNIdWJIEEPoemPzpL681bkIqF4kaPI+gEZCBYn1GiDera2PNhwERCQWiQk9zl9XnOA9FbgK6dd4qu61bcMi4VnanwjBx/kTyaq9a1371eSOPAzFNOlGqXnEh3EiI1oxwmOS9WEBE6IgNoGytIAKqTTC+Z4CNDergfSvOExlP6fSIhgVLjwDedAdFD9buWwr9q7Vj3K52EiSjWIOhsUT/mWIc4jQX3mASq+dgYQiUzf8V0SCShJgSVhrC4/X/TKBdLJ8XyVTlfLczjyKIDdIgKqIROURVdoBqqI4ru0AN6Qs/WvfVovVivs9aMNZ/ZRz9kvX0Cag2aFw==</latexit> MSE
uk <latexit sha1_base64="blHMIbrT4YWFISR/tO0iQ4+Jv20=">AAACz3icjVHLTsJAFD3UF+ILdemmkZi4IgVNdEl04xISeSRASFsGmNBXplMNIRi3/oBb/SvjH+hfeGcsiUqMTtP2zLn3nJl7rxN5PJaW9ZoxlpZXVtey67mNza3tnfzuXiMOE+Gyuht6oWg5dsw8HrC65NJjrUgw23c81nTGlyrevGEi5mFwLScR6/r2MOAD7tqSqE7Ht+XIGUyTWW/cyxesoqWXuQhKKSggXdUw/4IO+gjhIoEPhgCSsAcbMT1tlGAhIq6LKXGCENdxhhlypE0oi1GGTeyYvkPatVM2oL3yjLXapVM8egUpTRyRJqQ8QVidZup4op0V+5v3VHuqu03o76RePrESI2L/0s0z/6tTtUgMcK5r4FRTpBlVnZu6JLor6ubml6okOUTEKdynuCDsauW8z6bWxLp21Vtbx990pmLV3k1zE7yrW9KASz/HuQga5WLppFiunRYqF+moszjAIY5pnmeo4ApV1Mk7wiOe8GzUjFvjzrj/TDUyqWYf35bx8AH2sZR8</latexit>
<latexit sha1_base64="7ZEDa/tu3TkRsM9b+CrwjF39J6Y=">AAAB8HicdVDLSgMxFM34rPVVdekmWARXQ2ZsbZdFNy4r2Ie0Q8lk0jY0yQxJRihDv8KNC0Xc+jnu/BszbQUVPXDhcM693HtPmHCmDUIfzsrq2vrGZmGruL2zu7dfOjhs6zhVhLZIzGPVDbGmnEnaMsxw2k0UxSLktBNOrnK/c0+VZrG8NdOEBgKPJBsygo2V7rSJEzhSOBqUysitnteqXgUiF3m+j+o5QdVapQI9S3KUwRLNQem9H8UkFVQawrHWPQ8lJsiwMoxwOiv2U00TTCZ4RHuWSiyoDrL5wTN4apUIDmNlSxo4V79PZFhoPRWh7RTYjPVvLxf/8nqpGdaDjMkkNVSSxaJhyqGJYf49jJiixPCpJZgoZm+FZIwVJsZmVLQhfH0K/ydt3/UuXP/GLzcul3EUwDE4AWfAAzXQANegCVqAAAEewBN4dpTz6Lw4r4vWFWc5cwR+wHn7BBBdkJc=</latexit> stop grad
<latexit sha1_base64="7ZEDa/tu3TkRsM9b+CrwjF39J6Y=">AAAB8HicdVDLSgMxFM34rPVVdekmWARXQ2ZsbZdFNy4r2Ie0Q8lk0jY0yQxJRihDv8KNC0Xc+jnu/BszbQUVPXDhcM693HtPmHCmDUIfzsrq2vrGZmGruL2zu7dfOjhs6zhVhLZIzGPVDbGmnEnaMsxw2k0UxSLktBNOrnK/c0+VZrG8NdOEBgKPJBsygo2V7rSJEzhSOBqUysitnteqXgUiF3m+j+o5QdVapQI9S3KUwRLNQem9H8UkFVQawrHWPQ8lJsiwMoxwOiv2U00TTCZ4RHuWSiyoDrL5wTN4apUIDmNlSxo4V79PZFhoPRWh7RTYjPVvLxf/8nqpGdaDjMkkNVSSxaJhyqGJYf49jJiixPCpJZgoZm+FZIwVJsZmVLQhfH0K/ydt3/UuXP/GLzcul3EUwDE4AWfAAzXQANegCVqAAAEewBN4dpTz6Lw4r4vWFWc5cwR+wHn7BBBdkJc=</latexit> stop grad
ficients, see Fig. 3. Without any state supervision we show that the model learns a representation which encodes positions in keypoints and appearan e and orientation in the coefficients.
• The Counterfactual Dynamic (CoDy) model based on ecurrent graph net orks, in the lines of (Baradel et al., 2020). It estimates a latent representation of th confounders z from the keypoint + coefficient trajector es of AB provided by the encoder, and then predicts D in this same space.
• The decoder that uses the predicted keypoints to generate a pixel-space representation of D.

heading: 4.1 DISENTANGLING VISUAL INFORMATION FROM DYNAMICS
text: The encoder takes an input image X and predicts a representation with three streams, sharing a common conv. backbone, as shown in Fig. 3. We propose an unsupervised training objective, which favors the emergence of a latent representation disentangling static and dynamic information.
1. A dense feature map F = F(X), which contains static information, such as the background. 2. A set of 2D keypoints k = K(X), which carry positional information from moving objects. 3. A set of corresponding coefficients c = C(X), one vector ck of size C per keypoint k, which
encodes orientation and appearance information.
The unsupervised objective is formulated on pairs of images (Xsource,Xtarget) randomly sampled from same D sequences (see appendix D.1 for details on sampling). Exploiting an assumption on the absence of camera motion1, the goal is to favor the emergence of disentangling static and dynamic information. To this end, both images are encoded, and the reconstruction of the target image is predicted with a decoder D fusing the source dense feature map and the target keypoints and coefficients. This formulation requires the decoder to aggregate dense information from the source and sparse values from the target, naturally leading to motion being predicted by the latter.
On the decoderD side, we add inductive bias, which favors the usage of the 2D keypoint information in a spatial way. The 2D coordinates kk for each keypoint k are encoded as Gaussian heatmaps G(kk), i.e. 2D Gaussian functions centered on the keypoint position. The additional coefficient information, carrying appearance information, is then used to deform the Gaussian mapping into an anisotropic shape using a fixed filter bank H, as follows:
D(F,k, c) = R ( F,G11, ...,G C 1 ,G 1 2, ...,G C K ) , Gik = c C+1 k c i k (G(kk) ∗Hi) , (3)
where R(...) is a refinement network performing trained upsampling with transposed convolutions, whose inputs are stacked channelwise. Gik are Gaussian mappings produced from keypoint positions k, deformed by filters from bank H and weighted by coefficients cik. The filters Hi are defined as fixed horizontal, vertical and diagonal convolution kernels. This choice is discussed in section 5. The joint encoding and decoding pipeline is illustrated in Fig. 3.
1If this assumption is not satisfied, global camera motion could be compensated after estimation.
The model is trained to minimize the mean squared error (MSE) reconstruction loss, regularized with a loss on spatial gradients∇X weighted by hyper-parameters γ1, γ2 ∈ R:
Lderen = γ1 ∣∣∣∣∣∣Xtarget − X̂target∣∣∣∣∣∣2
2 + γ2 ∣∣∣∣∣∣∇Xtarget −∇X̂target∣∣∣∣∣∣2 2 , (4)
where X̂target=D(F(Xsource),K(Xtarget), C(Xtarget)) is the reconstructed image, γ1, γ2 are weights. Related work – our unsupervised objective is somewhat related to Transporter (Kulkarni et al., 2019), which, as our model, computes visual feature vectors Fsource and Ftarget as well as 2D keypoints Ksource and Ktarget, modeled as a 2D vector via Gaussian mapping. It leverages a handcrafted transport equation: Ψ̂target=Fsource × (1−Ksource)×(1−Ktarget) + Ftarget×Ktarget. As in our case, the target image is reconstructed through a refiner network X̂target = R(Ψ̂target). The transporter suffers from a major drawback when used for video prediction, as it requires parts of the target image to reconstruct the target image — the model was originally proposed in the context of RL and control, where reconstruction is not an objective. It also does not use shape coefficients, requiring shapes to be encoded by several keypoints, or abusively be carried through the dense features Ftarget. This typically leads to complex dynamics non representative of the dynamical objects. We conducted an in-depth comparison between the Transporter and our representation in appendix C.2.

heading: 4.2 DYNAMIC MODEL AND CONFOUNDER ESTIMATION
text: Our counterfactual dynamic model (CoDy) leverages multiple graph network (GN) based modules (Battaglia et al., 2016) that join forces to solve the counterfactual forecasting tasks of FilteredCoPhy. Each one of these networks is a classical GN, abbreviated as GN (xk), which contextualizes input node embeddings xk through incoming edge interactions eik, providing output node embeddings x̂k (parameters are not shared over the instances):
GN (xk) = x̂k, such that x̂k = g ( xk, ∑ i eik ) with eij = f (xi,xj) , (5)
where f is a message-passing function and g is an aggregation function.
We define the state of frame Xt at time t as a stacked vector composed of keypoints and coefficients computed by the encoder (the de-rendering module), i.e. s(t) = [s1(t) ... sk(t)] where sk(t) = [kk c 1 k ...c C+1 k ](t). In the lines of (Baradel et al., 2020), given the original initial condition and outcome AB, CoDy estimates an unsupervised representation uk of the latent confounder variables per keypoint k through the counterfactual estimator (CF estimator in Fig. 4). It first contextualizes the sequence sAB(t) through a graph network GN ( sAB(t) ) = hAB(t) = [h1(t) ... hK(t)]. We then model the temporal evolution of this representation with a gated recurrent unit (Cho et al., 2014) per keypoint, sharing parameters over keypoints, taking as input the sequence hk. Its last hidden vector is taken as the confounder estimate uk.
Recent works on the Koopman Operator (Lusch et al., 2018) and Kazantzis-Kravaris-Luenberger Observer (Janny et al., 2021; Peralez & Nadri, 2021) have theoretically shown that, under mild assumptions, there exists a latent space of higher dimension, where a dynamical system given as an EDP can have a simpler dynamics. Inspired by this idea, we used an encoder-decoder structure within CoDy, which projects our dynamic system into a higher-dimensional state space, performs forecasting of the dynamics in this latent space, and then projects predictions back to the original keypoint space. Note that this dynamics encoder/decoder is different from the encoder/decoder of the de-rendering/rendering modules discussed in section 4.1. The state encoder E(s(t)) = σ(t) is modeled as a graph network GN , whose aggregation function projects into an output embedding space σ(t) of dimension 256. The decoder ∆(σ(t)) = s(t) temporally processes the individual contextualized states σ(t) with a GRU, followed by new contextualization with a graph network GN . Details on the full architecture are provided in appendix D.2. The dynamic model CoDy performs forecasting in the higher-dimensional space σ(t), computing a displacement vector δ(t + 1) such that σ(t + 1) = σ(t) + δ(t + 1). It takes the projected state embeddings σCDk (t) per keypoint k concatenated with the confounder representation uk and contextualizes them with a graph network, resulting in embeddings hCDk (t) = GN ([σCDk (t),uk]), which are processed temporally by a GRU. We compute the displacement vector at time t + 1
Published as a conference paper at ICLR 2022
⇥ <latexit sha1_base64="PcNBZ+k3dnNGojhYt75BSmjl47k=">AAACyXicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLoRnBTwT6gLZJMp3VsXiYTsRZX/oBb/THxD/QvvDOmoBbRCUnOnHvPmbn3upEnEmlZrzljZnZufiG/WFhaXlldK65vNJIwjRmvs9AL45brJNwTAa9LIT3eimLu+K7Hm+7wWMWbNzxORBicy1HEu74zCERfMEcS1ehI4fPkoliyypZe5jSwM1BCtmph8QUd9BCCIYUPjgCSsAcHCT1t2LAQEdfFmLiYkNBxjnsUSJtSFqcMh9ghfQe0a2dsQHvlmWg1o1M8emNSmtghTUh5MWF1mqnjqXZW7G/eY+2p7jaiv5t5+cRKXBL7l26S+V+dqkWij0Ndg6CaIs2o6ljmkuquqJubX6qS5BARp3CP4jFhppWTPptak+jaVW8dHX/TmYpVe5blpnhXt6QB2z/HOQ0albK9V66c7ZeqR9mo89jCNnZpngeo4gQ11Mn7Co94wrNxalwbt8bdZ6qRyzSb+LaMhw/935G5</latexit>
<latexit sha1_base64="dzJMrr7Qp1HTfhONc4LWxWuc564=">AAAB+XicbVBNS8NAEN3Ur1q/oh69BIvQU0l6UI8FLx4r2A9oQ9lsJu3SzSbsTool9J948aCIV/+JN/+N2zYHbX0w8Hhvhpl5QSq4Rtf9tkpb2zu7e+X9ysHh0fGJfXrW0UmmGLRZIhLVC6gGwSW0kaOAXqqAxoGAbjC5W/jdKSjNE/mIsxT8mI4kjzijaKShbQ8QnjCI8hBYEoKaD+2qW3eXcDaJV5AqKdAa2l+DMGFZDBKZoFr3PTdFP6cKORMwrwwyDSllEzqCvqGSxqD9fHn53LkySuhEiTIl0VmqvydyGms9iwPTGVMc63VvIf7n9TOMbv2cyzRDkGy1KMqEg4mziMEJuQKGYmYIZYqbWx02pooyNGFVTAje+subpNOoe9f1xkOj2qwVcZTJBbkkNeKRG9Ik96RF2oSRKXkmr+TNyq0X6936WLWWrGLmnPyB9fkDLDqT7Q==</latexit> decoder
C <latexit sha1_base64="5quqgpaO6fbEC2su/WcpYM1UN0w=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLYjcsK9gFtkcl02obmRTIplFLc+gNu9bPEP9C/8M6YglpEJyQ5c+45d+be60Sem0jLes0ZK6tr6xv5zcLW9s7uXnH/oJmEacxFg4deGLcdlgjPDURDutIT7SgWzHc80XLGNRVvTUScuGFwK6eR6PlsGLgDlzNJVKfrMznizJvV5nfFklW29DKXgZ2BErJVD4sv6KKPEBwpfAgEkIQ9MCT0dGDDQkRcDzPiYkKujgvMUSBvSipBCkbsmL5D2nUyNqC9yploN6dTPHpjcpo4IU9IupiwOs3U8VRnVuxvuWc6p7rblP5OlssnVmJE7F++hfK/PlWLxACXugaXaoo0o6rjWZZUd0Xd3PxSlaQMEXEK9ykeE+baueizqT2Jrl31lun4m1YqVu15pk3xrm5JA7Z/jnMZNCtl+6xcuTkvVa+yUedxhGOc0jwvUMU16mjojj/iCc9G3ZgYc+P+U2rkMs8hvi3j4QNsSpPe</latexit>
<latexit sha1_base64="pwjVXc1t6q1Jb883GpAaFSJ4954=">AAAB+XicbVBNS8NAEN3Ur1q/oh69BIvQU0l6UI8FLx4r2A9oQ9lsJu3SzSbsTool9J948aCIV/+JN/+N2zYHbX0w8Hhvhpl5QSq4Rtf9tkpb2zu7e+X9ysHh0fGJfXrW0UmmGLRZIhLVC6gGwSW0kaOAXqqAxoGAbjC5W/jdKSjNE/mIsxT8mI4kjzijaKShbQ8QnjCIcpAsCUHNh3bVrbtLOJvEK0iVFGgN7a9BmLAsBolMUK37npuin1OFnAmYVwaZhpSyCR1B31BJY9B+vrx87lwZJXSiRJmS6CzV3xM5jbWexYHpjCmO9bq3EP/z+hlGt37OZZqh+Wy1KMqEg4mziMEJuQKGYmYIZYqbWx02pooyNGFVTAje+subpNOoe9f1xkOj2qwVcZTJBbkkNeKRG9Ik96RF2oSRKXkmr+TNyq0X6936WLWWrGLmnPyB9fkDO5+T9w==</latexit> encoder <latexit sha1_base64="7iznDX4dY/I7m39kebZMYDm8mFc=">AAACCXicbVDLSsNAFJ3UV42vqEs3g0XoqiRdqMtCN65KBfuAppTJ5LYdOpmEmYlQQrdu/BU3LhRx6x+482+ctEW09cDA4Zx7uHNPkHCmtOt+WYWNza3tneKuvbd/cHjkHJ+0VZxKCi0a81h2A6KAMwEtzTSHbiKBRAGHTjCp537nHqRisbjT0wT6ERkJNmSUaCMNHOwHMGIioyA0yJldbzRsH0T4Iwyckltx58DrxFuSElqiOXA+/TCmaWTylBOlep6b6H5GpGaUw8z2UwUJoRMygp6hgkSg+tn8khm+MEqIh7E0T2g8V38nMhIpNY0CMxkRPVarXi7+5/VSPbzuZ0wkqQZBF4uGKcc6xnktOGQSqOZTQwiVzPwV0zGRhJoSVF6Ct3ryOmlXK95lpXpbLdXKyzqK6AydozLy0BWqoRvURC1E0QN6Qi/o1Xq0nq03630xWrCWmVP0B9bHN+DAmbk=</latexit> CNN K <latexit sha1_base64="SfiPy99ecpVKC/2nKPcBUIO6hnA=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLoRnBTwT6gLTKZTtvQvEgmhVKKW3/ArX6W+Af6F94ZU1CL6IQkZ849587ce53IcxNpWa85Y2l5ZXUtv17Y2Nza3inu7jWSMI25qPPQC+OWwxLhuYGoS1d6ohXFgvmOJ5rO6FLFm2MRJ24Y3MpJJLo+GwRu3+VMEtXu+EwOOfOm17O7YskqW3qZi8DOQAnZqoXFF3TQQwiOFD4EAkjCHhgSetqwYSEirospcTEhV8cFZiiQNyWVIAUjdkTfAe3aGRvQXuVMtJvTKR69MTlNHJEnJF1MWJ1m6niqMyv2t9xTnVPdbUJ/J8vlEysxJPYv31z5X5+qRaKPc12DSzVFmlHV8SxLqruibm5+qUpShog4hXsUjwlz7Zz32dSeRNeuest0/E0rFav2PNOmeFe3pAHbP8e5CBqVsn1SrtyclqoX2ajzOMAhjmmeZ6jiCjXUdccf8YRno2aMjZlx/yk1cplnH9+W8fABf1KT5g==</latexit> F <latexit sha1_base64="ncQtksAl8WfAXclpYeQaOKUgjAg=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LIoiMsK9gFtkcl02obmRTIplFLc+gNu9bPEP9C/8M6YglpEJyQ5c+45d+be60Sem0jLes0ZS8srq2v59cLG5tb2TnF3r5GEacxFnYdeGLcclgjPDURdutITrSgWzHc80XRGlyreHIs4ccPgVk4i0fXZIHD7LmeSqHbHZ3LImTe9mt0VS1bZ0stcBHYGSshWLSy+oIMeQnCk8CEQQBL2wJDQ04YNCxFxXUyJiwm5Oi4wQ4G8KakEKRixI/oOaNfO2ID2Kmei3ZxO8eiNyWniiDwh6WLC6jRTx1OdWbG/5Z7qnOpuE/o7WS6fWIkhsX/55sr/+lQtEn2c6xpcqinSjKqOZ1lS3RV1c/NLVZIyRMQp3KN4TJhr57zPpvYkunbVW6bjb1qpWLXnmTbFu7olDdj+Oc5F0KiU7ZNy5ea0VL3IRp3HAQ5xTPM8QxXXqKGuO/6IJzwbNWNszIz7T6mRyzz7+LaMhw9zbZPh</latexit>
<latexit sha1_base64="pwjVXc1t6q1Jb883GpAaFSJ4954=">AAAB+XicbVBNS8NAEN3Ur1q/oh69BIvQU0l6UI8FLx4r2A9oQ9lsJu3SzSbsTool9J948aCIV/+JN/+N2zYHbX0w8Hhvhpl5QSq4Rtf9tkpb2zu7e+X9ysHh0fGJfXrW0UmmGLRZIhLVC6gGwSW0kaOAXqqAxoGAbjC5W/jdKSjNE/mIsxT8mI4kjzijaKShbQ8QnjCIcpAsCUHNh3bVrbtLOJvEK0iVFGgN7a9BmLAsBolMUK37npuin1OFnAmYVwaZhpSyCR1B31BJY9B+vrx87lwZJXSiRJmS6CzV3xM5jbWexYHpjCmO9bq3EP/z+hlGt37OZZqh+Wy1KMqEg4mziMEJuQKGYmYIZYqbWx02pooyNGFVTAje+subpNOoe9f1xkOj2qwVcZTJBbkkNeKRG9Ik96RF2oSRKXkmr+TNyq0X6936WLWWrGLmnPyB9fkDO5+T9w==</latexit>
encoder
<latexit sha1_base64="7iznDX4dY/I7m39kebZMYDm8mFc=">AAACCXicbVDLSsNAFJ3UV42vqEs3g0XoqiRdqMtCN65KBfuAppTJ5LYdOpmEmYlQQrdu/BU3LhRx6x+482+ctEW09cDA4Zx7uHNPkHCmtOt+WYWNza3tneKuvbd/cHjkHJ+0VZxKCi0a81h2A6KAMwEtzTSHbiKBRAGHTjCp537nHqRisbjT0wT6ERkJNmSUaCMNHOwHMGIioyA0yJldbzRsH0T4Iwyckltx58DrxFuSElqiOXA+/TCmaWTylBOlep6b6H5GpGaUw8z2UwUJoRMygp6hgkSg+tn8khm+MEqIh7E0T2g8V38nMhIpNY0CMxkRPVarXi7+5/VSPbzuZ0wkqQZBF4uGKcc6xnktOGQSqOZTQwiVzPwV0zGRhJoSVF6Ct3ryOmlXK95lpXpbLdXKyzqK6AydozLy0BWqoRvURC1E0QN6Qi/o1Xq0nq03630xWrCWmVP0B9bHN+DAmbk=</latexit> CNN
F <latexit sha1_base64="ncQtksAl8WfAXclpYeQaOKUgjAg=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LIoiMsK9gFtkcl02obmRTIplFLc+gNu9bPEP9C/8M6YglpEJyQ5c+45d+be60Sem0jLes0ZS8srq2v59cLG5tb2TnF3r5GEacxFnYdeGLcclgjPDURdutITrSgWzHc80XRGlyreHIs4ccPgVk4i0fXZIHD7LmeSqHbHZ3LImTe9mt0VS1bZ0stcBHYGSshWLSy+oIMeQnCk8CEQQBL2wJDQ04YNCxFxXUyJiwm5Oi4wQ4G8KakEKRixI/oOaNfO2ID2Kmei3ZxO8eiNyWniiDwh6WLC6jRTx1OdWbG/5Z7qnOpuE/o7WS6fWIkhsX/55sr/+lQtEn2c6xpcqinSjKqOZ1lS3RV1c/NLVZIyRMQp3KN4TJhr57zPpvYkunbVW6bjb1qpWLXnmTbFu7olDdj+Oc5F0KiU7ZNy5ea0VL3IRp3HAQ5xTPM8QxXXqKGuO/6IJzwbNWNszIz7T6mRyzz7+LaMhw9zbZPh</latexit>
C <latexit sha1_base64="5quqgpaO6fbEC2su/WcpYM1UN0w=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLYjcsK9gFtkcl02obmRTIplFLc+gNu9bPEP9C/8M6YglpEJyQ5c+45d+be60Sem0jLes0ZK6tr6xv5zcLW9s7uXnH/oJmEacxFg4deGLcdlgjPDURDutIT7SgWzHc80XLGNRVvTUScuGFwK6eR6PlsGLgDlzNJVKfrMznizJvV5nfFklW29DKXgZ2BErJVD4sv6KKPEBwpfAgEkIQ9MCT0dGDDQkRcDzPiYkKujgvMUSBvSipBCkbsmL5D2nUyNqC9yploN6dTPHpjcpo4IU9IupiwOs3U8VRnVuxvuWc6p7rblP5OlssnVmJE7F++hfK/PlWLxACXugaXaoo0o6rjWZZUd0Xd3PxSlaQMEXEK9ykeE+baueizqT2Jrl31lun4m1YqVu15pk3xrm5JA7Z/jnMZNCtl+6xcuTkvVa+yUedxhGOc0jwvUMU16mjojj/iCc9G3ZgYc+P+U2rkMs8hvi3j4QNsSpPe</latexit>
K <latexit sha1_base64="SfiPy99ecpVKC/2nKPcBUIO6hnA=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLoRnBTwT6gLTKZTtvQvEgmhVKKW3/ArX6W+Af6F94ZU1CL6IQkZ849587ce53IcxNpWa85Y2l5ZXUtv17Y2Nza3inu7jWSMI25qPPQC+OWwxLhuYGoS1d6ohXFgvmOJ5rO6FLFm2MRJ24Y3MpJJLo+GwRu3+VMEtXu+EwOOfOm17O7YskqW3qZi8DOQAnZqoXFF3TQQwiOFD4EAkjCHhgSetqwYSEirospcTEhV8cFZiiQNyWVIAUjdkTfAe3aGRvQXuVMtJvTKR69MTlNHJEnJF1MWJ1m6niqMyv2t9xTnVPdbUJ/J8vlEysxJPYv31z5X5+qRaKPc12DSzVFmlHV8SxLqruibm5+qUpShog4hXsUjwlz7Zz32dSeRNeuest0/E0rFav2PNOmeFe3pAHbP8e5CBqVsn1SrtyclqoX2ajzOMAhjmmeZ6jiCjXUdccf8YRno2aMjZlx/yk1cplnH9+W8fABf1KT5g==</latexit>
D<latexit sha1_base64="UyKrPxuiQxfO3AmecNLyZ+GyUn8=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LKoC5cV7APaIpPptA3Ni2RSKKW49Qfc6meJf6B/4Z0xBbWITkhy5txz7sy914k8N5GW9ZozlpZXVtfy64WNza3tneLuXiMJ05iLOg+9MG45LBGeG4i6dKUnWlEsmO94oumMLlW8ORZx4obBrZxEouuzQeD2Xc4kUe2Oz+SQM296NbsrlqyypZe5COwMlJCtWlh8QQc9hOBI4UMggCTsgSGhpw0bFiLiupgSFxNydVxghgJ5U1IJUjBiR/Qd0K6dsQHtVc5Euzmd4tEbk9PEEXlC0sWE1Wmmjqc6s2J/yz3VOdXdJvR3slw+sRJDYv/yzZX/9alaJPo41zW4VFOkGVUdz7Kkuivq5uaXqiRliIhTuEfxmDDXznmfTe1JdO2qt0zH37RSsWrPM22Kd3VLGrD9c5yLoFEp2yflys1pqXqRjTqPAxzimOZ5hiquUUNdd/wRT3g2asbYmBn3n1Ijl3n28W0ZDx9uq5Pf</latexit>
G <latexit sha1_base64="jJ8/U6+2lPSTKQoDUC9TAEpF0Z8=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLoQpcV7APaIpPptA3Ni2RSKKW49Qfc6meJf6B/4Z0xBbWITkhy5txz7sy914k8N5GW9ZozlpZXVtfy64WNza3tneLuXiMJ05iLOg+9MG45LBGeG4i6dKUnWlEsmO94oumMLlW8ORZx4obBrZxEouuzQeD2Xc4kUe2Oz+SQM296NbsrlqyypZe5COwMlJCtWlh8QQc9hOBI4UMggCTsgSGhpw0bFiLiupgSFxNydVxghgJ5U1IJUjBiR/Qd0K6dsQHtVc5Euzmd4tEbk9PEEXlC0sWE1Wmmjqc6s2J/yz3VOdXdJvR3slw+sRJDYv/yzZX/9alaJPo41zW4VFOkGVUdz7Kkuivq5uaXqiRliIhTuEfxmDDXznmfTe1JdO2qt0zH37RSsWrPM22Kd3VLGrD9c5yLoFEp2yflys1pqXqRjTqPAxzimOZ5hiquUUNdd/wRT3g2asbYmBn3n1Ijl3n28W0ZDx91zpPi</latexit>
R <latexit sha1_base64="dlNprzJQqhs6DvIH0UgfEIimoQg=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLoxmUV+4C2yGQ6bUPzIpkUSilu/QG3+lniH+hfeGdMQS2iE5KcOfecO3PvdSLPTaRlveaMpeWV1bX8emFjc2t7p7i710jCNOaizkMvjFsOS4TnBqIuXemJVhQL5jueaDqjSxVvjkWcuGFwKyeR6PpsELh9lzNJVLvjMznkzJvezO6KJats6WUuAjsDJWSrFhZf0EEPIThS+BAIIAl7YEjoacOGhYi4LqbExYRcHReYoUDelFSCFIzYEX0HtGtnbEB7lTPRbk6nePTG5DRxRJ6QdDFhdZqp46nOrNjfck91TnW3Cf2dLJdPrMSQ2L98c+V/faoWiT7OdQ0u1RRpRlXHsyyp7oq6ufmlKkkZIuIU7lE8Jsy1c95nU3sSXbvqLdPxN61UrNrzTJviXd2SBmz/HOciaFTK9km5cn1aql5ko87jAIc4pnmeoYor1FDXHX/EE56NmjE2Zsb9p9TIZZ59fFvGwweP+ZPt</latexit>
H<latexit sha1_base64="dK8x0xNcQwIOPzuXgCh9zy+W0EQ=">AAACzXicjVHLTsJAFD3UF+ILdemmkZi4IgVNdEl0w05M5BGBmLYMMKGvtFMTgrj1B9zqbxn/QP/CO+OQqMToNG3PnHvPmbn3OpHHE2FZrxljYXFpeSW7mltb39jcym/vNJIwjV1Wd0MvjFuOnTCPB6wuuPBYK4qZ7Tseazqjcxlv3rI44WFwJcYR6/r2IOB97tqCqOuOb4uh059Upzf5glW01DLnQUmDAvSqhfkXdNBDCBcpfDAEEIQ92EjoaaMECxFxXUyIiwlxFWeYIkfalLIYZdjEjug7oF1bswHtpWei1C6d4tEbk9LEAWlCyosJy9NMFU+Vs2R/854oT3m3Mf0d7eUTKzAk9i/dLPO/OlmLQB+nqgZONUWKkdW52iVVXZE3N79UJcghIk7iHsVjwq5SzvpsKk2iape9tVX8TWVKVu5dnZviXd6SBlz6Oc550CgXS0fF8uVxoXKmR53FHvZxSPM8QQVV1FAn7wCPeMKzcWGkxp1x/5lqZLRmF9+W8fABV0uTcQ==</latexit>
features F<latexit sha1_base64="T4/tPEtvPPaGO50ibg+Xny4mHGM=">AAAC2nicjVHLSsNAFD3G97sqrtwEW8FVSepCl6IgLivYKrQik+mkBvNiMhGkuHEnbv0Bt/pB4h/oX3hnHMEHohOSnDn3njNz7w3yOCqU5z0POcMjo2PjE5NT0zOzc/OVhcV2kZWSixbP4kweB6wQcZSKlopULI5zKVgSxOIoON/V8aMLIYsoSw/VZS5OEtZPozDiTBF1WlkOBVOlFIVb6yZMnQXhYO+qdlqpenXPLPcn8C2owq5mVnlCFz1k4CiRQCCFIhyDoaCnAx8ecuJOMCBOEopMXOAKU6QtKUtQBiP2nL592nUsm9JeexZGzemUmF5JShdrpMkoTxLWp7kmXhpnzf7mPTCe+m6X9A+sV0Kswhmxf+k+Mv+r07UohNgyNURUU24YXR23LqXpir65+6kqRQ45cRr3KC4Jc6P86LNrNIWpXfeWmfiLydSs3nObW+JV35IG7H8f50/QbtT9jXrjoFHd3rGjnsAKVrFO89zENvbRRIu8B7jHAx6drnPt3Di376nOkNUs4cty7t4AvSWX0w==</latexit>
keypoints kk <latexit sha1_base64="Ucvztct7NPzWSOoEs+sexuGniIE=">AAAC3XicjVHLSsNAFD2Nr1pfVTeCm2AruCppXeiy6MZlBfuAtpQknbZDniQToZS6cydu/QG3+jviH+hfeGdMQS2iE5KcOfeeM3PvtUKXx8IwXjPawuLS8kp2Nbe2vrG5ld/eacRBEtmsbgduELUsM2Yu91ldcOGyVhgx07Nc1rSccxlvXrMo5oF/JcYh63rm0OcDbpuCqF5+z2HjMOC+iPVixzPFyBpMnGnPKfbyBaNkqKXPg3IKCkhXLci/oIM+AthI4IHBhyDswkRMTxtlGAiJ62JCXESIqzjDFDnSJpTFKMMk1qHvkHbtlPVpLz1jpbbpFJfeiJQ6DkkTUF5EWJ6mq3iinCX7m/dEecq7jelvpV4esQIjYv/SzTL/q5O1CAxwqmrgVFOoGFmdnbokqivy5vqXqgQ5hMRJ3Kd4RNhWylmfdaWJVe2yt6aKv6lMycq9neYmeJe3pAGXf45zHjQqpfJxqXJZKVTP0lFnsY8DHNE8T1DFBWqok/cNHvGEZ62n3Wp32v1nqpZJNbv4trSHD8SjmWc=</latexit>
⇤<latexit sha1_base64="cUNqWkioiZKjoifx/1yOQeG3wyY=">AAACxHicjVHLSsNAFD2Nr1pfVZdugkUQFyVRQZdFQVy2YB9QiyTTaQ3Ni8lEKEV/wK1+m/gH+hfeGaegFtEJSc6ce8+Zuff6aRhk0nFeC9bc/MLiUnG5tLK6tr5R3txqZUkuGG+yJExEx/cyHgYxb8pAhryTCu5Ffsjb/uhcxdt3XGRBEl/Jccp7kTeMg0HAPElU4+CmXHGqjl72LHANqMCselJ+wTX6SMCQIwJHDEk4hIeMni5cOEiJ62FCnCAU6DjHPUqkzSmLU4ZH7Ii+Q9p1DRvTXnlmWs3olJBeQUobe6RJKE8QVqfZOp5rZ8X+5j3RnupuY/r7xisiVuKW2L9008z/6lQtEgOc6hoCqinVjKqOGZdcd0Xd3P5SlSSHlDiF+xQXhJlWTvtsa02ma1e99XT8TWcqVu2Zyc3xrm5JA3Z/jnMWtA6r7lH1sHFcqZ2ZURexg13s0zxPUMMl6mhq70c84dm6sEIrs/LPVKtgNNv4tqyHD7MbjzM=</latexit> convolution<latexit sha1_base64="jedMeJ1Sts3z20QERwSwzeYGKCQ=">AAACznicjVHLSsNAFD2Nr1pfVZdugkVwVZK60GXRjcsK9gG1SDKd1qFJJiSTQinFrT/gVj9L/AP9C++MKahFdEKSM+eec2fuvX4ciFQ5zmvBWlpeWV0rrpc2Nre2d8q7e61UZgnjTSYDmXR8L+WBiHhTCRXwTpxwL/QD3vZHFzreHvMkFTK6VpOY90JvGImBYJ4iqstkNJZBpvFtueJUHbPsReDmoIJ8NWT5BTfoQ4IhQwiOCIpwAA8pPV24cBAT18OUuISQMHGOGUrkzUjFSeERO6LvkHbdnI1or3Omxs3olIDehJw2jsgjSZcQ1qfZJp6ZzJr9LffU5NR3m9Dfz3OFxCrcEfuXb678r0/XojDAmalBUE2xYXR1LM+Sma7om9tfqlKUISZO4z7FE8LMOOd9to0nNbXr3nom/maUmtV7lmszvOtb0oDdn+NcBK1a1T2p1q5qlfp5PuoiDnCIY5rnKeq4RANN0/FHPOHZalhja2bdf0qtQu7Zx7dlPXwAKO6UKw==</latexit> Gik<latexit sha1_base64="oBbqLZ1o7Q7gzY4DftZY+PkvZSU=">AAAC0XicjVHLSsNAFD3GV62vqks3wSK4KkkVdFl0ocuKVoXWSjKdtkPzYjIRSimIW3/Arf6U+Af6F94ZI/hAdEKSM+fec2buvX4SiFQ5zvOENTk1PTNbmCvOLywuLZdWVs/SOJOMN1gcxPLC91IeiIg3lFABv0gk90I/4Of+4EDHz6+5TEUcnaphwi9DrxeJrmCeIqrdCj3V97ujw3FbXA2uSmWn4phl/wRuDsrIVz0uPaGFDmIwZAjBEUERDuAhpacJFw4S4i4xIk4SEibOMUaRtBllccrwiB3Qt0e7Zs5GtNeeqVEzOiWgV5LSxiZpYsqThPVptolnxlmzv3mPjKe+25D+fu4VEqvQJ/Yv3Ufmf3W6FoUu9kwNgmpKDKOrY7lLZrqib25/qkqRQ0Kcxh2KS8LMKD/6bBtNamrXvfVM/MVkalbvWZ6b4VXfkgbsfh/nT3BWrbjblerxTrm2n4+6gHVsYIvmuYsajlBHg7wl7vGAR+vEGlo31u17qjWRa9bwZVl3b7dZlSk=</latexit>
Xsource <latexit sha1_base64="3BdEPFXv6Q8Qbp93tPQ29T9zwL8=">AAAC3XicjVHLSsNAFD2Nr1pfUTeCm2ARXJVEBV0W3bisYB/QlpKk0zY0L5KJWELduRO3/oBb/R3xD/QvvDNNQS2iE5KcOfeeM3PvtULXibmuv+WUufmFxaX8cmFldW19Q93cqsVBEtmsagduEDUsM2au47Mqd7jLGmHETM9yWd0anot4/ZpFsRP4V3wUsrZn9n2n59gmJ6qj7rQ8kw+sXtoYd1qc3fB0Yj3uqEW9pMulzQIjA0VkqxKor2ihiwA2Enhg8MEJuzAR09OEAR0hcW2kxEWEHBlnGKNA2oSyGGWYxA7p26ddM2N92gvPWKptOsWlNyKlhn3SBJQXERanaTKeSGfB/uadSk9xtxH9rczLI5ZjQOxfumnmf3WiFo4eTmUNDtUUSkZUZ2cuieyKuLn2pSpODiFxAncpHhG2pXLaZ01qYlm76K0p4+8yU7Bib2e5CT7ELWnAxs9xzoLaYck4Kh1eHhfLZ9mo89jFHg5onico4wIVVMn7Fk94xovSUe6Ue+VhkqrkMs02vi3l8RPZeJpH</latexit>
Xtarget <latexit sha1_base64="obFu1rb125azrYwEt+98pHWcFxM=">AAAC3XicjVHLSsNAFD2Nr1pfVTeCm2ARXJVEBV0W3bisYB9gpUzitAbzYjIRS6g7d+LWH3CrvyP+gf6Fd8YU1CI6IcmZc+85M/deJ/a9RFrWa8GYmJyaninOlubmFxaXyssrzSRKhcsbbuRHou2whPteyBvSkz5vx4KzwPF5y7k8VPHWFReJF4UnchDzs4D1Q6/nuUwS1S2vdQImL5xe1h52O5Jfy0wy0edy2C1XrKqllzkO7BxUkK96VH5BB+eI4CJFAI4QkrAPhoSeU9iwEBN3how4QcjTcY4hSqRNKYtTBiP2kr592p3mbEh75ZlotUun+PQKUprYJE1EeYKwOs3U8VQ7K/Y370x7qrsN6O/kXgGxEhfE/qUbZf5Xp2qR6GFf1+BRTbFmVHVu7pLqrqibm1+qkuQQE6fwOcUFYVcrR302tSbRtaveMh1/05mKVXs3z03xrm5JA7Z/jnMcNLer9k51+3i3UjvIR13EOjawRfPcQw1HqKNB3jd4xBOeja5xa9wZ95+pRiHXrOLbMh4+AMFemj0=</latexit>
X̂target <latexit sha1_base64="fVd7oFZUaQprsMfHIhOBg9NsjQg=">AAAC43icjVHLSsNAFD3G97vqUpBgEVyVpAq6FN24rGBroZUySadtaF5MJmIJ2blzJ279Abf6L+If6F94Z4zgA9EJSc6ce8+Zufc6se8l0rKex4zxicmp6ZnZufmFxaXl0spqI4lS4fK6G/mRaDos4b4X8rr0pM+bseAscHx+5gyPVPzsgovEi8JTOYr5ecD6odfzXCaJ6pQ22gMms3bA5MDpZc0877Qlv5SZZKLPZd4pla2KpZf5E9gFKKNYtaj0hDa6iOAiRQCOEJKwD4aEnhZsWIiJO0dGnCDk6ThHjjnSppTFKYMRO6Rvn3atgg1przwTrXbpFJ9eQUoTW6SJKE8QVqeZOp5qZ8X+5p1pT3W3Ef2dwisgVmJA7F+6j8z/6lQtEj3s6xo8qinWjKrOLVxS3RV1c/NTVZIcYuIU7lJcEHa18qPPptYkunbVW6bjLzpTsWrvFrkpXtUtacD293H+BI1qxd6pVE92yweHxahnsI5NbNM893CAY9RQJ+8r3OMBjwY3ro0b4/Y91RgrNGv4soy7NwFonQo=</latexit>
reconstructed target <latexit sha1_base64="8dM0FmRGl2PXFUj+kH2AVRrjP+I=">AAAC8XicjVHLSsQwFD3W9/gademmOAguZOiMC12KblyO4IyCFWkzcQx2kpKkgog/4c6duPUH3OpPiH+gf+FNrOAD0ZS25557z0lubppnwtgoeh4IBoeGR0bHxisTk1PTM9XZuY5RhWa8zVSm9H6aGJ4JydtW2Izv55on/TTje+nplsvvnXFthJK79jznh/2kJ8WxYIkl6qi6EjMuLddC9sJYKiG7FIaaMyWN1QWzvBvGccUmusftUbUW1SO/wp+gUYIaytVS1SfE6EKBoUAfHBKWcIYEhp4DNBAhJ+4QF8RpQsLnOS5RIW1BVZwqEmJP6duj6KBkJcXO03g1o10yejUpQyyRRlGdJux2C32+8M6O/c37wnu6s53TPy29+sRanBD7l+6j8r8614vFMdZ9D4J6yj3jumOlS+FvxZ08/NSVJYecOIe7lNeEmVd+3HPoNcb37u428fkXX+lYF7OytsCrOyUNuPF9nD9Bp1lvrNabO83axmY56jEsYBHLNM81bGAbLbTJ+wr3eMBjYILr4Ca4fS8NBkrNPL6s4O4NIaih6g==</latexit>
Gaussian map. <latexit sha1_base64="o2YT39aSkhSPjkONs8AvOZYSaxQ=">AAAC0HicjVHLSsNAFD2Nr1pfVZdugkVwVdK60GXRhS6r2AfUIpN0WgfzMjMRSxFx6w+41a8S/0D/wjtjCmoRnZDkzLn3nJl7rxv7QirHec1ZU9Mzs3P5+cLC4tLySnF1rSmjNPF4w4v8KGm7THJfhLyhhPJ5O044C1yft9zLAx1vXfNEiig8VcOYdwM2CEVfeEwR1T1kqZSChXbA4vJ5seSUHbPsSVDJQAnZqkfFF5yhhwgeUgTgCKEI+2CQ9HRQgYOYuC5GxCWEhIlz3KJA2pSyOGUwYi/pO6BdJ2ND2mtPadQeneLTm5DSxhZpIspLCOvTbBNPjbNmf/MeGU99tyH93cwrIFbhgti/dOPM/+p0LQp97JkaBNUUG0ZX52UuqemKvrn9pSpFDjFxGvconhD2jHLcZ9topKld95aZ+JvJ1Kzee1luind9Sxpw5ec4J0GzWq7slKvH1VJtPxt1HhvYxDbNcxc1HKGOBnlf4RFPeLZOrBvrzrr/TLVymWYd35b18AGLmpRG</latexit>
coe cients ck<latexit sha1_base64="wuNk1MYSxGZVYQu8v5C2b3ASCfE=">AAAC4HicjVG7TsMwFD2EV3kXGLtEFCSmKikDjAgWxiLRggSocowDVvOS4yChqgMbG2LlB1jhaxB/AH/BtUklHkLgKMm5595z7OsbZJHMtee9jDijY+MTk5Wp6ZnZufmF6uJSJ08LxUWbp1GqjgKWi0gmoq2ljsRRpgSLg0gcBr1dkz+8FCqXaXKgrzJxGrPzRIaSM01Ut1rjqQgplCLRubt6EjN9EYR9Puj2VrvVutfw7HJ/Ar8EdZSrlVafcYIzpOAoEEMggSYcgSGn5xg+PGTEnaJPnCIkbV5ggGnSFlQlqIIR26PvOUXHJZtQbDxzq+a0S0SvIqWLNdKkVKcIm91cmy+ss2F/8+5bT3O2K/oHpVdMrMYFsX/phpX/1ZleNEJs2R4k9ZRZxnTHS5fC3oo5ufupK00OGXEGn1FeEeZWObxn12py27u5W2bzr7bSsCbmZW2BN3NKGrD/fZw/QafZ8Dcazf1mfXunHHUFNaxgnea5iW3soYU2eV/jAY94cgLnxrl17j5KnZFSs4wvy7l/B7tPmok=</latexit>
source appearance <latexit sha1_base64="OzpkV4ZmIOyxA3Yo5NYnwZIaCuU=">AAAC1nicjVHLTsJAFD3UF+ILdOmmkZi4IgUXuiS6cYmJPBIkZDoM2FDaZjrVEKI749YfcKufZPwD/QvvDCVRidFp2p45954zc+91I9+LleO8ZayFxaXllexqbm19Y3MrX9huxGEiuajz0A9ly2Wx8L1A1JWnfNGKpGAj1xdNd3iq481rIWMvDC7UOBKdERsEXt/jTBHVzRemTjaLIsEkC7jo5otOyTHLngflFBSRrlqYf8UlegjBkWAEgQCKsA+GmJ42ynAQEdfBhDhJyDNxgVvkSJtQlqAMRuyQvgPatVM2oL32jI2a0yk+vZKUNvZJE1KeJKxPs008Mc6a/c17Yjz13cb0d1OvEbEKV8T+pZtl/lena1Ho49jU4FFNkWF0dTx1SUxX9M3tL1UpcoiI07hHcUmYG+Wsz7bRxKZ23Vtm4u8mU7N6z9PcBB/6ljTg8s9xzoNGpVQ+LFXOK8XqSTrqLHaxhwOa5xGqOEMNdfK+wROe8WK1rDvr3nqYplqZVLODb8t6/ASAQZaZ</latexit>
as a linear transformation from the hidden state of the GRU. We apply the dynamic model in an auto-regressive way to forecast long-term trajectories in the projected space σk, and apply the state decoder to obtain a prediction ŝCD(t).
The dynamic model is trained with a loss in keypoint space, Lphysics = ∑ t ‖sCD(t)− ŝCD(t)‖22 + γ3‖sCD(t)−∆(E(sCD(t)))‖22. (6)
The first term enforces the model to learn to predict the outcomes and the second term favors correct reconstruction of the state in keypoint space. The terms are weighted with a scalar parameter γ3.

heading: 4.3 TRAINING
text: End-to-end training of all three modules jointly is challenging, as the same pipeline controls both the keypoint-based state representation and the dynamic module (CoDy), involving two adversarial objectives: optimizing reconstruction pushes the keypoint encoding to be as representative as possible, but learning the dynamics favors a simple representation. Faced with these two contradictory tasks, the model is numerically unstable and rapidly converges to regression to the mean. As described above, we separately train the encoder+decoder pair without dynamic information on reconstruction only, c.f. Equation (4). Then we freeze the parameters of the keypoint detector and train CoDy to forecast the keypoints from D minimizing the loss in Equation (6).

heading: 5 EXPERIMENTS
text: We compare the proposed model to three strong baselines for physics-inspired video prediction.
• PhyDNet (Le Guen & Thome, 2020) is a non-counterfactual video prediction model that forecasts future frames using a decomposition between (a) a feature vector that temporally evolves via an LSTM and (b) a dynamic state that follows a PDE learned through specifically designed cells.
• V-CDN (Li et al., 2020) is a counterfactual model based on keypoints, close to our work. It identifies confounders from the beginning of a sequence and learns a keypoint predictor through auto-encoding using the Transporter equation (see discussion in Sect. 4.1). As it is, it cannot be used for video prediction and is incomparable with our work, see details in appendix C. We therefore replace the Transporter by our own de-rendering/rendering modules, from which we remove the additional coefficients. We refer to this model as UV-CDN (for Unsupervised V-CDN).
• PredRNN (Wang et al., 2017) is a ConvLSTM-based video prediction model that leverages spatial and temporal memories a through a spatiotemporal LSTM cell.
All models have been implemented in PyTorch, architectures are described in appendix D. For the baselines PhyDNet, UV-CDN and PredRNN, we used the official source code provided by the authors. We evaluate on each scenario of Filtered-CoPhy on the counterfactual video prediction task. For the two counterfactual models (Ours and UV-CDN), we evaluate on the tasks as intended: we provide the observed sequence AB and the CF initial condition C, and forecast the sequence D. The non-CF baselines are required to predict the entire video from a single frame, in order to prevent them from leveraging shortcuts in a part of the video and bypass the need for physical reasoning.
We measure performance with time-averaged peak signal-to noise ratio (PSNR) that directly measures reconstruction quality. However, this metric is mainly dominated by error on the static background, which is not our main interest. We also introduce Localized PSNR (L-PSNR), which measures area error on the important regions near moving objects, computed on masked images. We compute the masks using classical background subtraction techniques.
Comparison to the SOTA — We compare our model against UV-CDN, PhyDNet and PredRNN in Table 1, consistently and significantly outperforming the baselines. The gap with UV-CDN is particularly interesting, as it confirms the choice of additional coefficients to model the dynamics of moving objects. PredRNN shows competitive performances, especially on collisionCF. However, our localized PSNR tends to indicates that the baseline does not reconstruct accurately the foreground, favoring the reconstruction of the background to the detriment of the dynamics of the scene. Fig. 6 visualizes the prediction on a single example, more can be found in appendix G. We also compare to trivial copying baselines in Table 2, namely Copy B, which assumes no intervention and outputs the B sequence, and Copy C, which assumes a stable tower. We evaluate these models in keypoints space measuring MSE on keypoints and coefficients averaged over time, as copying baselines are unbeatable in the regions of static background, making the PSNR metrics unusable.
We provide additional empirical results by comparing the models using Multi-object Tracking metrics and studies on the impact of the do-operations on PSNR in appendix E. We also compute an upper bound of our model using CoPhyNet baseline as described in Baradel et al. (2020).
Performance on real-world data — is reported in appendix F, showing experiments on 516 videos of real wooden blocks introduced in (Lerer et al., 2016).
Impact of appearance coefficients — are reported in Table 3, comparing to the baseline using a keypoint-only representation. The coefficients have a significant impact: even increasing the number of keypoints to compensate for the loss of information cannot overcome the advantage of disentangling positions and shapes, as done in our model. We provide a deeper analysis of the
de-rendering/rendering modules in appendix B, which includes visualizations of the navigation of the latent shape space in B.2.
Learning filters — does not have a positive impact on reconstruction performance compared to the choice of the handcrafted bank, as can be seen in table 5. We conjecture that the additional degrees of freedom are redundant with parts of the filter kernels in the refinement module R: this corresponds to jointly learning a multi-channel representation {Gik}i=1..Ck=1..K for shapes as well as the mapping which geometrically distorts them into the target object shapes. Fixing the latent representation does not constrain the system, as the mappingR can adjust to it — see Fig. 5. Impact of the high-dimensional dynamic space — We evaluate the impact of modeling object dynamics in high-dimensional space through the CoDy encoder in Table 4, comparing projection to 256 dimensions to the baseline reasoning directly in keypoint + coefficient space. The experiment confirms this choice of KKL-like encoder (Janny et al., 2021).

heading: 6 CONCLUSION
text: We introduced a new benchmark for counterfactual reasoning in physical processes requiring to perform video prediction, i.e. predicting raw pixel observations over a long horizon. The benchmark has been carefully designed and generated imposing constraints on identifiability and counterfactuality. We also propose a new method for counterfactual reasoning, which is based on a hybrid latent representation combining 2D keypoints and additional latent vectors encoding appearance and shape. We introduce an unsupervised learning algorithm for this representation, which does not require any supervision on confounders or other object properties and processes raw video. Counterfactual prediction of video frames remains a challenging task, and Filtered CoPhy still exhibits failures in maintaining rigid structures of objects over long prediction time-scales. We hope that our benchmark will inspire further breakthroughs in this domain.

heading: Appendix

heading: A FURTHER DETAILS ON DATASET GENERATION
text: Confounders in our setup are masses, which we discretize in {1, 10}. For BallsCF and CollisionCF, we can also consider the continuous initial velocities of each object as confounders variables, since they have to be identified in AB to forecast CD.
We simulate all trajectories associated with the various possible combinations of masses from the same initial condition.
Do-interventions, however, depend on the task. For BlocktowerCF and BallsCF, dointerventions consist in (a) removing the top cube or a ball, or (b) shifting a cube/ball on the horizontal plane. In this case, for BlocktowerCF, we make sure that the cube does not move too far from the tower, in order to maintain contact. For CollisionCF, the do-interventions are restricted to shifting operations, since there are only two objects (a ball and a cylinder). It can consist of either a switch of the cylinder’s orientation between vertical or horizontal, or a shift of the position of the moving object relative to the resting one in one of the three canonical directions x, y and z.

heading: A.1 ENFORCING THE IDENTIFIABILTY CONSTRAINT
text: The identifiabilty and counterfactuality constraints described in section 3 are imposed numerically, i.e. we first sample and simulate trajectories with random parameters and then reject those that violate these constraints.
As stated in section 3, an identifiable experiment guarantees that there is no pair (z, z′) that gives the same trajectory AB but a different counterfactual outcome CD. Otherwise, there will be no way to choose between z and z′ only from looking at AB, thus no way to correctly forecast the counterfactual experiment. By enforcing this constraint, we make sure that there exists at least a set {z, z′, ...} of confounders that give at the same time similar observed outcomes AB and similar counterfactual outcomes CD.
In practice, there exists a finite set of possible variables zi, corresponding to every combination of masses for each object in the scene (masses take their value in {1, 10}). During generation, we submit each candidate experiment (AB,CD, z) to a test ensuring that the candidate is identifiable. Let ψ(X0, z) be the function that gives the trajectory of a system with initial condition X0 and confounders z. We simulate all possible trajectories ψ(A, zi) and ψ(C, zi) for every possible zi. If there exists z′ 6= z such that the experiment is not identifiable, the candidate is rejected. This constraint requires to simulate the trajectory of each experiment several times by modifying physical properties of the objects.
Equalities in Definition 1 are relaxed by thresholding distances between trajectories. We reject a candidate experiment if there exists a z′ such that
T∑ t=0 ‖ψ(A, z)− ψ(A, z′)‖2 < ε and T∑ t=0 ‖ψ(C, z)− ψ(C, z′)‖2 > ε. (7)
The choice of the threshold value ε is critical, in particular for the identifiability constraint:
• If the threshold is too high, all AB trajectories will be considered as equal, which results in acceptance of unidentifiable experiments.
• If the threshold is too low, all trajectories CD are considered equal. Again, this leads to mistakenly accepting unidentifiable experiments.
There exists an optimal value for ε, which allows to correctly reject unidentifiable experiences. To measure this optimal threshold, we generated a small instance of the BlocktowerCF dataset without constraining the experiments, i.e. trajectories can be unidentifiable and non-counterfactual. We then plot the percentage of rejected experiments in this unfiltered dataset against the threshold
value (Fig. 7, left). We chose the threshold ε = 100 which optimizes discrimination and rejects the highest number of “unidentifiable” trajectories.
To demonstrate importance of this, we train a recurrent graph network on BlocktowerCF to predict the cube masses from ground-truth state trajectories AB, including pose and velocities, see Fig. 8. It predicts each cube’s mass by solving a binary classification task. We train this model on both BlocktowerCF and an alternative version of the scenario generated without the identifiability constraint. The results are shown in Table 6a. We are not aiming 100% accuracy, and this problem remains difficult in a sense that the identifiability constraint ensures the identifiability of a set of confounder variables, while our sanity check tries to predict a unique z.
However, the addition of the identifiability constraint to the benchmark significantly improves the model’s accuracy, which indicates that the property acts positively on the feasability of FilteredCoPhy. The corrected accuracy metric focuses solely on the critical cubes, i.e. those cubes whose masses directly define the trajectory CD.

heading: A.2 ENFORCING THE COUNTERFACTUALITY CONSTRAINT
text: Let (AB,CD, z) be a candidate experiment, and zk be a combination of masses identical to z except for the kth value. The counterfactuality constraint consists in checking that there exists at least one k such that ψ(C, z) 6= ψ(C, zk). To do so, we simulate ψ(C, zk) for all k and measure the difference with the candidate trajectory ψ(C, z). Formally, we verify the existence of k such that:
T∑ t=0 ‖ψ(C, zk)− ψ(C, z)‖2 < ε. (8)

heading: A.3 ANALYZING TEMPORAL RESOLUTION
text: We analyzed the choice of temporal frequency for the benchmark with another sanity check. We simulate a non-counterfactual dataset from BlocktowerCF where all cubes have equal masses. A
Published as a conference paper at ICLR 2022
2
664
Position
Quaternion
Lin. vel. Ang. vel.
3
775
t=0..1s <latexit sha1_base64="7Tcng+M2cB8I/IdlmHd9DkoqMYM=">AAADRXicjVFNb9QwFHwJHy1LgQUEFy4RKyROUVKQ6AWpwIUDh63EtpU2q5XjfZta9TqR7VSsokj8O34C4g8gOHBDXOHZuFJLhcBRkvG8mbGfXTZSGJtln6L40uUrVzc2rw2ub924eWt4+86+qVvNccJrWevDkhmUQuHECivxsNHIVqXEg/L4lasfnKA2olZv7brB2YpVSiwFZ5ao+fB9IXFpp0WJlVAd05qt+473g8LiO9uNayOcrk+KIlB7LbOo1XnyjVBpcoIyPcO9UFXgBgWqRcgutKiO7Gze2edZmuamnw9HWZr5kVwEeQAjCGNcDz9CAQuogUMLK0BQYAlLYGDomUIOGTTEzaAjThMSvo7Qw4C8LamQFIzYY/pWNJsGVtHcZRrv5rSKpFeTM4FH5KlJpwm71RJfb32yY/+W3flMt7c1/cuQtSLWwhGx//KdKv/X53qxsIQd34OgnhrPuO54SGn9qbidJ2e6spTQEOfwguqaMPfO03NOvMf43t3ZMl//4pWOdXMetC18dbukC87/vM6LYH87zZ+k23tPR7svw1VvwgN4CI/pPp/BLryGMUwo+3O0Fd2L7scf4m/x9/jHb2kcBc9dODfin78AX6vBeQ==</latexit>
2 664
Position
Quaternion
Lin. vel. Ang. vel.
3 775
t=1s..2s<latexit sha1_base64="OlKLufKVZfEZk+NcaCjtk4azREE=">AAADRnicjVFNb9QwFHxJ+SgLhQUkOHCJWCFxipIFCS5IBS4cOGwltq20Wa0c79vUqteJbKdiFeXAv+MfIH4BcOGGuPJsXKmlQuAoyXjezNjPLhspjM2yz1G8denylavb1wbXb+zcvDW8fWff1K3mOOW1rPVhyQxKoXBqhZV42Ghk61LiQXn82tUPTlAbUat3dtPgfM0qJVaCM0vUYvihkLiys6LESqiOac02fcf7QWHxve0mtRFO1ydFEai9llnU6jz5Vqg0OUGZnuFeqipwgwLVMmQXWlRHdr7o7IvcpOnY9IvhKEszP5KLIA9gBGFM6uEnKGAJNXBoYQ0ICixhCQwMPTPIIYOGuDl0xGlCwtcRehiQtyUVkoIRe0zfimazwCqau0zj3ZxWkfRqcibwiDw16TRht1ri661Pduzfsjuf6fa2oX8ZstbEWjgi9l++U+X/+lwvFlbw3PcgqKfGM647HlJafypu58mZriwlNMQ5vKS6Jsy98/ScE+8xvnd3tszXv3qlY92cB20L39wu6YLzP6/zItgfp/mTdLz3dLT7Klz1NjyAh/CY7vMZ7MIbmMCUsr9EO9G96H78Mf4e/4h//pbGUfDchXNjC34BzInA+Q==</latexit> hT <latexit sha1_base64="kztCHEH/Qi2tjI6t/mhTMsHXKr8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLopsuKfUEtJZlO29C8SCZKKYI/4FY/TfwD/QvvjFNQi+iEJGfOvefM3Hvd2PdSYVmvOWNpeWV1Lb9e2Njc2t4p7u610ihLGG+yyI+Sjuuk3PdC3hSe8HknTrgTuD5vu5NLGW/f8iT1orAhpjHvBc4o9IYecwRR1+N+o18sWWVLLXMR2BqUoFc9Kr7gBgNEYMgQgCOEIOzDQUpPFzYsxMT1MCMuIeSpOMc9CqTNKItThkPshL4j2nU1G9JeeqZKzegUn96ElCaOSBNRXkJYnmaqeKacJfub90x5yrtN6e9qr4BYgTGxf+nmmf/VyVoEhjhXNXhUU6wYWR3TLpnqiry5+aUqQQ4xcRIPKJ4QZko577OpNKmqXfbWUfE3lSlZuWc6N8O7vCUN2P45zkXQqpTtk3Ll6rRUvdCjzuMAhzimeZ6hihrqaJL3CI94wrNRM0IjM+4+U42c1uzj2zIePgA8WZA4</latexit>
GN <latexit sha1_base64="8LZdPDoHp8i1UUsxyTj2GjFetyM=">AAACxXicjVHLSsNAFD2Nr1pfVZdugkVwVdK60GXRRV1JFfuAWiSZTuvQvEgmhVLEH3Crvyb+gf6Fd8YpqEV0QpIz595zZu69XuyLVDrOa85aWFxaXsmvFtbWNza3its7rTTKEsabLPKjpOO5KfdFyJtSSJ934oS7gefztjc6U/H2mCepiMJrOYl5L3CHoRgI5kqiruoXt8WSU3b0sudBxYASzGpExRfcoI8IDBkCcISQhH24SOnpogIHMXE9TIlLCAkd57hHgbQZZXHKcIkd0XdIu65hQ9orz1SrGZ3i05uQ0sYBaSLKSwir02wdz7SzYn/znmpPdbcJ/T3jFRArcUfsX7pZ5n91qhaJAU50DYJqijWjqmPGJdNdUTe3v1QlySEmTuE+xRPCTCtnfba1JtW1q966Ov6mMxWr9szkZnhXt6QBV36Ocx60quXKUbl6WS3VTs2o89jDPg5pnseo4RwNNMl7gEc84dmqW4ElrfFnqpUzml18W9bDB9axj6Y=</latexit>
GRU <latexit sha1_base64="9czWfm05RXgBuzO5RCr7cs1yMyg=">AAACxnicjVHLTsJAFD3UF+ILdemmkZi4Ii0udEl0IUt8FEiQmLYM2NBXplMNISb+gFv9NOMf6F94ZxwSlRidpu2Zc+85M/deLw2DTFjWa8GYm19YXCoul1ZW19Y3yptbrSzJuc8cPwkT3vHcjIVBzBwRiJB1Us7cyAtZ2xudyHj7lvEsSOJLMU5ZL3KHcTAIfFcQdXF67lyXK1bVUsucBbYGFejVTMovuEIfCXzkiMAQQxAO4SKjpwsbFlLiepgQxwkFKs5wjxJpc8pilOESO6LvkHZdzca0l56ZUvt0SkgvJ6WJPdIklMcJy9NMFc+Vs2R/854oT3m3Mf097RURK3BD7F+6aeZ/dbIWgQGOVA0B1ZQqRlbna5dcdUXe3PxSlSCHlDiJ+xTnhH2lnPbZVJpM1S5766r4m8qUrNz7OjfHu7wlDdj+Oc5Z0KpV7YNq7axWqR/rURexg13s0zwPUUcDTTjkPcQjnvBsNIzYyI27z1SjoDXb+LaMhw/Qe5AJ</latexit>
⇥1 second<latexit sha1_base64="dMXh7xQcsvXLq05/AtO2XPpp0YU=">AAAC3HicjVHLSsNAFD2N7/qqunDhJlgEVyVRQZdFNy4VrC20VZLptA7Ni8xELMWdO3HrD7jV7xH/QP/CO2MKPhCdkOTMufecuXeunwRCKsd5KVhj4xOTU9Mzxdm5+YXF0tLyqYyzlPEai4M4bfie5IGIeE0JFfBGknIv9ANe9/sHOl6/5KkUcXSiBglvh14vEl3BPEXUeWm1pUTIpe3aLcWv1NCWnMVR5/q8VHYqjln2T+DmoIx8HcWlZ7TQQQyGDCE4IijCATxIeppw4SAhro0hcSkhYeIc1yiSNqMsThkesX369mjXzNmI9tpTGjWjUwJ6U1La2CBNTHkpYX2abeKZcdbsb95D46lrG9Dfz71CYhUuiP1LN8r8r073otDFnulBUE+JYXR3LHfJzK3oyu1PXSlySIjTuEPxlDAzytE920YjTe/6bj0TfzWZmtV7ludmeNNV0oDd7+P8CU63Ku52Zet4p1zdz0c9jTWsY5PmuYsqDnGEmqn/AY94ss6sG+vWuvtItQq5ZgVflnX/DgW3mLo=</latexit>
CD states 1 second length <latexit sha1_base64="gjnlc+UdFG4dQxI6cCmXhLPZx50=">AAADFXicjVFNb9QwEH0NX2X5WuDIxWKLxGmVLAc4VpQDx0Vi20pNVSXe2V2riR3ZDqJE+zv4J9y4Ia4cERJXEPwLxiaVgAqBo9jPb+Y9ezxlUynn0/TTRnLu/IWLlzYvD65cvXb9xvDmrV1nWitpJk1l7H5ZOKqUpplXvqL9xlJRlxXtlcc7Ib73gqxTRj/3Jw0d1sVSq4WShWfqaDjNtVF6TtrnkieySi9zTy/ZqtvK68KvykW382S9JZwvPDmR54PcSasa79Qr6jLhSBo9FxXppV+t10fDUTpO4xBnQdaDEfoxNcOPyDGHgUSLGgQNz7hCAcffATKkaJg7RMecZaRinLDGgLUtZxFnFMwe87zk3UHPat4HTxfVkk+p+LesFLjHGsN5lnE4TcR4G50D+zfvLnqGu53wWvZeNbMeK2b/pTvN/F9dqMVjgUexBsU1NZEJ1cnepY2vEm4ufqnKs0PDXMBzjlvGMipP31lEjYu1h7ctYvxbzAxs2Ms+t8X3cEtucPZnO8+C3ck4ezCePJuMth/3rd7EHdzFfe7nQ2zjKaaYsfcbfMYXfE1eJ2+Td8n7n6nJRq+5jd9G8uEHPESxWA==</latexit>
CD states next second <latexit sha1_base64="5Dd6VL913MRDJIf0qgzHW52pEDI=">AAADEXicjVHLbtQwFD0Nj5bhNcCSTcQUidUoM13AsqJdsGDRSkxbqakqx3NnajWxI9tBLdF8BX/SHTvElg8AlmUFf8G1SSWgQuAozvG555z42kVdKuez7MtScuXqtevLKzd6N2/dvnO3f+/+jjONlTSRpjR2rxCOSqVp4pUvaa+2JKqipN3ieCPUd1+TdcroV/60poNKzLWaKSk8U4f9l7k2Sk9J+1zyRFbpee7phKPa1bwS/qiYtRubi9XUeeHJpXney520qvZOvaFWszR1JI2eLhaH/UE2zOJIL4NRBwboxpbpf0KOKQwkGlQgaHjGJQQcP/sYIUPN3AFa5iwjFeuEBXrsbVhFrBDMHvM859V+x2peh0wX3ZL/UvJr2ZniMXsM6yzj8Lc01puYHNi/ZbcxM+ztlL9Fl1Ux63HE7L98F8r/9YVePGZ4FntQ3FMdmdCd7FKaeCph5+kvXXlOqJkLeMp1y1hG58U5p9HjYu/hbEWsf4vKwIa17LQNvodd8gWP/rzOy2BnPBytDcfb48H68+6qV/AQj/CE7/Mp1vECW5hw9hk+4xxfk7fJu+R98uGnNFnqPA/w20g+/gCg07Ac</latexit>
2 664
Position Quaternion Lin. vel. Ang. vel.
3 775
t=0..T <latexit sha1_base64="cLfKOCogqyN9evxahqU1SJ8fJ4w=">AAADRXicjVFBa9RAGP0Sq9ZY7aropZfgIngK2bagF6HWiwcPW+i2hc2yTGa/TYfOTsJkUlxCwH/nT5D+AdGDN/HafjNOobWITkjy5n3vvZlvJq+kqE2angXhrZXbd+6u3ovurz14uN579PigLhvNccRLWeqjnNUohcKREUbiUaWRLXKJh/nJO1s/PEVdi1Ltm2WFkwUrlJgLzgxR096nTOLcjLMcC6FapjVbdi3voszgR9MOy1pYXRdnmaf2GmZQq+vkB6GS+BRlcoV7qwrPRRmqmc/OtCiOzWTamjdpkux30bTXT5PUjfgmGHjQBz+GZe8LZDCDEjg0sAAEBYawBAY1PWMYQAoVcRNoidOEhKsjdBCRtyEVkoIRe0LfgmZjzyqa28zauTmtIunV5IzhBXlK0mnCdrXY1RuXbNm/Zbcu0+5tSf/cZy2INXBM7L98l8r/9dleDMzhtetBUE+VY2x33Kc07lTszuMrXRlKqIizeEZ1TZg75+U5x85Tu97t2TJX/+aUlrVz7rUNfLe7pAse/HmdN8HBZjLYSjb3tvs7u/6qV2EDnsNLus9XsAPvYQgjyv4arAVPg2fh5/BH+DP89VsaBt7zBK6N8PwCuazBMw==</latexit>
AB states<latexit sha1_base64="qiQTca0pFXDvsOJ34EMDLkm/HfI=">AAAC4nicjVHLSsNAFD2Nr1pfVZciBFvBVUnrQpdaNy4r2Ae0RSbptA3mRWYiSunKnTtx6w+41Y8R/0D/wjtjCj4QnZDkzLn3nJl7rx15rpCW9ZIxpqZnZuey87mFxaXllfzqWkOESezwuhN6YdyymeCeG/C6dKXHW1HMmW97vGmfH6l484LHwg2DU3kV8a7PBoHbdx0miTrLbxY7PpNDuz86rI6LZkfyS3IZCckkF+OzfMEqWXqZP0E5BQWkqxbmn9FBDyEcJPDBEUAS9sAg6GmjDAsRcV2MiIsJuTrOMUaOtAllccpgxJ7Td0C7dsoGtFeeQqsdOsWjNyaliW3ShJQXE1anmTqeaGfF/uY90p7qblf0t1Mvn1iJIbF/6SaZ/9WpWiT62Nc1uFRTpBlVnZO6JLor6ubmp6okOUTEKdyjeEzY0cpJn02tEbp21Vum4686U7Fq76S5Cd7ULWnA5e/j/AkalVJ5t1Q5qRQOqumos9jAFnZonns4wDFqqJP3NR7wiCejZ9wYt8bdR6qRSTXr+LKM+3dIhJuL</latexit>
hT <latexit sha1_base64="kztCHEH/Qi2tjI6t/mhTMsHXKr8=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZIq6LLopsuKfUEtJZlO29C8SCZKKYI/4FY/TfwD/QvvjFNQi+iEJGfOvefM3Hvd2PdSYVmvOWNpeWV1Lb9e2Njc2t4p7u610ihLGG+yyI+Sjuuk3PdC3hSe8HknTrgTuD5vu5NLGW/f8iT1orAhpjHvBc4o9IYecwRR1+N+o18sWWVLLXMR2BqUoFc9Kr7gBgNEYMgQgCOEIOzDQUpPFzYsxMT1MCMuIeSpOMc9CqTNKItThkPshL4j2nU1G9JeeqZKzegUn96ElCaOSBNRXkJYnmaqeKacJfub90x5yrtN6e9qr4BYgTGxf+nmmf/VyVoEhjhXNXhUU6wYWR3TLpnqiry5+aUqQQ4xcRIPKJ4QZko577OpNKmqXfbWUfE3lSlZuWc6N8O7vCUN2P45zkXQqpTtk3Ll6rRUvdCjzuMAhzimeZ6hihrqaJL3CI94wrNRM0IjM+4+U42c1uzj2zIePgA8WZA4</latexit> m 2 {1, 10}
<latexit sha1_base64="e2qmBwH6oS+Mx5fQGybFs3ED4SE=">AAAC1HicjVHLSsNAFD2Nr1ofrbp0EyyCCylJFXRZdOOygn1AU0qSTuvQvEgmQqldiVt/wK1+k/gH+hfeGVNQi+iEJGfOPefO3HudyOOJMIzXnLawuLS8kl8trK1vbBZLW9vNJExjlzXc0AvjtmMnzOMBawguPNaOYmb7jsdazuhcxls3LE54GFyJccS6vj0M+IC7tiCqVyr6usUD3ZqYh7ppWNNeqWxUDLX0eWBmoIxs1cPSCyz0EcJFCh8MAQRhDzYSejowYSAirosJcTEhruIMUxTIm5KKkcImdkTfIe06GRvQXuZMlNulUzx6Y3Lq2CdPSLqYsDxNV/FUZZbsb7knKqe825j+TpbLJ1bgmti/fDPlf32yFoEBTlUNnGqKFCOrc7MsqeqKvLn+pSpBGSLiJO5TPCbsKuesz7ryJKp22Vtbxd+UUrJy72baFO/yljRg8+c450GzWjGPKtXL43LtLBt1HrvYwwHN8wQ1XKCOhpr5I57wrDW1W+1Ou/+UarnMs4NvS3v4AJjQlDQ=</latexit>
GN <latexit sha1_base64="8LZdPDoHp8i1UUsxyTj2GjFetyM=">AAACxXicjVHLSsNAFD2Nr1pfVZdugkVwVdK60GXRRV1JFfuAWiSZTuvQvEgmhVLEH3Crvyb+gf6Fd8YpqEV0QpIz595zZu69XuyLVDrOa85aWFxaXsmvFtbWNza3its7rTTKEsabLPKjpOO5KfdFyJtSSJ934oS7gefztjc6U/H2mCepiMJrOYl5L3CHoRgI5kqiruoXt8WSU3b0sudBxYASzGpExRfcoI8IDBkCcISQhH24SOnpogIHMXE9TIlLCAkd57hHgbQZZXHKcIkd0XdIu65hQ9orz1SrGZ3i05uQ0sYBaSLKSwir02wdz7SzYn/znmpPdbcJ/T3jFRArcUfsX7pZ5n91qhaJAU50DYJqijWjqmPGJdNdUTe3v1QlySEmTuE+xRPCTCtnfba1JtW1q966Ov6mMxWr9szkZnhXt6QBV36Ocx60quXKUbl6WS3VTs2o89jDPg5pnseo4RwNNMl7gEc84dmqW4ElrfFnqpUzml18W9bDB9axj6Y=</latexit> GRU
<latexit sha1_base64="9czWfm05RXgBuzO5RCr7cs1yMyg=">AAACxnicjVHLTsJAFD3UF+ILdemmkZi4Ii0udEl0IUt8FEiQmLYM2NBXplMNISb+gFv9NOMf6F94ZxwSlRidpu2Zc+85M/deLw2DTFjWa8GYm19YXCoul1ZW19Y3yptbrSzJuc8cPwkT3vHcjIVBzBwRiJB1Us7cyAtZ2xudyHj7lvEsSOJLMU5ZL3KHcTAIfFcQdXF67lyXK1bVUsucBbYGFejVTMovuEIfCXzkiMAQQxAO4SKjpwsbFlLiepgQxwkFKs5wjxJpc8pilOESO6LvkHZdzca0l56ZUvt0SkgvJ6WJPdIklMcJy9NMFc+Vs2R/854oT3m3Mf097RURK3BD7F+6aeZ/dbIWgQGOVA0B1ZQqRlbna5dcdUXe3PxSlSCHlDiJ+xTnhH2lnPbZVJpM1S5766r4m8qUrNz7OjfHu7wlDdj+Oc5Z0KpV7YNq7axWqR/rURexg13s0zwPUUcDTTjkPcQjnvBsNIzYyI27z1SjoDXb+LaMhw/Qe5AJ</latexit>
⇥T<latexit sha1_base64="GON65GiF9QecFjOLbm1o8aaeSeQ=">AAACy3icjVHLSsNAFD2Nr1pfVZdugkVwVVIVdFl040ao0Be0RZLptA7Ni8xEqNWlP+BW/0v8A/0L74wpqEV0QpIz555zZ+69XuwLqRznNWfNzS8sLuWXCyura+sbxc2tpozShPEGi/woaXuu5L4IeUMJ5fN2nHA38Hze8kZnOt664YkUUVhX45j3AncYioFgriKq3VUi4NKuXxVLTtkxy54FlQyUkK1aVHxBF31EYEgRgCOEIuzDhaSngwocxMT1MCEuISRMnOMeBfKmpOKkcIkd0XdIu07GhrTXOaVxMzrFpzchp4098kSkSwjr02wTT01mzf6We2Jy6ruN6e9luQJiFa6J/cs3Vf7Xp2tRGODE1CCoptgwujqWZUlNV/TN7S9VKcoQE6dxn+IJYWac0z7bxiNN7bq3rom/GaVm9Z5l2hTv+pY04MrPcc6C5kG5clg+uDwqVU+zUeexg13s0zyPUcU5amiYOT7iCc/WhSWtW+vuU2rlMs82vi3r4QNiL5JB</latexit>
MLP<latexit sha1_base64="9J6TWPodTZHrQrnxTaOwbv6lVcc=">AAACxnicjVHLSsNAFD2Nr1pfVZdugkVwVZK60GXRTRcKFe0DapFkOq1D0yRMJkopgj/gVj9N/AP9C++MKahFdEKSM+fec2buvX4ciEQ5zmvOmptfWFzKLxdWVtfWN4qbW80kSiXjDRYFkWz7XsIDEfKGEirg7Vhyb+QHvOUPT3S8dctlIqLwUo1j3h15g1D0BfMUURdnp/XrYskpO2bZs8DNQAnZqkfFF1yhhwgMKUbgCKEIB/CQ0NOBCwcxcV1MiJOEhIlz3KNA2pSyOGV4xA7pO6BdJ2ND2mvPxKgZnRLQK0lpY480EeVJwvo028RT46zZ37wnxlPfbUx/P/MaEatwQ+xfumnmf3W6FoU+jkwNgmqKDaOrY5lLarqib25/qUqRQ0ycxj2KS8LMKKd9to0mMbXr3nom/mYyNav3LMtN8a5vSQN2f45zFjQrZfegXDmvlKrH2ajz2MEu9mmeh6iihjoa5D3AI57wbNWs0Eqtu89UK5dptvFtWQ8fxKGQBA==</latexit>
Figure 9: Visual examples of the impact of temporal resolution on dynamical information for each task in Filtered-CoPhy. Black dots are sampled at 6 FPS while red dots are sampled at 25 FPS.
recurrent graph network takes as input cube trajectories (poses and velocities) over a time interval of one second and predicts the rest of the trajectory over the following second. We vary the sampling frequency; for example, at 5 FPS, the model receives 5 measurements, and predicts the next 5 time steps, which correspond to a one second rollout in the future. Finally, we compare the error in 3D positions between the predictions and the GT on the last prediction. Results are shown in Table 6b. This check shows clearly that 25 FPS corresponds to the best trade-off between an accurate representation of the collision and the amount of training data. Fig. 9 shows a practical example of the effect of time resolution on dynamical information in a trajectory.

heading: A.4 SIMULATION DETAILS
text: We used Pybullet as physics engine to simulate Filtered-CoPhy. Each experiment is designed to respects the balance between good coverage of confounder combinations and counterfactuality and identifiability constraints described above. We generate the trajectories iteratively:
1. We sample a combination of masses and other physical characteristics of the given experiment, such as stability of the tower, or object motion in CollisionCF, or if the do-operation consists in removing an object. This allows us to maintain a balance of confounder configurations.
2. Then we search for an initial configuration A. For BlocktowerCF, we make sure that this configuration is unstable to ensure identifiability. Then we simulate the trajectory B.
3. We look for a valid do-operation such that identifiability and counterfactuality constraints are satisfied. If no valid do-operation is found after a fixed number of trials, we reject this experiment.
4. If a valid pair (AB,CD) is found, we add the sample to the dataset.
The trajectories were simulated with a sample time of 0.04 seconds. The video resolution is 448 × 448 and represents 6 seconds for BlocktowerCF and BallsCF, and 3 seconds for CollisionCF. We invite interested readers to look at our code for more details, such as dooperation sampling, or intrinsic camera parameters. Fig. 10 shows the confounder distribution in the three tasks.

heading: B PERFORMANCE EVALUATION OF THE DE-RENDERING MODULE
text: B.1 IMAGE RECONSTRUCTION
We evaluate the reconstruction performance of the de-rendering module in the reconstruction task. Note that there is trade-off between the reconstruction performance and the dynamic forecasting accuracy: a higher number of keypoints may lead to better reconstruction, but can hurt prediction performance, as the dynamic model is more difficult to learn.
Reconstruction error – We first investigate the impact of the number of keypoints in Table 7 by measuring the Peak Signal to Noise Ratio (PSNR) between the target image and its reconstruction. We vary the number of keypoints among multiples of N , the maximum number of objects in the scene. Increasing the number
of keypoints increases reconstruction quality (PSNR) up to a certain point, but results in degradation in forecasting performance. Furthermore, doubling the number of keypoints only slightly improves reconstruction accuracy. This tends to indicate that our additional coefficients are already sufficient to model finer-grained visual details. Table 3 in the main paper measures the impact of the number of keypoints and the presence of the additional appearance coefficients on the full pipeline including the dynamic model. Table 8 illustrates the impact of the number of keypoints and the additional appearance coefficient on the reconstruction performance alone. As we can see, the addition of the coefficient consistently improves PSNR for low numbers of keypoints (over 2 dB for N keypoints).
The improvement is less visible for larger numbers of keypoints, since 3D visual details could be encoded via keypoints position, hence coefficients become less relevant. Visualizations are shown in Fig. 11.

heading: B.2 NAVIGATING THE LATENT COEFFICIENT MANIFOLD
text: We evaluate the influence of the additional appearance coefficients on our de-rendering model by navigating its manifold. To do so, we sample a random pair (Xsource, Xtarget) from an experiment in BlocktowerCF and compute the corresponding source features and target keypoints and coefficients. Then, we vary each component of the target keypoints and coefficients and observe the reconstructed image (fig. 12). We observed that the keypoints accurately control the position of the cube along both spatial axes. The rendering module does infer some hints on 3D shape information from the vertical position of the cube, exploiting a shortcut in learning. On the other hand, while not being supervised, the coefficients naturally learn to encode different orientations in space and distance from the camera. Interestingly, a form of disentanglement emerges. For example, coefficients n◦ 1 and 2 control rotation around the z-axis, and coefficient n◦4 models rotation around the y-axis. The last coefficient represents both the size of the cube and its presence in the image.

heading: C COMPARISON WITH THE TRANSPORTER BASELINE

heading: C.1 COMPARISON WITH OUR DE-RENDERING MODEL
text: As described in section 4.1, the Transporter (Kulkarni et al., 2019) is a keypoint detection model somewhat close to our de-rendering module. It leverages the transport equation to compute a reconstruction vector:
Ψ̂target = Fsource × (1−Ksource)× (1−Ktarget) + Ftarget ×Ktarget. (9)
This equation allows to transmit information from the input by two means: the 2D position of the keypoints (Ktarget) and the dense visual features of the target (Ftarget). In comparison, our derendering solely relies on the keypoints from the target image and does not require a dense vector to be computed on the target to reconstruct the target. This makes the Transporter incomparable with our de-rendering module. We nevertheless compare the performances of the two models in
Table 13, and provide visual examples in Fig. 13. Even though the two models are not comparable, as the Transporter uses additional information, our model still outperforms the Transporter for small numbers of keypoints. Interestingly, for higher numbers of keypoints Transporter tends to discover keypoints far from the object. We investigate this behavior in the following section, and show that this is actually a critical problem for learning causal reasoning on the discovered keypoints.

heading: C.2 ANALYSIS OF TRANSPORTER’S BEHAVIOR
text: The original version of the V-CDN model (Li et al., 2020) is based on Transporter (Kulkarni et al., 2019). We have already highlighted the fact that this model is not comparable with our task, as it requires not only the target keypoints Ktarget but also a dense feature map Ftarget, whose dynamics can hardly be learned due to its high dimensionality. More precisely, the transport equation (Eq. 9) allows to pass information from the target by two means: the 2D position of the keypoints (Ktarget) and the dense feature map of the target (Ftarget). The number of keypoints therefore becomes a highly sensible parameter, as the transporter can decide to preferably transfer information through the target features rather than through the keypoint locations. When the number of keypoints is low, they act as a bottleneck, and the model has to carefully discover them to reconstruct the image.
On the other hand, when we increase the number of keypoints, Transporter stops tracking objects in the scene and transfers visual information through the dense feature map, making the predicted keypoints unnecessary for image reconstruction, and therefore not representative of the dynamics.
To illustrate our hypothesis, we set up the following experiment. Starting from a trained Transporter model, we fixed the source image to be X0 (the first frame from the trajectory) during the evaluation step. Then, we compute features and keypoints on the target frameXt regularly sampled in time. We reconstruct the target image using the transport equation, but without updating the target keypoints. Practically, this consists in computing Ψ̂target with Eq. (9) substituting Ktarget for Ksource.
Results are shown in Fig. 14. There is no dynamic forecasting involved in this figure, and the Transporter we used was trained in a regular way, we only change the transport equation on evaluation time. Even though the keypoint positions have been fixed, the Transporter manages to reconstruct a significant part of the images, which indicates that a part of the dynamics has been encoded in the dense feature map.
In contrast, this issue does not arise from our de-rendering module, since our decoder solely relies on the target keypoints to reconstruct the image. Note that this is absolutely not contradictory with the claim in Li et al. (2020), since they do not evaluate V-CDN in pixel space. A rational choice of the number of keypoints leads to satisfactory performance, allowing V-CDN to accurately forecast the trajectory in keypoints space, and retrieve the hidden confounders on their dataset.

heading: C.3 TEMPORAL INCONSISTENCY ISSUES
text: Increasing the number of keypoints of the Transporter may lead to temporal inconsistency during the long-range reconstruction. For example, a keypoint that tracks the edge of a cube in the first
frame may target a face of this same cube in the future, since dynamics does not intervene in the keypoint discovery process.
Our de-rendering directly addresses this through the usage of additional appearance coefficients, which allows us to limit the number of keypoints to the number of objects in the scene, effectively alleviating the consistency issue. Fig. 15 illustrates this phenomenon by plotting the discovered keypoint locations forward in time, as well as the 2D location of the center of mass of each object. Note that the Transporter suffers from the temporal inconsistency issue with numbers of keypoints as low as 4 (green cube). In contrast, our model manages to solve the problem and accurately tracks the centers of mass, even though they were never supervised.

heading: D DETAILS OF MODEL ARCHITECTURES

heading: D.1 DE-RENDERING MODULE
text: We call a “block” a 2D convolutional layer followed by a 2D batch norm layer and ReLU activation. The exact architecture of each part of the encoder is described in Table 10a. The decoder hyperparameters are described in Table 10b.
Dense feature map estimator F – We compute the feature vector from Xsource by applying a convolutional network F on the output of the common CNN of the encoder. This produces the source feature vector Fsource of shape (batch, 16, 28, 28).
Keypoin detector K – The convolutional network K outputs a set of 2D heatmaps of shape (batch,K, 28, 28), whereK is the desired number of keypoints. We apply a spatial softmax function on the two last dimensions, then we extract a pair of coordinates on each heatmap by looking for the location of the maximum, which gives us Ktarget of shape (batch,K, 2).
Coefficient estimator C – We obtain the coefficient by applying a third convolutional network C to the output of the common encoder CNN, which again results in a set of 2D vectors of shape (batch,K, 28, 28). These vectors are flattened channel-wise and provide a tensor of shape (batch,K, 28 × 28)) fed to an MLP (see Table 10a for the exact architecture) that estimates the coefficients Ctarget of shape (batch,K,C + 1).
Gaussian mapping G – The keypoint vector Ktarget is mapped to a 2D vector through a Gaussian mapping process :
G(k)(x, y) = exp ( − (x− kx)
2 + (y − ky)2 σ2
) , (10)
where G(k) ∈ R28×28 is the Gaussian mapping of the keypoint k = [kx ky]. We deform these Gaussian mappings by applying convolutions with filters Hi controlled by the coefficients cik.
The filters from H are 5 × 5 kernels that elongate the Gaussian in a specific direction. Practically, we obtain the filter Hi by drawing a line crossing the center of the kernel and with a slope angle of i πC where C is the number of coefficients. We then apply a 2D convolution :
Gik = c C+1 k c i k (G(kk) ∗Hi) . (11)
Note that we also compute a supplementary coefficient αC+1k used as a gate on the keypoints. By setting this coefficient to zero, the de-rendering module can de-activate a keypoint (which is redundant with deactivating the full set of coefficients for this keypoint).
RefinerR – To reconstruct the target image, we channel-wise stack feature vectors from the source with the constructed filters and feed them to the decoder CNNR (Table 10b). We trained the de-rendering module on pairs of images (Xsource,Xtarget) randomly sampled from sequences D. For a given sequence D, we take T − 25 first frames of the trajectory as a source (where T is the number of frames in the video). The last 25 frames are used as a target. For evaluation, we take the 25th frame as the source, and the 50th frame as the target. We use Adam optimizer with a learning rate of 10−3, γ1 = 104 and γ2 = 10−1 to minimize equation 4.

heading: D.2 CODY
text: We describe the architectural choices made in CoDy. Let
s(t) = [kk c 1 k ... c C+1 k k̇k ċ 1 k ... ċ C+1 k ]k=1..K (12)
be the state representation of an image Xt, composed of the K keypoints 2D coordinates with their C + 1 coefficients. The time derivative of each component of the state is computed via an implicit Euler derivation scheme k̇(t) = k(t) − k(t − 1). We use a subscript notation to distinguish the keypoints from AB and CD.
CF estimator – The latent representation of the confounders is discovered from sAB. The graph neural network from this module implements the message passing function f() and the aggregation function g() (see equation 5) by an MLP with 3 hidden layers of 64 neurons and ReLU activation unit. The resulting nodes embeddings hAB(t) = GN (sAB(t)) belong to R128. We then apply a gated recurrent unit with 2 layers and a hidden vector of size 32 to each node in hAB(t) (sharing parameters between nodes). The last hidden vector is used as the latent representation of the confounders uk.
State encoder-decoder – The state encoder is modeled as a GN where the message passing function and the aggregation function are MLPs with one hidden layer of 32 units. The encoded state σCD = E(sCD) lies in R256. We perform dynamical prediction in this σ space, and then project back the forecasting in the keypoint space using a decoder. The decoder ∆(σ(t)) first applies a shared GRU with one layer and a hidden vector size of 256 to each keypoint σk(t), followed by a graph neural network with the same structure as the state encoder.
Dynamic system – Our dynamic system forecasts the future state σ̂(t + 1) from the current estimation σ̂(t) and the confounders u = [u1 ... uK ]. It first applies a graph neural network to the concatenated vector [σ̂(t) u]. The message passing function f and the aggregation function g are MLPs with 3 hidden layers of 64 neurons and ReLU activation function. The resulting nodes embeddings GN ((σ)(t)) belong to R64 and are fed to a GRU sharing weights among each nodes with
2 layers and a hidden vector of size 64. This GRU updates the hidden vector vCD(t) = [v1 ... vK ], that is then used to compute a displacement vector with a linear transformation:
σ̂k(t+ 1) = σk(t) + Wvk(t) + b. (13)
CoDy is trained using Adam to minimize equation 6 (learning rate 10−4, γ3 = 1). We train each CoDy instance by providing it with fixed keypoints states sAB(t) and the initial condition sCD(t = 0) computed by our trained de-rendering module. CoDy first computes the latent confounder representation uk, and then projects the initial condition into the latent dynamic space σ(t = 0) = E(sCD(t = 0)). We apply the dynamic model multiple times in order to recursively forecast T time steps from CD. We then apply the decoder ∆(σ̂(t)) to compute the trajectory in the keypoint space.

heading: E ADDITIONAL QUANTITATIVE EVALUATION

heading: E.1 MULTI-OBJECT TRACKING METRICS
text: When the number of keypoints matches the number of objects in the scene, the keypoint detector naturally and in an unsupervised manner places keypoints near the center of mass of each object (see Figure 15). Leveraging this emerged property, we provide additional empirical demonstration of the accuracy of our model by computing classical Multi-Object Tracking (MOT) metrics. In particular, we computed the Multi-Object Tracking Precision (MOTP) and the Multi-Object Tracking (MOTA) as described in Bernardin & Stiefelhagen.
• MOTA requires to compute the number of missed objects (i.e. not tracked by a keypoints) and the number of false positives (i.e. keypoints that do not represent an actual object). MOTA takes values in [−1, 1], where 1 represents perfect tracking:
MOTA = 1− ∑ tmt + ft + st∑
t gt , (14)
where mt is the number of missed objects at time t, ft is the number of false positives at time t, st is the number of swaps at time t, and gt is the number of objects at time t.
• MOTP is a measurement of the distance between the keypoints and the ground-truth centers of mass conditioned on the pairing process:
MOTP =
∑ i,t d
i t∑
t ct , (15)
where ct is the number of accurately tracked objects at time t and dit is the distance between the keypoint and the center of mass of the ith association {keypoints+center of mass}.
Note that these metrics are related: low MOTP indicates that the tracked objects are tracked precisely, and low MOTA indicates that many objects are missed. Thus, to be efficient, a model needs to achieve both low MOTP and high MOTA.
We also reported the performances of CoPhyNet (Baradel et al., 2020) that predicts counterfactual outcomes in euclidian space using the ground-truth 3D space. As it uses GT object positions during training, it is not comparable and should be considered as a soft upper bound of our method. We present our results in Table 11. This confirms the superiority of our method over UV-CDN in keypoint space. The upper bound CoPhyNet takes advantage of the non-ambiguous 3D representation modeled by the ground-truth state of the object of the scene.
Our method also outperforms CoPhyNet on the ballsCF task, probably due to two phenomena. First, ballsCF is the only 2D task of FilteredCoPhy. Thus, CoPhyNet does not have an advantage of using ground-truth 3D positions. Second, the state-encoder in CoDy projects the 2D position of each sphere in a space where the dynamics is easier to learn, probably by breaking the non-linearity of collisions.
E.2 IMPACT OF THE DO-OPERATIONS
We also measure the impact of the do-operation types on the video forecasting. Fig. 16 (left) is obtained by computing PSNR for each example of the training set and reporting the result on a 2D graph, depending on the amplitude of the displacement that characterizes the do-operation. We applied the same method to obtain Fig.16 (right) that focuses on the type of do-operation, that is moving, removing or rotating an object. These figures are computed using the 2N keypoints models.
Our method generalizes well across different do-operations, including both the type of the operation, and the amplitude. A key to this success it the careful design of the dataset (balanced with respect to the types of do-operations), and a reasonable representation (our set of keypoints and coefficients) able to detect and model each do-operation from images.

heading: F EXPERIMENTS ON REAL-WORLD DATA
text: Our contributions are focused on the discovery of causality in physics through counterfactual reasoning. We designed our model in order to solve the new benchmark and provided empirical evidence that our method is well suited for modeling rigid-body physics and counterfactual reasoning. The following section aims to demonstrate that our approach can also be extended to a real-world dataset. We provide qualitative results obtained on a derivative of BlocktowerCF using real cubes tower (Lerer et al., 2016).
We refer to this dataset as Blocktower IRL. It is composed of 516 videos of wooden blocks stacked in a stable or unstable manner. The amount of cubes in a tower varies from 2 to 4. We aim to predict the dynamics of the tower in pixel space. This is highly related with our task BlocktowerCF (which inspired by the seminal work from (Lerer et al., 2016)) with three main differences: (1) the dataset shows real cube towers, (2) the problem is not counterfactual, i.e. every cube has the same mass and (3) the dataset contains only few videos.
To cope with the lack of data, we exploit our pre-trained models on BlocktowerCF and fine-tune on Blocktower IRL. The adaptation of the de-rendering module is straightforward: we choose the 4 keypoints-5 coefficients configuration and train the module for image reconstruction after loading the weights from previous training on our simulated task. CoDy, on the other hand, requires careful tuning to preserve the learned regularities from BlocktowerCF and prevent over-fitting. Since Blocktower IRL is not counterfactual, we de-activate the confounder estimator and set uk to vectors of ones. We also freeze the weights of the last layers of the MLPs in the dynamic model.
To the best of our knowledge, we are the first to use this dataset for video prediction. (Lerer et al., 2016) and Wu et al. (2017) leverage the video for stability prediction but actual trajectory forecasting was not the main objective. To quantitatively evaluate our method, we predict 20 frames in the future from a single image sampled in the trajectory. We measured an average PSNR of 26.27 dB, which is of the same order of magnitude compared to the results obtained in simulation. Figure 17 provides visual example of the output.

heading: G QUALITATIVE EVALUATION: MORE VISUAL EXAMPLES

Review summaries:
This paper studies an interesting problem of counterfactual video prediction, which aims to predict the future frame (D) based on the initial frame (C) and an observed video sequence (AB). AB can be seen as a demonstration that is driven by the same confounders, i.e., the physics parameters such as mass and initial velocities.   The paper follows the previous models from CoPhyNet for dynamics modeling and confounder estimation, and has two improvements over the work of CoPhy, in my point of view:  - First, it improves the CoPhy benchmark. - Second, it presents a new model based on the representations of high-dimensional features, 2D keypoints, and corresponding coefficients. The new form of representation allows the model to be trained in an unsupervised manner only with the supervision of RGB images, as opposed to the training procedure with the supervision of object positions in CoPhyNet.   

This work extends CoPhy, in which the author(s) address the problem of predicting counterfactual outcomes of physics-based tasks from pixel space (CoPhy used ground truth object positions). To do this, they learn a keypoint representation of the scene and use them to extract the confounders (such as velocities of objects, masses, etc). The author(s) also propose a new benchmark (built upon CoPhy's benchmark) for counterfactual prediction (after intervening the initial set of objects in the scene) which satisfies the Identifiability and the Counterfactuality constraints of causality.

The paper introduces a testing approach, dataset, and method for counterfactual video predictions, using 3d physics simulation videos.  Importantly, the approach predicts directly from pixel space rather than requiring spoon-fed keypoints.  It employs a counterfactual approach to establish a network's capacity to learn causal relations.  Separating the problem into one of parsing the inputs into keypoints and additional coefficients, inferring object attributes, and sequential prediction from an input frame + object attributes, it proposes an architecture which is based on combining modules for each of these learning tasks (but where keypoints are learned in an unsupervised way).  Besides this key architectural innovation, it adds an inductive bias by applying directional Gaussian filters to the keypoint maps.  The paper checks that the network actually works as intended by empirically examining the effect of changing object coefficients.  Three other ablation analyses, one that combines two of the modules (rather than separating them via stop-grad) and one that removes coefficients, and one comparing the handcrafted filter bank with learnable ones, add confidence to the approach.  In the supplement, the paper also analyzes in detail how the method compares to a previous method (Transporter).  It uses sensible benchmarks including previous methods and common-sense baselines where either the original or counterfactual input is used as a prediction, and does relatively well on this challenging problem, although the video on the website shows that there is still plenty of room for improvement.