Filtered-CoPhy: Unsupervised Learning of Counterfactual Physics in Pixel Space
Authors: Steeven JANNY
Fabien Baradel
Natalia Neverova
Madiha Nadri
Greg Mori
Christian Wolf
Venue: ICLR 2022 Oral (2022)

Abstract: Learning causal relationships in high-dimensional data (images, videos) is a hard task, as they are often defined on low dimensional manifolds and must be extracted from complex signals dominated by appearance, lighting, textures and also spurious correlations in the data. We present a method for learning counterfactual reasoning of physical processes in pixel space, which requires the prediction of the impact of interventions on initial conditions. Going beyond the identification of structural relationships, we deal with the challenging problem of forecasting raw video over long horizons. Our method does not require the knowledge or supervision of any ground truth positions or other object or scene properties. Our model learns and acts on a suitable hybrid latent representation based on a combination of dense features, sets of 2D keypoints and an additional latent vector per keypoint. We show that this better captures the dynamics of physical processes than purely dense or sparse representations. We introduce a new challenging and carefully designed counterfactual benchmark for predictions in pixel space and outperform strong baselines in physics-inspired ML and video prediction.

Published as a conference paper at ICLR 2022
FILTERED-COPHY: UNSUPERVISED LEARNING OF COUNTERFACTUAL PHYSICS IN PIXEL SPACE
Steeven Janny
Fabien Baradel
Natalia Neverova
LIRIS, INSA Lyon, France
Naver Labs Europe, France fabt en. @ravar) aba..com
Meta Al
steeeen- staevon-janny@e wow.-aannnnnlooooooooo
nove commom
Madiha Nadri
Greg Mori
Christian Wolf
LAGEPP, Univ. Lyon 1, France
Simon Fraser Univ., Canada morleca. serrlec..sto stu.ca
LIRIS, INSA Lyon, France
madiha madri-wolln wwwnieeeeooonnoee Tyonl. Ir
christian, w...taa.wwo......oo.........o... wol te maaa-lyon... fr
ABSTRACT
Learning causal relationships in nigh-dimensional data (images, videos) is a hard task, they are often defined now-dimensional manifolds and must be ex- tracted from complex signals dominated by appearance, lighting, textures and also spurious correlations in the data. We present method for learning counterfactual reasoning of physical processes in pixel space, which requires the prediction of the impact of interventions on initial conditions. Going beyond the identification of structural relationships, we deal with the challenging problem of forecasting raw video over long horizons. Our method does not require the knowledge or su- pervision of any ground truth positions or other object or scene properties. Our model learns and acts on a suitable hybrid latent representation based on a combi- nation of dense features, sets of 2D keypoints and additional latent vector per keypoint. We show that this better captures the dynamics of physical processes than purely dense or sparse representations. We introduce new challenging and carefully designed counterfactual benchmark for predictions in pixel space and outperform strong baselines in physics-inspired ML and video prediction.
INTRODUCTION
Reasoning on complex, multi-modal and high-dimensional data is natural ability of humans and other intelligent agents (Martin-Ordal et al., 2008), and one of the most important and difficult chal- lenges of AI. While machine learning is well suited for capturing regularities in high-dimensional signals, particular by using high-capacity deep networks, some applications also require an ac- curate modeling of causal relationships. This is particularly relevant in physics, where causation is considered as a fundamental axiom. In the context of machine learning, correctly capturing or mod- robust predictions, in particular better generalization
eling causal relationships can also lead to samples, indicating that a model has overcome the exploitation of biases and
to out-of-distributte shortcuts in the training data. In recent literature on physics-inspired machine learning, causality has often been forced through the addition of prior knowledge about the physical laws that govern the studied phenomena, e.g. (Yin al., 2021). A similar idea lies behind structured causal models, widely used in the causal inference community, where domain experts model these relationships directly graphical notation. This particular line of work allows to perform predictions beyond statistical forecasting, for instance by predicting unobserved counterfactuals, the impact of unob- served interventions (Balke & Pearl, 1994) - "What alternative outcome would have happened, if the observed event X had been replaced with event Y (after an intervention]". Counterfactuals are interesting, as causality intervenes through the effective modification of an outcome. As an exam- ple, taken from (Schölkopf et al., 2021), an agent identify the direction of causal relationship between an umbrella and rain from the fact that removing an umbrella will not affect the weather.
We focus counterfactual reasoning high- dimenssonn signals, in particular videos of complex physical processes. Learning such causal interactions from data is a challenging task, as spurious correlations naturally and easily picked up by trained models. Previous work in this direction

---

Published as a conference paper at ICLR 2022
was restricted to discrete outcomes, as in CLEVRER (Yi et al., 2020), to the prediction of 3D trajectories, as in CoPhy (Baradel et al., 2020), which also requires supervision of object positions. In this work, we address the hard problem of predicting the alternative (counterfactual) outcomes of physical processes in pixel space, í.e. we forecast sequences of 2D projective views of the 31) scene, requiring the prediction over long horizons (150 frames corresponding to 6 seconds). We conjecture that causal relationships can be modeled on low dimensional manifold of the data, and propose a suitable latent representation for the causal model, in particular for the estimation of the confounders and the dynamic model itself. Similar to V-CDN (Kulkami et al., 2019; Li et al., 2020), our latent representation based on the unsupervised discovery of keypoints, complemented by additional information in our case. Indeed, while keypoint-based representations can easily be encoded from visual input, as stable mappings from images to points arise naturally, we claim that they are not the most suitable representation for dynamic models. We identified and addressed two principal problems: (i) the individual points of a given set are discriminated through their 2D positions only, therefore shape, geometry and relationships between multiple moving objects need to be encoded through the relative positions of points to each other, and (ii) the optimal representation for a physical dynamic model is not necessarily 2D keypoint space, where the underlying object dynamics has also been subject to the imaging process (projective geometry).
We propose new counterfactual model, which learns a sparse representation of visual input in the form of 2D keypoints coupled with a (small) set of coefficients per point modeling complementary shape and appearance information. Confounders (object masses and initial velocities) in the studied problem are extracted from this representation, and a learned dynamic model forecasts the entire trajectory of these keypoints from single (counterfactual) observation. Building on recent work in data-driven analysis of dynamic systems (Janny et al., 2021; Peralez & Nadri, 2021), the dynamic model is presented in nigher-dimmnsional state space, where dynamics are less complex. We show, that these design choices are key to the performance of our model, and that they significantly improve the capability to perform long-term predictions. Our proposed model outperforms strong baselines for physics-infomme
learning of video prediction.
We introduce a new challenging dataset for this problem, which builds on CoPhy, a recent coun- terfactual physics benchmark (Baradel et al., 2020). We go beyond the prediction of sequences of 3D positions and propose a counterfactual task for predictions in pixel space after interventions on initial conditions (displacing, re-orienting or removing objects). In contrast to the literature, our benchmark also better controls for the identifiability of causal relationships and counterfactual vari- ables and provides more accurate physics simulation.
2 RELATED WORK
Counterfactual (CF) reasoning - and learning of causal relationships in ML was made popular by works of J. Pearl, e.g. (Pearl, 2000), which motivate and introduce mathematical tools detailing the principles of do-calculas, i.e. study of unobserved interventions links these concepts to the literature in ML (Schölkopf et al., 2021). The last years have seen the emergence of several benchmarks for CF reasoning in physics. CLEVRER (Yi et al., 2020) is vi- sual question answering dataset, where an agent is required to answer a CF question after observing a video showing 3D objects moving and colliding. ei al. (2020) introduce a CF benchmark with two tasks: a scenario where balls interact with each other according to unknown interaction laws (such as gravity or elasticity), and scenario where clothes are folded by the wind. The agent needs to identify CF variables and causal relationships between objects, and to predict future frames. Co- Phy (Baradel et al., 2020) clearly dissociates the observed experiment from the CF one, and contains three complex 3D scenarios involving rigid body dynamics. However, the proposed method relies on the supervision of 3D object positions, while our work does not require any meta data.
data. A more recent survey
Physics-inspired MI. -L--an and learning visual dynamics has been dealt early on with recurrent mod- els (Srivastava et al., 2015; Finn et al., 2016; Lu et al., 2017), or GANs (Vondrick et al., 2016; Mathieu et al., 2016). Kwon & Park (2019) adopt a Cycle-GAN with two discriminator heads, in charge of identifying false images and false sequences in order to improve the temporal consistency of the model in long term prediction. Nonetheless, the integration of causal reasoning and prior knowledge in these models is not straightforward Typical work in physics-informed models relies on disentanglement between physics-informed features and residual features (Villegas et al., 2017a;
2

---

Published as a conference paper at ICLR 2022
Denton & Birodkar, 2017) and may incorporate additional information based on the available priors on the scene (Villegas et al., 2017;; Walker et al., 2017). PhyDNet Le Guen & Thome (2020) ex- plicitly disentangles visual features from dynamiica features, which are supposed to follow a PDE. achieves SOTA performance on Human3.66 6M (lonescu et al., 2014) and Sea Surface Temperature (de Bezenac et al., 20, but we show that it fails on our challenging benchmark.
Keypoint detection - is a well researched problem in vision with widely used handcrafted base- lines (Lowe, 1999). New unsupervised variants emerged recently and have been shown provide suitable object-centric representation, close to attention models, which simplify the use of physical and/or geometric priors (Locatello et al., 2020; Veerapaneni al., 2020). They are of interest in robotics and reinforcement learning, where physical agent has to interact with objects (Kulkarni et al., 2019; Manuelli et al., 2020; 2019). KeypointsNet (Suwajanakorn et al., 2018) is a geometric reasoning framework, which discovers meaningful keypoints in 3D through spatial coherence be- tween viewpoints. Close to our work, (Minderer et al., 2019) proposes learn a keypoints-basee stochastic dynamic model. However, the model is not suited for CF reasoning in physics and may suffer from inconsistency in the prediction of dynamics over long horizons.
3 THE CLTERED-COPPH BENCHMARK
We build on CoPhy (Baradel et al., 2020), retaining its strengths, but explicitly focusing on coun- terfactual scenario in pixel space and eliminating the ill-posedness of tasks we identified in the existing work. Each data sample is called an experiment, represented as a pair trajectories: an observed one with initial condition Xo = A and outcome XI=1.T = B (a sequence), and a counter- factual one Xo = C and XI-1..T = D (a sequence). Throughout this paper we will use the letters B,B.C. C and D to distinguish the diffeernnt parts of each experiment. The initial conditions A and C are linked through a do-operator do Xn = C), which modifies the initial condition (Pearl, 2018). Experiments parameterized by set of intrinsic physical parameters which are not observable from a single initial image A. We refer to these as confounders. As in CoPhy, in our benchmark the do-operator is observed during training, but confounders are not not they have been used generate the data, but are not used during training or testing. Following (Pearl, 2018), the counterfactual task consists in inferring the counterfactual outcome D given the observed trajectory AB and the counterfactual initial state C, following a three-step process:
Abduction: use the observed data AB to compute the counterfactual variables, i.e. physi- cal parameters, which are not affected by the do-operation.
update the causal model; keep the same identified confounders and apply the do- operator, i.c. replace the initial state A by C.
Action:
Prediction: Compute the counterfactual outcome D using the causal graph.
The benchmark contains three scenarios involving rigid body dynamics. BlocktowerCF studies stable and unstable 3D cube towers, the confounders are masses. Ba sCF focuses on 2D collisions between moving spheres (confounders are masses and initial velocities). CollisionCF is about collisions between sphere and cylinder (confounders masses and initial velocities) (Fig. 1).
Unlike CoPhy, our benchmark involves predictions in RGB pixel space only. The do-operation consists in visually observable interventions A, such moving removing an object. The confounders cannot be identified from the single-frame observation A, identification requires the analysis of the entire AB trajectory.
Identifiability of confounders - For an experiment (AB, CD, 2) to be well-posed, the con- founders must be retrievable from AB. For example, since the masses of stable cube tower cannot be identified generally in all situations, it can be impossible to predict the counterfactual outcome of an unstable tower, as collisions are not resolvable without known masses. In contrast to CoPhy, we ensure that each experiment e (Xo,z) + XI=1.T, given initial condition X₂ and confounders 2, well posed and satisfies the following constraints:
Definition (Identifiability, (Pearl, 2018)) The experiment (AB, CD, is identifiable if for any sei of confounders z': ((A,)) = (bA, (CC =C)=').
3

---

Published as a conference paper at ICLR 2022
(a) BlocktewerCF (BT-CF)
(b) CollisionCE (C-CF)
(c) BallsCF (B-CF)
Figure 1: The Filtered-CoPhy benchmark suite contains three challenging scenarios involving 2D or 3D rigid body dynamics with complex interactions, including collision and resting contact. Initial conditions A are modified to C by an intervention. Initial motion is indicated through arrows.
In an identifiable experiment there is no pair (z, that gives the same trajectory AB but different counterfactual outcomes CD. Details implementation and impact are in appendix A.I.
Counterfactuality - We enforce sufficient difficulty of the problem through the meaningfulness of confounders. We move initial situations where the choice of confounder values has no significant impact on the final outcome:
Definition 2 (Counterfaetuality) Let 2h be the set of con- founders 2, where the 1tth value has been modified. The ex- periment (AB, CD, )) is counterfactual if and only if:
3k (CC 2 (C.
In other words, impose the existence of an object of the scene for which the (unobserved) physical properties have a determining effect the trajectory. Details on how this con- straint was enforced are given in appendix A.2.
Figure 2: Impact of temporal fre- quency dynamics, 3D trajecto- ries of each cube are shown. Black dots are sampled
5 FPS, col- ored dots at 25 FPS. Collisions be- tween the red cube and the ground are not well described by the black dots, making it hard infer physi- cal laws from regularities in data.
Temporal resolution the physical laws target involve highly non-linear phenomena, in particular collision and rest- ing contacts. Collisions are difficult to learn because their tions are both intense, brief, and highly non-linear, depend- ing on the geometry of the objects in 3D space. The temporal resolution of physical simulations is of prime importance. A frequency, as
parallel can be made with Nyquist-Shamme trajectory sampled with too low frequency cannot be reconstructed with precision. We simulate and record trajectories at FPS, compared to FPS chosen in CoPhy, justified with two experiments. Firstly, Fig. shows the trajectories of the center of masses of cubes in Bl ock locktoo cowkowe colored dots are shown at 25 FPS and black dots at FPS. We can see that collisions with the ground fall below the sampling rate of 5 FPS, making it hard infer physical laws from regularities in data at this frequency. A second experiment involves learning a prediction model at different frequencies, confirming the choice 25 FPS details given in appendix A.3.
4 UNSUPERVISED LEARNING OF COUNTERFACTUAL PHYSICS
We introduce new model for counterfactual learning of physical processes capable of predicting visual sequences the image space over long horizons. The method does not require any super- vision other than videos of observed and counterfactual experiences. The code is publicly available online at https: htlps://ffitteeeee //iltteeeccohhy github. io. The model consists of three parts, learning the latent representation and its (counterfactual) dynamics:
The encoder Dee-Rendging module) learns a hybrid representation of an image in the form of a (i) dense feature map and (ii) 2D keypoints combined with (iii) low-dimensional vector of coef-
4

---

Published as a conference paper at ICLR 2022
encoddr
decoder D
CNN K.
cargee
R
Ntarget
GL
keypoiuss &z
H
torget
cooffctents
Figure 3: We de-render visual imput into a latent space composed of a dense feature map F modeling static information, a set of keypoints k, and associated coefficients c. We here show the training configuration taking as input pairs (X sorrce X (urge) of images. Without any supervision, a tracking strategy emerges naturally through the unsupervised objective: we optimize reconstruction of Largets given features from source and keypoints keypoints tcoefficiens from Xurget-
ficients, see Fig. 3. Without any state supervision we show that the model learns representation which encodes positions in keypoints and appearance and orientation in the coefficients.
The Counterfactual Dynamic (CoDy) model hased on recurrent graph networks, in the lines of (Baradel al., 2020). estimates a latent representation of the confounders from the keypoint + coefficient trajectories of AB provided by the encoder, and then predicts D in this same space.
The decoder that uses the predicted keypoints to generate pixel-space representation of D.
4.1 DISENTANGLING VISUAL INFORMATION FROM DYNAMICS
The encoder takes an input image X and predicts representation with three streams, sharing common conv. backbone, as shown in Fig. 3. We propose an unsupervised training objective. which favors the emergence of latent representation disentangling static and dynamic information.
1. dense feature map F F(X), which contains static information, such as the background.
2. A set of 2D keypoints k (X),, which carry positional information from moving objects.
3. A set of corresponding coefficients C(X), one vector of size C' per keypoint k, which encodes orientation and appearance information.
The unsupervised objective is formulated on pairs of images (X (Xounces Xarget) randomly sampled from same D sequences (see appendix D.1 details on sampling). Exploiting an assumption on the absence of camera motion', the goal is to favor the emergence of disentangling static and dynamic information. To this end, both images are encoded, and the reconstruction of the target image is predicted with decoder DD fusing the source dense feature map and the target keypoints and coefficients. This formulation requires the decoder to aggregate dense information from the source and sparse values from the target, naturally leading motion being predicted by the latter.
On the decoder D side, we add inductive bias, which favors the usage of the 2D keypoint information in spatial way. The 2D coordinates kg for each keypoint k encoded as Gaussian heatmaps (kk)), i.e. 2D Gaussian functions centered on the keypoint position. The additional coefficient information, carrying appearance information, is then used to deform the Gaussian mapping into an anisotropic shape using a fixed filter bank H, as follows:
D(F, k, c) = Gi,.., Gf GGG G.,,
G = C+1 (O(k)) H.),
(3)
where R(...) is refinement network performing trained upsampling with transposed convolutions,
whose inputs are stacked channelwise. Gj, are Gaussian mappings produced from keypoint positions k, deformed by filters from bank H and weighted by coefficients ct. The filters H; defined as fixed horizontal, vertical and diagonal convolution kernels. This choice is discussed in section 5.
The joint encoding and decoding pipeline is illustrated in Fig.
If this assumption is satisfied, global camera motion could be compensated after estimation.

---

Published as a conference paper at ICLR 2022
The model is trained to minimize the mean squared error (MSE) reconstruction loss, regularized with a loss on spatial gradients VX weighted by paperepas 71,
12:72 E R:
Xnager Xiagos ||
Larren 71
72
V V target
target
(4)
where Xunget - D(F(( (F sounce) ), (urger)))) C( (arrge))) is the reconstructed image, Y1, 12 are weights.
unsupervised objective is somewhat related Transporter (Kulkamn et al,, 2019), which, as our model, computes visual feature vectors Fsuurce and Fuurget as well as 2D key- points source and Karget modeled as a 2D vector via Gaussian mapping. It leverages handcrafted transport equation: L turgett= Fource x soure) x (ane))xx1xKKK)a) (u-KKaga) + Faarger x Kurget. As in our case, the target image is reconstructed through a refinr network X Xurger = R(v The transporter suffers from major drawback when used for video prediction, as requires parts the target image to reconstruct the target image- the model was originally proposed in the context of RL and control, where reconstruction is not an objective. also does not use shape coefficients, requiring shapes to be encoded by several keypoints, or abusively be carried through the dense features Ftarget- This typically leads to complex dynamics non representative of the dynamical objects. We conducted an in-depth comparison between the Transporter and representation in appendix C.2.
Related work
4.2 DYNAMIC MODEL AND CONFOUNDER ESTIMATION
Our counterfactual dynamic model (CoDy) leverages multiple graph network (GN) based modules (Battaglia et al., 2016) that join forces to solve the counterfactual forecasting tasks of Filtered- CoPhy. Each one of these networks is a classical GN, abbreviated as GN(x;), which contextu- alizes input node embeddings X through incoming edge interactions eik, providing output node embeddings & (parameters are not shared over the instances):
GX((x)) = &x, such that &t = Xx.
Σ eik
with eij = (x::
(5)
where f message-passing function and g is an aggregation function.
We define the state of frame X, at time i. as stacked vector composed of keypoints and coefficients
computed by the encoder (the de-rendering module), i.e. s(t) [st (t) Sy (t)] where sx(t)
[kg cf Cj C+1 (1). the lines of (Baradel et al., 2020), given the original initial condition and outcome AB, CoDy estimates an unsupervised representation ug of the latent confounder variables per kcypoint through the counterfactual estimator (CF estimator in Fig. 4). It first contextualizes the sequence s (t) through graph network (NABB)(tt)
AB (t) = [h, (t) ... hx(t)), We then model the temporal evolution of this representation with a gated recurrent unit (Cho et al., 2014) per keypoint, sharing parameters
keypoints, taking as input the sequence hę- Its last hidden vector is taken as the confounder estimate ug.
Recent works on the Koopman Operator (Lusch et al., 2018) and Observer (Janny et al., 2021; Peralez & Nadri, 2021) have theoretically shown that, under mild assumptions, there exists a latent space of higher dimension, where dynamical system given as an EDP can have simpler dynamics. Inspired by this idea, we used an encoder-decoder structure within CoDy, which projects our dynamic system into a higher-dimensional state space, performs forecasting of the dynamics in this latent space, and then projects predictions back to the original keypoint space. Note that this dynamics encoder/decoder different from the encoder/decoder of the de-renderinggreenerrng modules discussed in section 4. The state encoder E(s(t))
(1) is modeled as a graph network GN, whose aggregation function projects into an output embedding space ar (t) of dimension 256. The decoder (ta s(t) temporally processes the individual contextualized states or (t) with a GRU, followed by new contextualization with graph network GN. Details on the full architecture provided in appendix D.2.
The dynamic model CoDy performs forecasting in the nigher-dimensionee space ( computing displacement vector 5(t + 1) such that at + 1) (tt) + 5(t + 1). It takes the projected state embeddings CD (t) per keypoint & concatenated with the confounder representation uę and contextualizes them with a graph network, resulting in embeddings h (t) = GN (t), u.l). which are processed temporally by GRU. We compute the displacement vector time

---

Copy assumes absence of intervention (always outputs sequence B). Published as a conference paper at ICLR 2022
AB
Fou uon AB
CoDy
encoder
stop grad
CoDY meoddr
Fentures
decsder
encoder
dy tytanzics
decoder
leat D
recossomacioo
D
encoder
keypoiuts and codficieuts
Figure 4: During training, we disconnect the dynamic prediction module (CoDy) from the rendering module (decoder). On test time, we reconnect the two modules. CoDy forecasts the counterfactual outcome D from the sparse keypoints representation of AB and C. The confounders are discovered in an unsupervised manner and provided to the dynamical model.
Ours
UV-CDN
PhyD Pred NET RNN
Copy
Copy
Ours
2N
2N
BT-CF
43.2
20.0
9.58
B-CF C-CF
44.3
92.3
36.12
PSNR
23.48
24.69
21.07
21.99
16.49 22.04
BT-CF
7.6
40.3
5.14
L-PSNR
25.51
26.79
22.36
23.64
23.03 24.97
PSNR
21.19
21.33
19.51
19.54
18.56 22.31
B-CF
L-PSNR
23.88
24.12
22.35
22.38
22.55 22.63
C-CF
PSNR
24.09
24.09
23.73
23.83
19.69 24.70
Copy C: assumes that the tower stable (always outputs input C).
L-PSNR
26.07 26.55
26.08
26.34
24.61 26.39
Table 1: Comparison with the state-of-the-arn models in physics-inspired machine learning of video signals, report- ing reconstruction error (PSNR and introduced L-PSNR).
Comparison with copying baselines. We report MSEx 10 on prediction of keypoints coefficients.
Table 2:
linear transformation from the hidden state of the GRU. We apply the dynamic model in auto-regressive way to forecast long-term trajectories in the projected space dx, and apply the state decoder obtain prediction (CD(U) (t).
The dynamic model is trained with a loss keypoint space,
Cphysics Σ
||s CD s (C) 73l1s CD A(E(S
The first term enforces the model to learn to predict the outcomes and the second term favors correct reconstruction of the state in keypoint space. The terms are weighted with a scalar parameter "/-
4.3 TRAINING
End-to-end training of all three modules jointly challenging, as the same pipeline controls both the keypoint- based state representation and the dynamic module (CoDy), involving two adversarial ob- jectives: optimizing reconstruction pushes the keypoint encoding to be as representative as possible, but learning the dynamics favors a simple representation. Faced with these two contradictory tasks, the model numerically unstable and rapidly converges to regression to the mean. As described above, separately train the encoder+decoder pair without dynamic information reconstruction only, c.f. Equation (4). Then we freeze the parameters of the keypoint detector and train CoDy to forecast the keypoints from D minimizing the loss in Equation (6).
EXPERIMENTS
We compare the proposed model to three strong baselines for physics-inspired video prediction.
PhyDNet (Le Guen & Thome, 2020) is non-counterfactual video prediction model that forecasts future frames using a decomposition between (a) a feature vector that temporally evolves via an L.TTM and (b) dynamic state that follows PDE learned through specifically designed cells.

---

Published as a conference paper at ICLR 2022
Coefficients:
x
R
# Keypoints:
N
2N
4N
N
2N
4N
(G
PSNR
23.48
24.69
23.54
22.71
23.28
23.17
BT-CF
L-PSNR
21.75
23.03
21.80
21.18
21.86
21.70
PSNR
21.19
21.33
21.37
20.49
21.09
20.97
B-CF
L-PSNR
27.88
27.16
27.07
26.33
27.07
26.73
PSNR
24.09
24.09
24.26
23.84
23.66
24.06
C-CF
L-PSNR
23.32
23.46
23.44
22.58
22.81
23.45
Figure 5: Network R. learns distortion from multiple ori- ented ellipses to target shapes.
Table 3: Impact of having additional orientation/shape co-
efficients (V) compared to the keypoint-only solution (X), for different numbers of keypoints: equal to number of ob- jects (= N), 2N and 4N.
V-CDN (Li et al., 2020) is counterfactual model based keypoints, close our work. It identifies confounders from the beginning of a sequence and learns a keypoint predictor through auto-encoding using the Transporter equation (see discussion in Sect. 4.1). As it is, it cannot be used for video prediction and is incomparable with our work, see details in appendix C. We therefore replace the Transporter by our own de-renddring/renddrrnng modules, from which we remove the additional coefficients. We refer to this model as UV-CDN (for Unsupervised V-CDN).
PredRNN (Wang et al., 2017)ix is a ConvLSTM-BB -bseed video prediction model that leverages spatial and temporal memories through spatiotemporal LSTM cell.
All models have been implemented in PyTorch, architectures are described in appendix D. For the baselines PhyDNet, UV-CDN and PredRNN, we used the official souree code provided by the au- thors. We evaluate on each scenario of Filtered-CoPhy on the counterfactual video prediction task. For the two counterfactual models (Ours and UV-CDN), we evaluate on the tasks as intended: we provide the observed sequence AB and the CF initial condition C, and forecast the sequence D. The non-CF baselines are required predict the entire video from a single frame, in order to prevent them from leveraging shortcuts in part of the video and bypass the need for physical reasoning.
We measure performance with time-averaged peak signal-to noise ratio (PSNR) that directly mea- sures reconstruction quality. However, this metric is mainly dominated by error on the static back- ground, which is not our main interest. We also introduce Localized PSNR (L-PSNR), which mea- sures area error the important regions near moving objects, computed on masked images. We compute the masks using classical background subtraction techniques.
Comparison to the SOTA We compare model against UV-CDN, PhyDNet and PredRNN Table 1, consistently and significantly outperforming the baselines. The gap with UV-CDN is particularly interesting, it confirms the choice of additional coefficients to model the dynamics of moving objects. PredRNN shows competitive performances, especially on co collist lisioonCF How- ever, our localized PSNR tends indicates that the baseline does reconstruct accurately the foreground, favoring the reconstruction of the background to the detriment of the dynamics of the scene. Fig. visualizes the prediction on single example, more can be found in appendix G. We also compare to trivial copying baselines in Table 2, namely Copy B, which assumes no intervention and outputs the sequence, and Copy C, which assumes a stable tower. We evaluate these models in keypoints space measuring MSE on keypoints and coefficients averaged over time, copying baselines are unbeatable in the regions of static background, making the PSNR metrics unusable.
We provide additional empirical results by comparing the models using Multi-object Tracking met- rics and studies on the impact of the do-operations on PSNR in appendix E. We also compute an upper bound of our model using CoPhyNet bascline described in Baradel et al. (2020).
Performance on real-world data - is reported in appendix F, showing experiments on 516 videos of real wooden blocks introduced in (Lerer et al., 2016).
Impact of appearance coefficients - are reported Table 3, comparing to the baseline using keypoint-only representation. The coefficients have significant impact: even increasing the number of keypoints to compensate for the loss information cannot overcome the advantage of disentangling positions and shapes, done in our model. We provide deeper analysis of the
8

---

Published as a conference paper at ICLR 2022
H42
H45
M45
(412)
1-145
Elone Of alee.
Figure 6: Visualization of the counterfactual video prediction quality, comparing our proposed
model (Filtered CoPhy) with the baselines. PhyDNet and UV-CDN, over different timestamps.
State auto-encoder:
X
Filter bank:
Fixed
Learned
BT-CF
9.58
11.10
BT-CF
34.40
32.04
B-CF
36.12
36.88
B-CF
37.76
31.25
C-CF
5.14
16.16
C-CF
34.09
33.88
Table 4: Impact of the dynamic CoDy encoder
Table 5: Learning the filter bank from scratch has mild negative effect on the reconstruction task. We report the PSNR on static reconstruc- tion performance without the dynamic model.
(V) against the baseline operating the keypoint + coefficient space (X). We report MSEx 10 on prediction of keypoints + coefficients pts).
ce-rrendering/rendeinng modules in appendix B, which includes visualizations of the navigation of the latent shape space in B.2.
Learning filters - does not have positive impact on reconstruction performance compared to choice of the handcrafted bank, freedom are redundant with parts the filter kernels in the refinement module R: this corresponds to jointly learning multi-channel representation {G² V--.-CC for shapes well the mapping which geometrically distorts them into the target object shapes. Fixing the latent representation Fig. 5.
be seen in table 5. We conjecture that the additional degrees of
k
does not constrain the system, the mapping R can adjust it
Impact of the high-dimensional dynamic space - We evaluate the impact of modeling object dynamics in high-dimensional space through the CoDy encoder in Table comparing projection to 256 dimensions to the baseline reasoning directly in keypoint + coefficient space. The experiment confirms this choice of KKL-like encoder (Janny al., 2021).
CONCLUSION
We introduced new benchmark for counterfactual reasoning in physical processes requiring to per- form video prediction, i.c. predicting raw pixel observations over long horizon. The benchmark has been carefully designed and generated imposing constraints on identifiailltt and counterfac- tuality. We also propose new method for counterfactual reasoning, which is based on hybrid latent representation combining 2D keypoints and additional latent vectors encoding appearance and shape. We introduce unsupervised learning algorithm for this representation, which does not require any supervision on confounders or other object properties and processes raw video. Coun- terfactual prediction of video frames remains challenging task, and Filtered CoPhy still exhibits failures in maintaining rigid structures of objects over long prediction scales. We hope that our benchmark will inspire further breakthroughs in this domain.

---

Published as a conference paper at ICLR 2022
REFERENCES
Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications. In UAI, 1994.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counter- factual learning of physical dynamics. In International Conference on Learning Representations, 2020.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Proceedings of the 30th International Conference Neural Information Processing Systems, 2016.
Keni Bernardin and Rainer Stiefelhagen. Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metries. EURASIP Journal on Image and Video Processing. 2008.
Kyunghyun Cho, Bart van Merriënhoer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing, 2014.
Emmanuel de Bezenac, Arthur Pajot, and Patrick Gallinari. Deep learning for physical processes: Incorporating prior scientific knowledge. In International Conference on Learning Representa- tions, 2018.
Emily L. Denton and Vighnesh Birodkar. Unsupervised learning of disentangled representations from video. ArXiv, abs/1705.10915, 2017.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. In Proceedings of the 30th International Conference Neural Infor- mation Processing Systems, 2016.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m 3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.
Steeven Janny, Madiha Nadri Vincent Andrieu, and Christian Wolf. Deep kkl: Data-driven output prediction for non-linear systems. In Conference on Decision and Control (CDC21), 2021.
Tejas D. Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, Malcolm Reynolds, An- drew Zisserman, and Volodymyr Mnih. Unsupervised learning of object keypoints for perception and control. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, 2019.
Y. Kwon and M. Park. Predicting future frames using retrospective cycle gan. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1811-1820, 2019.
Vincent Le Guen and Nicolas Thome. Disentangling physical dynamics from unknown factors for unsupervised video prediction. In Computer Vision and Pattern Recognition (CVPR), 2020.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48. JMLR.org. 2016.
Yunzhu Li. Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg, Causal discov- ery in physical systems from videos. Advances in Neural Information Processing Systems, 33, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, G. Heigold, Jakob Uszkoreit, A. Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. ArXiv, abs/2006.15055. 2020.
David G. Lowe. Object recognition from local scale-invariant features. In ICCV, 1999.
10

---

Published as a conference paper at ICLR 2022
Chaochao Lu, Michael Hirsch, and Bernhard Schölkopf. Flexible apatio-temponn networks for video prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Bethany Lusch, Nathan Kutz, and Steven Brunton. Deep learning for universal linear embeddings of nonlinear dynamics. Nature Communicationss 2018.
Lucas Manuelli, Wei Gao, Peter R. Florence, and Russ Tedrake. kpam: Keypoint affordances for category-level robotic manipulation. ArXiv, abs/ abs//993.06684, 2019.
Lucas Manuelli, Yunzhu Li, Peter R. Florence, and Russ Tedrake. Keypoints into the future: Self- supervised correspondence in model-based reinforcement learning. CoRR, 2020.
G. Martin-Ordas Call, and F. Colmenares. Tubes, tables and traps: great apes solve twofunction- ally equivalent trap tasks but show no evidence of transfer across tasks. Animal cognition, 11(3): 432 430, 2008
Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square emror. In 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, K. Murphy, and Honglak Lee. Un- supervised learning of object structure and dynamics from videos. ArXiv, abs/1906.07889, 2019.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2000.
Judea Pearl. Causal and counterfactual inference. The Handbook of Rationality, pp. 1-41, 2018.
Johan Peralez and Madiha Nadri. Deep learning-based luenberger observer design for discrete-time non-linear systems. In Conference on Decision and Control (CDC21), 2021.
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEFE, 2021.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of video representations using Istms. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, 2015.
Supasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, and Mohammad Norouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. In NeurIPS 2018, 2018.
Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. Proceedings of the Conference on Robot Learning, 2020.
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. ArXiv, volume abs/1706.00 08003,, 2017a
Ruben Villegas, Jimei Yang, Yuliang Zou, Sungryull Sohn, Xunyu Lin, and Honglak Lee. Learning to generate long-term future via hierarchical prediction. In ICMI., 2017b.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. In Proceedings of the 30th International Conference on Neural Information Processing Systems, 2016.
Jacob Walker, Kenneth Marino, Abhinav Gupta, and M. Hebert. The pose knows: Video forecasting by generating pose futures. 2017 IEEE International Conference on Computer Vision (ICCV), pp- 3352-3361, 2017.
Yunbo Wang, Mingsheng Long, Jianmin Wang. Zhifeng Gao, and Philip S Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istms. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2017.

---

Published as a conference paper at ICLR 2022
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to physics via visual de- animation. In Advances in Neural Information Processing Systems, 2017.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. CLEVRER: collision events for video representation and reasoning. In ICLR, 2020.
Yuan Yin, Vincent LE GUEN, Jérémie DONA, Emmanuel de Bezenac, Ibrahim Ayed, Nicolas THOME, and patrick gallinari. Augmenting Physical Models with Deep Networks for Com- plex Dynamics Forecasting. In International Conference on Learning Representations (ICLR), 2021.
12

---

Published as a conference paper at ICLR 2022
Appendix
A FURTHER DETAILS ON DATASET GENERATION
Confounders in our setup are masses, which we discretize in {1, 10). For BallsCF and CollisionCF, we can also consider the continuous initial velocities of each object as confounders variables, since they have to be identified in AB to forecast CD.
We simulate all trajectories associated with the various possible combinations of masses from the same initial condition.
Do-interventions, however, depend on the task. For BlocktowerCF and BallsCF, do- interventions consist in (a) removing the top cube or ball, or (b) shifting cube/ball on the hori- zontal plane. this case, for owerCF, we make sure that the cube does not move too far from the tower, in order to maintain contact. For Collic lilissii onCF, the do-interventions are restricted to shifting operations, since there are only two objects (a ball and a cylinder). It can consist of either switch of the cylinder's orientation between vertical or horizontal, moving object relative to the resting one in one of the three canonical directions I, y and 2.
shift of the position of the
A.1 ENFORCING THE IDENTIFIABILTY CONSTRAINT
The identifiabilty and counterfactuality constraints described in section 3 are imposed numerically, ie. first sample and simulate trajectories with random parameters and then reject those that violate these constraints.
As stated in section 3, an identifiable experiment guarantees that there no pair (2,2') that gives the same trajectory AB but a different counterfactual outcome CD. Otherwise, there will be no way to choose between z and z' only from looking at AB, thus no way to correctly forecast the counterfactual experiment. By enforcing this constraint, we make sure that there exists at least set = ...) of confounders that give at the same time similar observed outcomes AB and similar counterfactual outcomes CD.
In practice, there exists a finite set possible variables ū, corresponding to every combination of masses for each object in the scene (masses take their value in {1, 10}). During generation, we submit each candidate experiment (AB, CD,.) 2) to a test ensuring that the candidate is identifiable. Let (Xo,, 2) be the function that gives the trajectory of system with initial condition Xo and confounders z. We simulate all possible trajectories (H,A,,) and (C(C, 2;) for every possible 2;- If there exists z' A such that the experiment is not identifiable, the candidate is rejected. This constraint requires to simulate the trajectory of each experiment several times by modifying physical properties of the objects.
Equalities in Definition 1 are relaxed by thresholding distances between trajectories. We reject a candidate experiment if there exists such that
T Σ (Iv(A.)) 2(A,2))) < and Σ I
The choice of the threshold value e critical, in particular for the identifiability constraint:
If the threshold is too high, all AB trajectories will be considered as equal, which results in acceptance of unidentitiable experiments.
If the threshold is too low, all trajectories CD are considered equal. Again, this leads to mistakenly accepting unidentifiable experiments.
There exists an optimal value for E, which allows to correctly reject unidentifiable experiences. To measure this optimal threshold, generated small instance of the Blocktowerde dataset without constraining the experiments, iec trajectories can be unidentifiable and non-counterfactual We then plot the percentage of rejected experiments in this unfiltered dataset against the threshold
13

---

Published as a conference paper at ICLR 2022
Throdool grid
Thnoheid grd for conntrfactualty
1075
Paocttgee sintitalic sleniitalccoooiinn cxpoiict
9099
6999
1075
609
275
50%
05
162 78
Thedhold
Theshodd
Figure 7: Experimental tuning of the threshold parameter. We generate unconstrained subset of BlocktowerCF and plot the percentage of identifiable experiments as function of the threshold s.
without constraint
with constraint
FPS
|
15
25
35
45
MSE
56% Corrected Acc.
Accuracy
84% 91%
(x10-3)
4.58
3,97
3.74
3.82
3.93
58%
(b)
(a)
Table 6: (a) Sanity check of the identifiability constraint in BlocktowerCP, which results in better estimation of cube masses. The corrected accuracy only considers those cubes for which changes in masses are consequential for the trajectory D. (b) MSE between ground truth 3D positions and predicted positions after second, depending on the sampling rate of the trajectory.
value (Fig. 7, left). We chose the threshold = 100 which optimizes discrimination and rejects the highest number of "unidentifiable" trajectories.
To demonstrate importance of this, we train recurrent graph network on BlocktowerCF to predict the cube masses from ground-truth state trajectories AB, including pose and velocities, see Fig. 8. It predicts each cube's 's mass by solving a binary classification task. We train this model on both Blockt owerCP and an alternative version of the scenario generated without the identifiability constraint. The results shown in Table 6a. We not aiming 100% accuracy, and this problem remains difficult in a sense that the identifiability constraint ensures the identifiability of a set of confounder variables, while sanity check tries predict unique
However, the addition of the identifiability constraint to the benchmark significantly improves the model's accuracy, which indicates that the property acts positively the feasability of Filtered- CoPhy. The corrected accuracy metric focuses solely on the critical cubes, i.e. those cubes whose masses directly define the trajectory CD.
A.2 ENFORCING THE COUNTERFACTUALITY CONSTRAINT
Let (AB, CD, 2) be a candidate experiment, and z be combination of masses identical to except for the th value. The counterfactuality constraint consists checking that there exists at least one k such that =) f (bCC, (,C,,** =*), To do so, we simulate (CC,, x*) for all & and measure the difference with the candidate trajectory (,CC,2, 2). Formally, we verify the existence of such that:
- <E
(8)
A.3 ANALYZING TEMPORAL RESOLUTION
We analyzed the choice of temporal frequency for the benchmark with another sanity check. We simulate non-counterfactual dataset from Bl ocktowerCF where all cubes have equal masses. A
14

---

Published as a conference paper at ICLR 2022
68
BILD
Fosition Unstemian Lin.
Podik Quainsss
Deccrder
Padio
Quleersee
1e-0.7
MILP +R (0,30)
AB states
Qu
CI viete
CD slatts
Figure 8: Impact of the choice of temporal resolution. Left: We check the identifiability constraint by training model predict the cube masses Blocktowerse from the observed trajectory AB. The model is graph neural network followed by gated recurrent unit. Right: We check the augmentation of the sampling rate by training an agent forecast second-length trajectory from the states of the previous second.
Figure 9: Visual examples of the impact of temporal resolution on dynamical information for each task in Filtered-CoPhy. Black dots are sampled at FPS while red dots are sampled at 25 FPS.
recurrent graph network takes as input cube trajectories (poses and velocities) over a time interval of second and predicts the rest of the trajectory over the following second. We vary the sampling frequency; for example, at 5 FPS, the model receives measurements, and predicts the next 5 time steps, which correspond to positions between the predictions and the GT on the last prediction. Results are shown in Table 6b. This check shows clearly that 25 FPS corresponds the best trade-off between an accurate representation of the collision and the amount of training data. Fig. 9 shows practical example of the effect of time resolution on dynamical information trajectory.
second rollout in the future. Finally, compare the error in 3D
A.4 SIMULATION DETAILS
used Pybullet as physics engine to simulate Filtered-CoPhy, Each experiment designed to respects the balance between good coverage of confounder combinations and counterfactuality and identifiability constraints described above. We generate the trajectories iteratively:
1. We sample a combination of masses and other physical characteristics of the given experiment, such as stability of the tower, or object motion Colli lelliss iillisionCF if the do-operation consists in removing an object. This allows to maintain balance of confounder configurations.
Then we search for an initial configuration A. For Bl glocktowerCP we make sure that this configuration unstable ensure identifiability. Then simulate the trajectory B.
15

---

Published as a conference paper at ICLR 2022
ballaCP
Blocktower27
callixient?
10
1.
3.1.3
10,
Ex.
10
1.100
1.1.1
1.1
L.
300
1000
1500
2000
200
410
630
300
3000
2000
4000
5000
Experiments
of expoiments
oiperimntt
Figure 10: During the dataset generation process, we carefully balance combinations of masses, i.c. the confounders. For Bl Blocktows CF,, we also guarantee that the proportion of stable CD towers is close to 50% for each confounder configuration.
We look for a valid do-operation such that identifiability and counterfactuality constraints are satisfied. no valid do-operation is found after fixed number of trials, reject this experiment.
4. If a valid pair (AB, CD) is found, we add the sample to the dataset.
The trajectories were simulated with a sample time of 0.04 seconds. The video resolution is 448 448 and represents 6 seconds for Blocktowers and BallsCF, and seconds for DollisionCP, We invite interested readers to look at our code for more details, such as do- operation sampling, intrinsic camera parameters. Fig. 10 shows the confounder distribution in the three tasks.
B PERFORMANCE EVALUATION OF THE DE-RENDERING MODULE
B. IMAGE RECONSTRUCTION
We evaluate the reconstruction performance of the de-rendering module in the reconstruction task. Note that there is trade-off between the reconstruction performance and the dynamic forecasting accuracy: a higher number of key- points may lead to better reconstruction, but can hurt prediction performance, as the dynamic model is more difficult to learn.
Keypoints
N
2N
4N
PSNR
34.40
35.41
34,92
BT-CF
MSE Grad
27.24
21.39
23,99
PSNR
37.76
37.06
36.98
B-CF
MSE Grad
3.47
3.77
3.95
PSNR
32.00
35.41
34.42
C-CF
MSE Grad
32.00
12.57
17.09
Reconstruction error- We first investigate the impact of the number of keypoints in Table 7 by measuring the Peak Signal to Noise Ratio (PSNR) between the target image and its construction. We vary the number of keypoints among multiples of N, the maximum number of objects in the scene. Increasing the number of keypoints increases reconstruction quality (PSNR) up to a certain point, but results in degradation in forecasting performance. Furthermore, doubling the number of keypoints only slightly improves reconstruction accuracy. This tends to indicate that our additional coefficients are already sufficient to model finer-grained visual details. Table 3 in the main paper measures the impact of the number of keypoints and the presence the additional appearance coefficients on the full pipeline including the dynamic model. Table illustrates the impact of the number of keypoints and the additional appearance coefficient on the reconstruction performance alone. As we can see, the addition of the coefficient consistently improves PSNR for low numbers of keypoints (over dB for N keypoints).
Table PSNR (dB) on the task of reconstructing the target from the source (both randomly sam- pled from Filtered-CoPhy) using coefficients per keypoint. We vary the number of keypoints in our model. Here N is the maximum number of the objects in the scene.
16

---

Published as a conference paper at ICLR 2022
Groend Thuh
IN
Wabcer cottriierr
Wathra coffieiker (2N Kpu) Wateer contioaee
Figure 11: Reconstructions produced by the de-rendering module. Our model correctly marks each object in the scene and achieves satisfactory reconstruction.
The improvement less visible for larger numbers of keypoints, since 3D visual details could be encoded via keypoints position, hence coefficients become less relevant. Visualizations are shown Fig. 11.
N
2N
4N
Coefficients
/
X
PSNR
32.53
34.40
33.97
35.41
34.57
34.92
BT-CF
MSE Grad
41.86
27.24
35.24
21.39
28.06
23.99
PSNR
34.62
37.76
36.94
37.06
37.15 36.98
B-CF
MSE Grad
6,22 3.47
3.77 4.16
4.07
3.95
PSNR
30.65
32.00
33.89
35.41
35.63
34.42
C-CF
MSE Grad
12.78
32.00
20.59
12.57
11.72
17.09
Table 8: Impact of the number of keypoints and the presence of additional appearance coefficient in the de-rendering module for pure image reconstruction (no dynamic model). We report PSNR (dB) and MSE on the image gradient. N is the maximum number of the objects in the scene. The coefficients significantly improve the reconstruction on low number of keypoints. This table related to table 3 in the main paper, which measures this impact the full pipeline.
B.2 NAVIGATING THE LATENT COFFFICIENT MANIFOLD
We evaluate the influence of the additional appearance coefficients on our de-rendering model by navigating its manifold. To do so, we sample a random pair (X Xaargtt) from an experiment in BlocktowerCE and compute the corresponding source features and target keypoints and co- efficients. Then, we vary each component of the target keypoints and coefficients and observe the reconstructed image (fig. 12). We observed that the keypoints accurately control the position of the cube along both spatial axes. The rendering module does infer some hints on 3D shape information from the vertical position of the cube, exploiting shoricut in learning. On the other hand, while not being supervised, the coefficients naturally learn to encode different orientations in space and dis- tance from the camera. Interestingly, form of disentanglement emerges. For example, coefficients n and 2 control rotation around the z-axis, and coefficient n² models rotation around the y-axis. TThe last coefficient represents both the size of the cube and its presence in the image.
17

---

Published as a conference paper at ICLR 2022
Figure 12:
Navigating the manifold the latent coefficient representation. Each line corresponds to variations of one keypoint coordinate coefficient and shows the effect a single cube.
C COMPARISON WITH THE TRANSPORTER BASELINE
C.1 COMPARISON WITH OUR DE-RENDERING MODEL
As described in section 4. the Transporter (Kulkarni et al., 2019) is keypoint detection model somewhat close to our de-rendering module. leverages the transport equation to compute a recon- struction vector:
Fsource x (1 sonrce) x (1 - (1-Kneget) + Fiuret X K
target
This equation allows transmit information from the input by two means: the 2D position of the keypoints (Kagge) and the dense visual features of the target (Furge). In comparison, our de- rendering solely relies on the keypoints from the target image and does not require dense vector to be computed on the target to reconstruct the target. This makes the Transporter incomparable with our de-rendering module. We nevertheless compare the performances of the two models in Table 13, and provide visual examples in Fig. 13. Even though the two models are not comparable, as the Transporter uses additional information, our model still outperforms the Transporter for small numbers of keypoints. Interestingly, for higher numbers of keypoints Transporter tends to discover keypoints far from the object. We investigate this behavior in the following section, and show that this is actually critical problem for learning causal reasoning on the discovered keypoints.
C.2 ANALYSIS TRANSPORTER BEHAVIOR
The original version of the V-CDN model (Li et al., 2020) is based on Transporter (Kulkarni et al., 2019). We have already highlighted the fact that this model is not comparable with our task, as requires not only the target keypoints Karget but also a dense feature map Furget whose dynamics can hardly be learned due to its high dimensionality. More precisely, the transport equation (Eq. 9) allows to pass information from the target by two means: the 2D position of the keypoints (Kunget) and the dense feature map of the target (Furge)) The number keypoints therefore becomes a highly sensible parameter, the transporter can decide to preferably transfer information through the target features rather than through the keypoint locations. When the number of keypoints is low, they act bottleneck, and the model has carefully discover them to reconstruct the image.
18

---

Published as a conference paper at ICLR 2022
Ours
Transporter (not comparable)
# Keypoints
8
16
8
16
BT-CF
34.40
35.41
34.92
34.10 10
34.88
39.20
B-CF
37.76
37.06
36,98
34.75
34.78
35.13
C-CF
35.41
34.42
35.98
32.66
33.39
34.47
Table 9: PSNR (dB) on the task of reconstructing target from the source (both randomly sampled from Filtered-CoPhy using 5 coefficients per keypoint. We vary the number of keypoints both our model and the Transporter. Note that Transporter uses target features to reconstruct the image, hence not comparable with our model.
Grond Trulh
Oun (4k)
Ours (8A)
Oun (166)
Lasenal (43)
Trarsponerr (4k)
Thunqeetar (8k)
Transpotic (16k)
Transponer (4k)
Transpecter (8k)
Transponer (16k)
Gound Truth
Oux (4k)
Oues (8k)
Ouns (16k)
Lazena
Transpanier (4k)
Traqueiirr (8k)
Transpatic (16k)
Figure 13: Example of reconstructed image by our-derendering module and the Transporter.
On the other hand, when we increase the number of keypoints, Transporter stops tracking objects in the scene and transfers visual information through the dense feature map, making the predicted keypoints unnecessary for image reconstruction, and therefore not representative of the dynamics.
To illustrate our hypothesis, we set up the following experiment. Starting from trained Transporter model, we fixed the source image to be Xo (the first frame from the trajectory) during the evaluation step. Then, we compute features and keypoints on the target frame X, regularly sampled in time. We reconstruct the target image using the transport equation, but without updating the target keypoints.
Practically, this consists in computing V Vangee with Eq. (9) substituting Kurget for K Kannce-
Results are shown in Fig. 14. There is no dynamic forecasting involved this figure, and the Trans- porter we used was trained in a regular way, only change the transport equation evaluation time. Even though the keypoint positions have been fixed, the Transporter manages to reconstruct significant part of the images, which indicates that part of the dynamics has been encoded in the dense feature map.
In contrast, this issue does not arise from de de-rendering module, since decoder solely relies on the target keypoints to reconstruct the image. Note that this absolutely not contradictory with the claim in Li et al. (2020), since they do not evaluate V-CDN in pixel space. A rational choice of the number keypoints leads to satisfactory performance, allowing V-CDN to accurately forecast the trajectory keypoints space, and retrieve the hidden confounders their dataset.
C.3 TEMPORAL INCONSISTENCY ISSUES
Increasing the number of keypoints of the Transporter may lead to temporal inconsistency during the long-range reconstruction. For example, keypoint that tracks the edge of cube in the first
19

---

Published as a conference paper at ICLR 2022
1-0
1- 15
t- 22
t- 30
Livaad
Ineeppate
Figure 14: We evaluate the Transporter with a varying number of keypoints to reconstruct images regularly sampled trajectory while having the target keypoints fixed. Even if the keypoints are not moving, the Transporter still manages to reconstruct significant part of the image, which indicates that the keypoints are not fully responsible for encoding dynamics of the scene.
frame may target face of this same cube in the future, since dynamics does not intervene in the keypoint discovery process.
Our de-rendering directly addresses this through the usage of additional appearance coefficients, which allows us to limit the number of keypoints the number of objects in the scene, effectively alleviating the consistency issue. Fig. 15 illustrates this phenomenon by plotting the discovered keypoint locations forward in time, well the 2D location of the center of mass of each object. Note that the Transporter suffers from the temporal inconsistency issue with numbers of keypoints low 4 (green cube). In contrast, model manages solve the problem and accurately tracks the centers mass, even though they were never supervised.
D DETAILS OF MODEL ARCHITECTURES
D.1 DE-RENDERING MODULE
call "block" 2D convolutional layer followed by 2D batch norm layer and ReLU activation. The exact architecture of each part of the encoder is described in Table 10a. The decoder hyper- parameters are described Table 10b.
Dense feature map estimator FF We compute the feature vector from Xsoure by applying volutional network F on the output of the common CNN of the encoder. This produces the source feature vector F Fource of shape (batch, 16, 28).
The convolutional network K. outputs a set of 2D heatmaps of shape (batch, K, 28, 28), where K is the desired number of keypoints. We apply spatial softmax function on the two last dimensions, then we extract a pair coordinates on each heatmap by looking for the location of the maximum, which gives us Kaagett of shape (batch, K,2).
Keypoin detector
20

---

Published as a conference paper at ICLR 2022
[=0
t=30
[=60
[=70
[=50
1=90
Transporter (4lggts)
Transporter (8kpts)
Ours (4kgits)
(Skpts)
Figure 15: Temporal inconsistency in long-range reconstruction. We show the keypoints discovered on images taken from different time steps (black dots). We also compute the 2D location of the center mass each object in the scene (white dots). Our de-rendering module accurately tracks the centers of mass, which have never been supervised.
Coefficient estimator C We obtain the coefficient by applying a third convolutional network C. to the output of the common encoder CNN, which again results in set of 2D vectors of shape (batch, K, 28, 28). These vectors are flattened channel-wise and provide tensor of shape (batch, K, 28 28)) fed
MIP (see Table 10a for the exact architecture) that estimates the coefficients Carget of shape (batch, K, C 1).
Gaussian mapping G The keypoint vector Karret is mapped 2D vector through Gaussian mapping process:
((k) () = exp (- (- k,) + (y kg))
(10)
o²
where g(k) E 28 the Gaussian mapping of the keypoint k = [k, ky). We deform these Gaussian mappings by applying convolutions with filters H; controlled by the coefficients of.
The filters from H are 5 x 5 kernels that elongate the Gaussian in specific direction. Practically. we obtain the filter H; by drawing line crossing the center of the kernel and with slope angle of where C is the number of coefficients. We then apply 2D convolution:
G c of (9(k)) H.).
(11)
Note that we also compute supplementary coefficient a C+1 used as gate on keypoints. By set- ting this coefficient to zero, the de-rendering module can de-activate keypoint (which is redundant with deactivating the full set of coefficients for this keypoint).
Refiner R To reconstruct the target image, channel-wise stack feature vectors from the source with the constructed filters and feed them to the decoder CNN (Table 10b).
We trained the de-rendering module on pairs of images (X Xtarget) randomly sampled from sequences D. For a given sequence D, we take T 25 first frames of the trajectory as a source used target. For
(where T is the number of frames in the video). The last 25 frames evaluation, we take the thth frame as the source, and the 50th frame as the target. We use Adam optimizer with learning rate of 10-3 71 10 and 72 10-1 minimize equation 4.
21

---

Published as a conference paper at ICLR 2022
CNN
Module in ch.
out ch.
kernel
stride
pad.
3
32
7
1
3
32
32
3
1
Block
64
2
64
1
R.O
64
128
2
in
out ch.
kernel stride pad.
FO)
16+K x
128
3
Block
128
16
Block
128
128
3
128
64
3
XC()
4
LpSampiingBllinea2222)
Block
128
5
64
64
3
2onv2d
K
Block
6
64
32
3
Softplus
7
CO
8
32
32
3
Block
Block
128
K
9
32
32
7
Flatten
10
Conv2d
32
3
Module
in
out
TanH
Lincar+ReLU
784
2048
Decoder architecture
Linear+Rel.U
2048
1024
Lincar+ReLU
1024
512
Linear+Rel.U
512
C
Sigmoid
(a) Encoder architecture
Table 10: Architectural details of the de-rendering module.
D.2 CoDY
We describe the architectural choices made in CoDy. Let
s(t) = [k c
CK C+I
k; c cx C1+1
KK=L.KK
(12)
be the state representation of an image X,, composed of the K keypoints 2D coordinates with their C' + 1 coefficients. The time derivative of each component of the state is computed via an implicit Euler derivation scheme k(t) k(t) k(t 1). We subscript notation to distinguish the keypoints from AB and CD.
CF estimator The latent representation of the confounders is discovered from AB The graph neural network from this module implements the message passing function f() and the aggregation function 9() (see equation 5) by MLP with 3 hidden layers of 64 neurons and ReLU activation unit. The resulting nodes embeddings AB (t) = Gs (t)) belong to R228 We then apply gated recurrent unit with 2 layers and a hidden vector of size 32 to each node h hB) (shar- ing parameters between nodes). The last hidden vector is used as the latent representation of the confounders uk.
State encoder-decoderr - The state encoder is modeled as G.V where the message passing function and the aggregation function are MLPs with one hidden layer of 32 units. The encoded state o CD) lies in 256 We perform dynamical prediction in this space, and then project back the forecasting in the keypoint space using a decoder. The decoder ((r(() first applies a shared GRU with one layer and hidden vector size of 256 to each keypoint o (t), followed by graph neural network with the same structure as the state encoder.
Dynamic system Our dynamic system forecasts the future state @(t + 1) from the current esti- mation (t) and the confounders u [uq ux]. J. It first applies a graph neural network to the concatenated vector (t(t) u]. The message passing function and the aggregation function MLPs with hidden layers of 64 neurons and ReLU activation function. The resulting nodes em- beddings (t)) belong 661 and are fed to GRU sharing weights among each nodes with
22

---

Published as a conference paper at ICLR 2022
2 layers and hidden vector of size 64. This GRU updates the hidden vector v (t) [v1 .. vk], that is then used to compute a displacement vector with a linear transformation:
Gz + 1) ox(() + (v)(() b.
(13)
CoDy trained using Adam to minimize equation (learning rate 10"4, 73 1). We train each CoDy instance by providing it with fixed keypoints states sAB(() and the initial condition s cCD (l 0) computed by our trained de-rendering module. CoDy first computes the latent con- founder representation ug. and then projects the initial condition into the latent dynamic space (t = 0) E(s CP (t = 0)). We apply the dynamic model multiple times in order to recursively forecast T time steps from CD. We then apply the decoder ( compute the trajectory in the keypoint space.
E ADDITIONAL QUANTITATIVE EVALUATION
E. MULTI-OBIECT TRACKING METRICS
When the number of keypoints matches the number of objects in the scene, the keypoint detector naturally and in an unsupervised manner places keypoints near the center of mass of each object (see Figure 15). Leveraging this emerged property, we provide additional empirical demonstration of the accuracy of our model by computing classical Multi-Objec Tracking (MOT) metrics. In particular, we computed the Multi-Object Tracking Precision (MOTP) and the Multi Julti-Object Tracking (MOTA) as described in Bernardin Stiefelhagen.
MOTA requires to compute the number of missed objects (i.e. not tracked by keypoints) and the number of false positives (i.e. keypoints that do not represent an actual object). MOTA takes values in [-1, 1]. where represents perfect tracking:
MOTA = 1 Ee m. + +.+
(14)
Σ.9:
where m, is the number of missed objects at time t, fr the number of false positives at time f, is the number of swaps at time t, and is the number of objects at time t.
MOTP a measurement of the distance between the keypoints and the ground-truth cen- ters of mass conditioned on the pairing process:
Eueh di
MOTP
(15)
Σ.a
where e is the number of accurately tracked objects at time t and d; is the distance between the keypoint and the center of mass of the ith association [keypoints+center of mass).
Note that these metrics are related: low MOTP indicates that the tracked objects are tracked pre- cisely, and low MOTA indicates that many objects are missed, Thus, to be efficient, model needs to achieve both low MOTP and high MOTA.
We also reported the performances of CoPhyNet (Baradel et al., 2020) that predicts counterfactual outcomes in euclidian space using the ground-truth 3D space. As it uses GT object positions during training, it is not comparable and should be considered as a soft upper bound of our method. We present our results in Table 11. This confirms the superiority of our method over UV-CDN key- point space. The upper bound CoPhyNet takes advantage of the non-ambiguous 3D representation modeled by the ground-truth state of the object of the scene.
Our method also outperforms CoPhyNet on the balls task, probably due to two phenomena. First, ballsCa is the only 2D task FilteredCoPhy. Thus, CoPhyNet does not have an advantage of using ground-truth truth 3D positions. Second, the state-encoder in CoDy projects the 2D position of each sphere a space where the dynamics is easier to learn, probably by breaking the non-linenn non-linearity of collisions.
23

---

Published as a conference paper at ICLR 2022
Ours
UV-CDN
CoPhyNet (not comparable)
MOTA
0.46
0.16
0.44
BT-CF
MOTP
3.34
4.51
0..2
MOTA
-0.07
-0.73
-0.16
B-CF
MOTP
4.64
5.83
5.10
MOTA
-0.14
-0.19
0.21
C-CF
MOTP
6.35
6.35
4.37
Table 11: MOT metrics for different methods. While not comparable, report the CoPhyNet performance as a soft upper bound. Our method and UV-CDN use one keypoint per object. MOTA higher is better; MOTP lower is better;
BlocktowrrC
PSNR: (4B)
lives awtrage
PSWR (68)
d.
2980 (18)
Movarmments Par ti awitrag
(68)
Per liies amorgge
Mour
Dispbrement amplitude
Divaperation type
Figure 16: Effect of the do-operation on the quality of the forecasted video. (left) our method generalizes well wide range of "Move" operation amplitudes. (right) We observe difference of 3dB in favor of the Move do-operation, which is unsurprising, as it is the least disturbing intervention
E.2 IMPACT OF THE DO-OPERATIONS
We also measure the impact of the do-operation types on the video forecasting. Fig. 16 (left) is obtained by computing PSNR for each example of the training set and reporting the result on 2D graph, depending on the amplitude of the displacement that characterizes the do-operation. We applied the same method to obtain Fig. 16 (right) that focuses the type of do-operation, that is moving, removing or rotating an object. These figures are computed using the 2N keypoints models.
Our method generalizes well across different do-operations, including both the type of the operation, and the amplitude. A key to this success the careful design of the dataset (balanced with respect to the types of do-operations). and reasonable representation (our of keypoints and coefficients) able to detect and model each do-operation from images.
F EXPERIMENTS ON REAL-WORLD DATA
Our contributions are focused on the discovery of causality in physics through counterfactual reason- ing. We designed our model in order solve the benchmark and provided empirical evidence that our method is well suited for modeling rigid-body body physics and counterfactual reasoning. The following section aims demonstrate that our approach can also be extended to real- real-world dataset. We provide qualitative results obtained on a derivative of Blocktowers owerCF using real cubes tower (Lerer al., 2016).
24

---

Published as a conference paper at ICLR 2022
Frasx
Foone
Fore
Pore
Rowre
Foree
Paune
Pram
Figure 17:
We evaluate our method on real-worke dataset Blocktower IRL. After fine-tuning, CoDy manages to accurately forecast future frames from real videos.
We refer to this dataset as Blocktower IRL. It is composed of 516 videos wooden blocks stacked in stable unstable manner. The amount of cubes in tower varies from to 4. We aim predict the dynamics of the tower in pixel space. This is highly related with our task Blocktowerss (which inspired by the seminal work from (Lerer et al., 2016)) with three main differences: (1) the dataset shows real cube towers, (2) the problem is not counterfactual, i.c. every cube has the same mass and (3) the dataset contains only few videos.
To cope with the lack of data, we exploit our pre-trained models on BlocktowerCE and fine-tune on Blocktower IRL. The adaptation of the de-rendering module straightforward: we choose the 4 keypoints-5 coefficients configuration and train the module for image reconstruction after loading the weights from previous training on our simulated task. CoDy, on the other hand, requires careful tuning to preserve the learned regularities from B. blocktower Blocktower IRI. is not counterfactual, we de-activate the confounder estimator and set uą to vectors of ones. We also freeze the weights of the last layers of the MLPs in the dynamic model.
and prevent over-fitting. Since
To the best of our knowledge, we are the first to use this dataset for video prediction. (Lerer et al., 2016) and Wu et al. (2017) leverage the video for stability prediction but actual trajectory forecasting was not the main objective. To quantitatively evaluate our method, we predict 20 frames in the future from single image sampled in the trajectory. We measured average PSNR of 26.27 dB, which is of the same order magnitude compared to the results obtained in simulation. Figure 17 provides visual example of the output.
25

---

Published as a conference paper at ICLR 2022
G
QUALITATIVE EVALUATION: MORE VISUAL EXAMPLES
More qualitative results produced by our model on different tasks from our datasets are given below.
I=21
m=42
1:61
1=/06
to 121
1=1 44
Grnnl
PlytNee
1-21
6-42
1-63
5-35
1-106
1-127
L-149
Carty GooadTra
OPLCENIR
Figure 18: Qualitative performance on the BlocktowerCF (BT-CF) benchmark.
26

---

Published as a conference paper at ICLR 2022
Groua Tunh
1-21
L-63
Croual Tock
Elvest
Plybii
Figure 19: Qualitative performance on the BallsCF (B-CF) benchmark.
27

---

Published as a conference paper at ICLR 2022
1-10
S-42
Groud Tuck
PleTANA
L-51
5-53
Cooud
Figure 20: Qualitative performance on the CollisionCE (C-CF) benchmark.
28


Reviews:
[{"review_id": "mvSF6FI45ix", "reviewer": "Reviewer_v4mS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written.\n- The proposed encoding-decoding framework and the corresponding two-stage learning process are reasonable.\n- the newly designed representation (i.e, high-dimensional features + 2D keypoints + coefficients) is shown to facilitate unsupervised learning from pixel space.\n- The paper provides good empirical results and extensive ablation studies for the effectiveness of each model component.\n\nWeaknesses: \n- The authors may give more empirical analyses about the impact of using different types of interventions. \n- The proposed method is only compared with two existing approaches, including PhyDNet and a modified version of V-CDN. It would be better if the authors could include more existing approaches into the experimental comparison, for example, the keypoint-based model [Minderer et al., 2019], the VAE-based stochastic model [Ref1], and the ConvLSTM-based deterministic model [Ref2]. \n- All experiments are conducted in a simulated environment based on CoPhy, which is somewhat insufficient compared with previous video prediction literature (though I understand the paper studies a new problem). Since the proposed method does not require the supervision of confounders, I wonder if it can be generalized to real-world datasets such as Human3.6M, KTH, or BAIR robot pushing.\n\n[Ref1] Stochastic Video Generation with a Learned Prior. ICML 2018.\n\n[Ref2] PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs. NIPS 2017.\n\n", "summary_of_the_paper": "This paper studies an interesting problem of counterfactual video prediction, which aims to predict the future frame (D) based on the initial frame (C) and an observed video sequence (AB). AB can be seen as a demonstration that is driven by the same confounders, i.e., the physics parameters such as mass and initial velocities. \n\nThe paper follows the previous models from CoPhyNet for dynamics modeling and confounder estimation, and has two improvements over the work of CoPhy, in my point of view: \n- First, it improves the CoPhy benchmark.\n- Second, it presents a new model based on the representations of high-dimensional features, 2D keypoints, and corresponding coefficients. The new form of representation allows the model to be trained in an unsupervised manner only with the supervision of RGB images, as opposed to the training procedure with the supervision of object positions in CoPhyNet. \n\n", "summary_of_the_review": "This paper explores an interesting problem of learning counterfactual physics from pixels. It presents a solution by extending a previous work, CoPhyNet, with new forms of unsupervised video representations. \n\nOne of my concerns is that it does not provide sufficient empirical comparisons with existing video prediction models other than PhyDNet and the modified V-CDN. Furthermore, I would increase my score if the effectiveness of the method can be validated in a real-world dataset.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Gu6sqRND4A", "reviewer": "Reviewer_tY4H", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n - Introduces and justifies an important new benchmark task for video predictions involving inanimate objects\n - Introduces a sophisticated approach that is well-described and justified\n - Strong empirical evaluation that includes ablation studies and exploration of the learned representations\n - Thorough comparison with a previous approach that sheds light on reasons for the improvement\n - The counterfactual style of evaluation may be used in other domains of AI\n\nWeaknesses\n - The paper would be even better if the diversity of the benchmark dataset could be improved.\n - Perhaps more could be said about the visible failures of the approach.  The network does not seem to succeed at learning the structure of rigid 3d bodies, as we see from the videos where the cubes visibly distort and lose their edges over successive frames.\n\nOther comments\n - Many incomplete sentences in the supplement.  I would be happy to list them if this would be beneficial to the authors; I assume this was due to time limitations.", "summary_of_the_paper": "The paper introduces a testing approach, dataset, and method for counterfactual video predictions, using 3d physics simulation videos.  Importantly, the approach predicts directly from pixel space rather than requiring spoon-fed keypoints.  It employs a counterfactual approach to establish a network's capacity to learn causal relations.  Separating the problem into one of parsing the inputs into keypoints and additional coefficients, inferring object attributes, and sequential prediction from an input frame + object attributes, it proposes an architecture which is based on combining modules for each of these learning tasks (but where keypoints are learned in an unsupervised way).  Besides this key architectural innovation, it adds an inductive bias by applying directional Gaussian filters to the keypoint maps.  The paper checks that the network actually works as intended by empirically examining the effect of changing object coefficients.  Three other ablation analyses, one that combines two of the modules (rather than separating them via stop-grad) and one that removes coefficients, and one comparing the handcrafted filter bank with learnable ones, add confidence to the approach.  In the supplement, the paper also analyzes in detail how the method compares to a previous method (Transporter).  It uses sensible benchmarks including previous methods and common-sense baselines where either the original or counterfactual input is used as a prediction, and does relatively well on this challenging problem, although the video on the website shows that there is still plenty of room for improvement.", "summary_of_the_review": "This paper introduces an ambitious new baseline for counterfactual 3d physics predictions in pixel space.  It introduces a sophisticated method based on separate modules for image parsing, parameter inference, and prediction, that outperforms two previous approaches V-CDN and PhyDNet.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7BDp2k5p6Yl", "reviewer": "Reviewer_sqvy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) Filtered-CoPhy seems to be the next intuitive step from CoPhy from a modeling perspective. It is incremental work and a crucial one to get counterfactual predictions (for physics-based simulations) working in an unsupervised fashion.\n\n2) The new benchmark proposal considering Identifiability and Counterfactuality constraints is systematic and alleviates the issues in the CoPhy benchmark.\n\n3) I acknowledge the code released during the review period by the author(s). I have had a chance to briefly look over it and hope that the authors document the code (with a README and instructions on how to run the code) if the paper is accepted.\n\n\nWeakness:\n\n1) Keypoint based encoder/decoder seems to be a good choice. Have the author(s) tried/given a thought how other object-centric representations such as slot-attention would affect the counterfactual performance?\n\n2) It would make the paper stronger if the CoPhy like baseline is reported i.e one using ground truth object positions. This would give at least an upper bound for the proposed benchmark.\n\n3) It would be beneficial to add a quantitative metric based on the predicted and ground truth location of the center of mass (for both Table 1 and Table 3).\n\n4) Also, since the keypoint points can be tracked, it would add more credibility if tracking metrics such as MOT are added as well. Here the author(s) can assume N=number of objects in the scene (otherwise this metric cannot be computed).\n\n5) During the training stage, can the sequences AB be sent to the CoDy (i.e dynamics model) as well? Since $u_k$ should be the same for AB and CD, this acts as a consistency check.", "summary_of_the_paper": "This work extends CoPhy, in which the author(s) address the problem of predicting counterfactual outcomes of physics-based tasks from pixel space (CoPhy used ground truth object positions). To do this, they learn a keypoint representation of the scene and use them to extract the confounders (such as velocities of objects, masses, etc). The author(s) also propose a new benchmark (built upon CoPhy's benchmark) for counterfactual prediction (after intervening the initial set of objects in the scene) which satisfies the Identifiability and the Counterfactuality constraints of causality.", "summary_of_the_review": "Overall I feel that the paper is a strong contribution towards visual counterfactual reasoning (making the framework unsupervised as opposed to CoPhy) and would strongly urge the author(s) to add the experiments mentioned in the Weaknesses section. Based on the novelty of the paper, I am voting for a weak acceptance of the paper.\n\n=======\nPost-rebuttal decision:\nAfter reading the comments of the authors as well as reviews of the other reviewers, I'm very much satisfied with the additional experiments added in the paper and I believe this paper would be a good contribution towards unsupervised visual counterfactual prediction. Hence, I've increased my score to 8 (Accept).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the Filtered-CoPhy method, an approach for learning counterfactual reasoning of physical processes in pixel space. The approach enables forecasting raw videos over long horizons, without requiring strong supervision, e.g. object positions or scene properties. 

The paper initially received one strong accept, one weak accept, and one weak reject recommendations. The main reviewers' concerns relate to clarifications and consolidations in experiments, including stronger baselines, experiments on real data, or more diversity on the datasets. The rebuttal did a good job in answering reviewers' concerns, especially by providing new experimental results and analysis. Eventually, all reviewers recommended a clear acceptance after authors' feedback. 

The AC's own readings confirmed the reviewers' recommendations. The proposed approach is a meaningful extension of CoPhy for the unsupervised prediction at the pixel level. The proposed approach is solid, clearly described, and overcomes important limitations of previous methods. The dataset is also an important outcome for the community. Causality and counterfactual reasoning are of primary importance for the design of effective and explainable AI prediction models: this paper brings therefore an important contribution to the ICLR community.
Decision: Accept (Oral)