CoPhy: Counterfactual Learning of Physical Dynamics
Authors: Fabien Baradel
Natalia Neverova
Julien Mille
Greg Mori
Christian Wolf
Venue:  (2020)
Keywords: intuitive physics
visual reasoning

Abstract: Understanding causes and effects in mechanical systems is an essential component of reasoning in the physical world. This work poses a new problem of counterfactual learning of object mechanics from visual input.  We develop the CoPhy benchmark to assess the capacity of the state-of-the-art models for causal physical reasoning in a synthetic 3D environment and propose a model for learning the physical dynamics in a counterfactual setting. Having observed a mechanical experiment that involves, for example, a falling tower of blocks, a set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention on its initial conditions, such as displacing one of the objects in the scene. The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model in an end-to-end fashion with no supervision. We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment, which results in significantly more accurate predictions at the level of super human performance.

Published as a conference paper at ICLR 2020
COPHY: COUNTERFACTUAL LEARNING OF PHYSICAL DYNAMICS
Fabien Baradel'
Natalia Neverova
Julien Mille
Greg Mori
Christian Wolf
Université Lyon, INSA Lyon, CNRS, LIRIS, Villeurbanne, France
Faacbooo AI Research, Paris, France
3 Laboratoiir d' Informatique de I'Uviv. de Tours, INSA Centre Val de Loire, Blois, France "Simon Fraser University and Borealis AI, Vancouver, Canada
IInia, Chroma group, CITI Laboratory, Villeurbanne, France
ABSTRACT
Understanding causes and effects in mechanical systems is an essential component of reasoning the physical world. This work poses a new problem of counter- factual learning of object mechanics from visual input. We develop the CoPhy benchmark to assess the capacity of the state-of-the-ar models for causal physical reasoning in synthetic 3D environment and propose model for learning the physical dynamics in a counterfactual setting. Having observed a mechanical experiment that involves, for example, falling tower of blocks, set of bouncing balls or colliding objects, we learn to predict how its outcome is affected by an arbitrary intervention its initial conditions, such displacing one of the ob- jects in the scene. The alternative future is predicted given the altered past and a latent representation of the confounders learned by the model an end-to-end fashion with no supervision of confounders. We compare against feedforward video prediction baselines and show how observing alternative experiences allows the network to capture latent physical properties of the environment, which results in significantly more accurate predictions at the level of super human performance.
INTRODUCTION
Reasoning is an essential ability of intelligent agents that enables them to understand complex rela- tionships between observations, detect affordances, interpret knowledge and beliefs, and to leverage this understanding to anticipate future events and act accordingly. The capacity for observational discovery of causal effects in physical reality and making sense of fundamental physical concepts, such as mass, velocity, friction, etc., may be one of differentiating properties of human intelligence that ensures ability to robustly generalize
scenarios (Martin-Ordas et al., 2008).
One way to express causality is based on the concept of counterfactual reasoning, that deals with a problem containing an if statement, which untrue unrealized. Predicting the effect of the interventions based on the given observations without explicitly observing the effect of the intervention on data a hard task and requires modeling of the causal relationships between the variable on which the intervention is performed and the variable whose alternative future should be predicted (Balke & Pearl, 1994). Using counterfactuals has been shown be way perform reasoning over causal relationships between the variables of low dimensional spaces and has been unexplored direction for high dimensional signals such as videos.
In this work, we develop the Counterfactual Physics benchmark (CoPhy) and propose a framework for causal learning of dynamics in mechanical systems with multiple degrees freedom, as illustrated in Fig. 1. For number of scenarios, such lower of blocks falling, balls bouncing against walls or objects colliding, we are given the starting frame A = Xo and a sequence of following frames B X1.T. where covers the range of 6 sec. The observed sequences B, conditioned on the initial state A, are direct effects of the physical principles (such as inertia, gravity or friction) applied to the closed system, that cause the objects change their positions and 3D poses time.

---

Published as a conference paper at ICLR 2020
Xo X
X
der
de
X,
BallaCF
CollisionCF
BlocktoverCF
We train model for performing counterfactual learning of physical dynamics. Given an observed frame A = Xg and sequence of future frames X1 we ask how the outcome B would have changed if we changed Xo to Xo by performing do-intervention (e.g. changing the initial positions objects in the scene).
Figure I:
The task is formulated as follows: having observed the tuple (A, B), we wish to predict positions and poses of all objects in the scene at t=me if we had changed the initial frame Xg by performing an intervention. The intervention is formalized by the do-operator introduced by Pearl et al. (Pearl, 2009; Pearl & McKenzie, 2018) for dealing with causal inference (Spirtes, 2010). In our case, it implies modification of the variable to C, defined as C do (Xxxx=Xo) Accordingly, for each experiment in the CoPhy benchmark, provide pairs of original sequences Xo.r and their modified counterparts Xo.r sharing the same values of all confounders.
We note the fundamental difference between this problem of counterfactual fitture forecasting and the conventional setup of feedforward future forecasting, like video prediction (Mathieu et al., 2016). The latter involves learning retioottempora regularities and thereby predicting future frames X1... from one or several past frame(s) Xo (the causal chain of this problem shown in Fig. 2a). On the other hand, counterfactual forecasting benefits from additional observations in the form of the original outcome X1.r before the do-operator. This adds a confounder variable U into the causal chain (Fig. 2b), which provides information not observable in frame Xo. For instance, in the case of the CoPhy benchmark, observing the pair (A, B) might give information on the masses, velocities or friction coefficients of the objects the scene, which otherwise cannot be inferred from frame Xo alone. Therefore, predicting the alternative outcome after performing counterfactual intervention then involves using the estimate of the confounder U together with the modified past do)
Overall, we employ the idea of counterfactual intervention in predictive models and argue that counterfactual reasoning an important step towards human-like reasoning and general intelligence. More specifically, key contributions of this work include:
a new task of counterfactual prediction of physical dynamics from high high-dimensional visual input, as way to access capacity of intelligent agents for causal discovery;
a large-seale CoPhy benchmark with three physical scenarios and 300k synthetic experiments including rendered sequences of frames, metadata (object positions, angles, sizes) and values of confounders (masses, frictions, frictiss, gravity). This benchmark was specifically designed in bias-free fashion make the counter-factus reasoning task challenging by optimizing the impact of the confounders on the outcome of the experiment. The dataset is publicly available!.
counterfactual neural model predicting alternative outcome of a physical experiment given an intervention, by estimating the latent representation of the confounders. The model outperforms state-of-the-art solutions implementing feedforward video prediction, successfully generalizes to unseen initial states and does not require supervision on the confounders. We provide extensive ablations on the effects of key design choices and compare results with human performance, that show that the task is hard for humans to solve. The code will be made publicly available.
RELATED WORK
This work is inspired by a significant number of prior studies from several subfields, including visual reasoning, learning intuitive physies and perceived causality.
Project page: http::/projes: .irriieee.iiiiiii. liris.crrs. www.ccrrr....o.ooooo fr/cophy
2

---

Published as a conference paper at ICLR 2020
A: A
A Ar
C: X = No)
initial state
initial state
modiledd initial state (pacitions, 3D poses
(peciiions, poses)
(positious, 3D pusees
confourdee U/ - [w*]]
confornder - (uf)
(cnasses, grarity, frietion)
(masses, gravity, frietion)
outcome
outcome
modifed outcome (ponitions, 3D posts)
(pasitions, 3D poses)
(positious, 3D puses]
B: Xert..r
:
D: Xe-t..
(a) feedforward future forceusting
(b) comnterfactual future forecusting
Figure 2: The difference between conventional video prediction (a) and counterfactual video predic- tion (b). The causal graph of the latter includes a confounder variable, which passes information from the original outcome to the outcome after do-intervention. The initially observed sequence (A, B) (on the left) and the counterfactual sequence after the do-intervention (on the right).
Visual reasoning. The most recent works on visual reasoning approach the problem in the setting of visual question answering (Hu et al., 2017; Hudson & Manning, 2018; Johnson et al., 2017: Mao et al., 2019; Perez et al., 2018; Santoro et al., 2017), embodied AI (Wijmans et al., 2019), well as learning intuitive physics (Lerer et al., 2016; Riochet et al., 2018). (Santoro et al., 2017) introduced Relation Networks (RN), fulle-lifferenttable trainable layer for reasoning in deep networks. Following the same trend, (Baradel et al., 2018) estimate object relations from semantically well defined entities using instance segmentation predictions for video understanding. (Santoro et al., 2018) build challenging dataset for solving the problem of abstract reasoning on the visual domain with some tasks such as interpolation or extrapolation. External memory (Graves al., 2016; Jaeger, 2016) extends known recurrent neural mechanisms by decoupling the size of a representation from the controller capacity and introduces the separation between long-term and short-term reasoning. (Reed et al., 2015) propose to learn analogies in a fully supervised way. Our work builds upon this literature and extends the idea of visual reasoning to the counterfactual setting.
Intuitive physics. Fundamental studies on cognitive psychology have shown that humans perform poorly when asked to reason about expected outcomes of a physics based event, demonstrating striking deviations from Newtonian physics in their intuitions (McCloskey & Kohl, 1983; McClooskey et al., 1980; 1983; Kubricht et al., 2017). The questions of approximating these mechanisms, learning from noisy observed and non-observed physical quantities (such sizes velocities vs masses or gravity), as well as justifying importance of explicit physical concepts vs cognitive constructs in intelligent agents have been raised and explored in recent works on deep learning (Wu et al., 2015). (Lerer et al., 2016; Groth et al., 2018) follow this direction by training networks to predict stability of block towers. (Ye et al., 2018) build an interpretable intuitive physical model from visual signals using full supervision on the physical properties of each object. On similar tasks, (Wu et al., 2017) propose to learn physics by interpreting and reconstructing the visual information stream leading to inverting physics or graphical engine. (Zheng et al., 2018) propose to solve this task by first extracting visual perception of the world state and then predict the future. (Battaglia et al., 2016) introduce a fully dillerriffeee network physics engine called Interaction Network (IN), which learns to predict physical systems such as gravitational ones, rigid body dynamics, and mass-spring systems. Similarly, (van Steenkiste et al., 2018) discover objects and their interactions in unsupervised manner from virtual environment. In (Veličković et al., 2018), 8), attention and relational modules are combined on a graph structure. Recent approaches (Chang al., 2017; Janner et al., 2019; Battaglia et al., 2018) based on Graph Convolution Networks (Kipf& & Welling, 2017) have shown promising results on learning physics but are restricted to setups where physical properties need to be fully observable, which is not the case of our approach. The most similar to ours is work by (Ehrhardt et al., 2019) on unsupervised learning of intuitive physics from unpaired past experiences.
Other physics benchmarks and simulators. The main objective for the creation of our benchmark (a) to focus specifically on evaluating capabilities of state of the art models for performing coun- terfactual reasoning, (b) to be unbiased in terms of distributions of parameters to be estimated and balanced with respect to possible outcomes, and (c) to have sufficient variety in terms of scenarios and latent physical characteristics of the scene that are not visually observed and therefore can act as confounders. To the best of our knowledge, none of existing intuitive physics benchmarks have these properties. IntPhys (Riochet et al., 2018) focuses on a high level task of estimating physical plausibility in a black box fashion and modeling out of distribution events at test time. CATER (Girdhar & Ramanan, 2019) introduces video dataset requiring spatiotemporal understanding in
3

---

Published as a conference paper at ICLR 2020
order to solve the tasks such action recognition, compositional action recognition, and adversarial target tracking. Phyre (Bakhtin et al., 2019) is an environment for solving physics based puzzles, where achieving sample efficiency may implicitly require counterfactual reasoning, but this com- ponent is not explicitly evaluated, construction of parallel data with several alternative outcomes is not straightforward, and the trivial baseline performance levels are not easy estimate. Adapting these benchmarks to counterfactual reasoning would require significant refactoring and changing the logic of the data sampling. CLEVRER (Yi et al., 2019) is diagnostic video dataset for systematic evaluation of models on wide range of reasoning tasks including counterfactual questions. They cast the task as a classification problem where the model has to choose between a set of possible answers, whereas our benchmark requires the predicting the dynamics of each object.
Perceptual causality. In the ML. community, causal reasoning gained mainstream attention relatively recently (Lopez-Paz et al., 2017; Lopez-Paz. & Oquab, 2017; Kocaoglu et al., 2018; Rojas-Carulla et al., 2018; Mooij et al., 2016; Schölkopf et al., 2012), due to limitations of statistical learning becoming increasingly apparent (Pearl, 2018; Lake et al., 2017). The concept of perceived causality has been however explored in cognitive psychology (Michotte, 1963), where human subjects have be shown to consistently report causal impressions not aligned with underlying physical principles of the events (Gerstenberg et al., 2015; Kubricht et al., 2017). Exploiting the colliding objects scenario as standard testbed for these studies led discovery of number of cognitive biases, e.g. Motor Object Bias (i.e. false perceived association of object's velocity with its mass).
In this work, we bring the domains of visual reasoning, intuitive physics and perceived causality together in single framework to tackle the new problem of counterfactual learning of physical dynamics. Following prior literature (Battaglia et al., 2013), we also compare counterfactual learning with human performance and expect that, similarly to learning intuitive vs Newtonian physics, modeling perceived vs true causality will get more attention from the ML community in the future.
COPHY: COUNTERFACTUAL PHYSICS BENCHMARK SUITE
In this paper we investigate visual reasoning problems involving a set of K physical objects and their interactions, while considering specific setting of learning counterfactual prediction with the objective of estimating objects' alternative 3D positions from images after do-intervention. We introduce the Counterfactual Physics benchmark suite (CoPhy) for counterfactual reasoning of physical dynamics from raw visual input. It is composed of three tasks based on three physical scenarios: Blocktowerde
defined similarly existing state-of- the-art environments for learning intuitive physics: Shape Stack (Groth et al., 2018), Bouncing balls environment (Chang et al., 2017) and Collision (Ye et al., 2018) respectively. This was done to ensure natural continuity between the prior art in the field and the proposed counterfactual formulation.
Ball 1fFF and Coolissione
Each scenario includes training and test samples, that we call experiments. Each experiment is represented by two sequences of T synthetic RGB images (covering the time span of sec at 5 fps):
an observed sequence x-(Xo..... X²} demonstrates evolution of the dynamic system under the influence of laws of physics (gravity, friction, etc.), from its initial state Xy to its final state X,. For simplicity, we denote A the initial state Xa and B the observed outcome Xo...., X;
counterfactual sequence X {X....... X, J. where Xo (C) corresponds to the initial state Xo after the do-intervention, and X1...., X, (D) correspond to the counterfactual outcome.
A do-intervention is a visually observable change introduced to the initial physical setup zn (such as, for instance, object displacement or removal).
Finally, the physical world in each experiment is parameterized by a set of visually unobservable quantities, or confounders (such as object masses, friction coefficients, direction and magnitude of gravitational forces), that cannot be uniquely estimated from a single time step. Our dataset provides ground truth values of all confounders for evaluation purposes. However, we do not assume access to this information during training or inference, and do not encourage it.
Each of the three scenarios in the CoPhy benchmark is defined as follows (see Fig. for illustrations).
BlocktowerCF - Each experiment involves K-3 or K-4 stacked cubes, which are initially at resting (but potentially unstable) positions. We define three different confounder variables:
4

---

Published as a conference paper at ICLR 2020
musses (K-4)
frictson couficients (K-4)
maseo (K-5)
bicsion cocficions (K-1)
grwily components Sta (K=1)
K-1--9-2
13-39-43
13-05-45
12-16
(400
0-1-1-1
25-11-11-1
00-1-000
15.
[400-05
10.1
10-10-11-3 13-11
13-18-18-08
35. 35 85
13-5-13-13
unstable coufiguration
ficetion corffricuts: top aid botton
gawily composmtia:
stable exnfiguration
9: %
02. 10-11-10--00 10-14
20K
Figure 3: Stability distribution for each confounder variable for heights K-3 and K=4 of the Bl BlockTowerrt task. Masses, friction cooefficients: 2 configurations per block, 2 total; gravity: 3 configurations for each axis fzz, y 9 total.
masses, mE (1, 10} and friction coefficients, μf (E,0.5, 1}, for each block, as well as gravity components in X and Y direction, E The do-interventions include block displacement or removal. This set contains 146k sample experiments corresponding to 73k different geometric block configurations.
BallsCF
Experiments show K bouncing balls (K=2..6). Each ball has an initial random velocity. The confounder variables are the masses, m 10), and the friction coefficients, (0,5, 1]. of each ball. There are two do-operators: block displacement or removal. There are in total 100k experiments corresponding to 50k different initial geometric configurations.
CollisionCF This set is about moving objects colliding with static objects (balls or cylin- ders). The confounder variables are the masses, me 10), and the friction coefficients, (0.5, 1}. of each object. The do-interventions are limited object displacement. This scenario includes 40k experiments with 20k unique geometric object configurations.
Given this data, the problem can be formalized as follows. During training, we are given the quadruplets of visual observations A, B, C, D (through sequences X and X, including GT object positions for supervision), but do not not have access the values of the confounders. During testing, the objective is to reason on new visual data unobserved at training time and to predict the counterfactual outcome D, having observed the first sequence (A, B) and the modified initial state C' after the do-intervention, which known.
The CoPhy benchmark is by construction balanced and bias free w.r.t. (1) global statistics of all confounder values within each scenario, (2) distribution of possible outcomes of each experiment the whole set of possible confounder values (for a given do-intervention). We make that the data does not degenerate to simple regularities which are solvable by conventional methods predicting the future from the past. In particular, for each experimental setup, we enforce existence of least two different confounder configurations resulting in significantly different object trajectories. This guarantees that estimating the confounder variable necessary for visual reasoning on this dataset.
More specifically, we ensure that for each experiment the set of possible counterfactual outcomes is balanced w.r.t. (1) tower stability for Blockt ower owe and (2) distribution of object trajectories for Bal 1CF and Col l iisiionn onCF. As a result, the B Blocktowe rCF set, for example, has 50 + 5% of stable and unstable counterfactual configurations. The exact distribution of stable/unstable examples for each confounder in this scenario is shown in Fig. 3.
All images for this benchmark have been rendered into the visual space (RGB, depth and instance segmentation) at resolution of 448 448 px with PyBullet (only RGB images are used in this work). We ensure diversity in visual appearance between experiments by rendering the pairs of sequences over set of randomized backgrounds. The ground truth physical properties of each object (3D pose, 4D quaternion angles, velocities) are sampled at a higher frame rate (20 fps) and also stored. The training/ / validation / test split is defined 0.7 0.2 0.1 for each of the three scenarios.
5

---

Published as a conference paper at ICLR 2020
GUN
U - (1*), latent representation of the confounders (masses, gravity, friction coeflicients)
GCN
GCN
de-rendering
GCN
GUN
0000 Plessi Sestrrive
updiceed WILL contest
voual Enalanes
hi-dra statee
coarnicantiin
Eocaatees siatie
updinte
Figure 4: Our model learns counterfactual reasoning in a weakly supervised way: while we supervise the do-operator, we do not supervise the confounder variables (masses, frictions, gravity). Input images of the original past (A) and the original outcome (B) are de-rendered into latent representations which are converted into fully-connected attributed graphs. Graph Network updates node features to augment them with contextual information, which is integrated temporally with set of RNNs, one for each object, running over time. The last hidden RNN state taken as an estimate of the confounder U. A second set of GCN+RNN predicts residual object positions (D) using the modified past (C) and the confounder representation U. For clarity we draw arrows for the red object only. Not shown: stability prediction and gating.
COUNTERFACTUAL LEARNING OF PHYSICS
The task as described in Section requires reasoning from visual inputs. We propose a single neural model which can be trained end-to-end, shown in Fig. 4. We address this problem by adding strong inductive biases to a deep neural network, structuring in a way to favor counterfactual reasoning. More precisely, we add structure for (i) estimation of physical properties from images, (ii) modelling interactions between objects through graph convolutions (GCN), (iii) estimating latent representations of the confounder variables, and (iv) exploiting these representations for predictions of the output object positions. At this point we would like to stress again, that the representation of the confounders U latent and discovered from data without supervision.
We
4.1 UNSUPERVISED ESTIMATION OF THE CONFOUNDERS
While our method capable of handling raw RGB frames as input, its internal reasoning is done on estimated representations in object-centric viewpoints. We train a convolutional neural network to detect the K objects and their 3D position in the scene, denoted as (=(o"), k=0... )=(o"),k=0... K-1 where of corresponds to the 3D position of object k. The de-rendering module is explained in the appendix.
Predicting the future of given block requires modelling its interactions (through friction and collisions) with the other blocks in the scene, which we do with Graph Convolution Networks (GCN) (Kipf Welling, 2017; Battaglia et al., 2018). The set of K objects the scene represented as graph G=(V,E) where the nodes V are associated to objects {o*}, }. and the object interactions to edges (o+ (o)) E È in the fully-connected graph. Object embeddings {o*) are updated classically and as follows, resulting in new embeddings {ō+}:
k
Σ DiE
f(o
e Eet 8t d'=gloOtet
12
where Pg is the set of neighboring objects of of f() and g.) are non-linear mappings (MLPs), and their inputs are by default concatenated. For simplicity, in what follows we will denote the update of an object o with GCN given graph with set of nodes and embeddings O by ã GCN( O).

---

Published as a conference paper at ICLR 2020
As mentioned above, want to infer latent representation U of the confounding quantities for each object k given the input sequences X1:r (the original past and the original outcome B), without any supervision. This latent representation U is trained end-to-end by optimizing the counterfactual prediction loss. To this end, we pass the updated object states o through a recurrent network to model the temporal evolution of this representation. In particular, run dedicated RNN for each object, each object maintaining its own hidden state ht:
h 6 N/_1)
(2)
where we index objects and states with subscript t indicating time, and is gated recurrent unit (GRU) (gate equations have been omitted for simplicity). The recurrent network parameters are shared over objects k; which results in model which invariant the number of objects present in the set. This allows to use do-operators which change the number of objects in the scene (removal). We set the latent representation of the confounders to be the set U=(u²), where u* & h is the temporally last hidden state of the recurrent network.
4.2 TRAJECTORY PREDICTION GATED BY STARILITY
We predict the counterfactual outcome D, i.c. the 3D positions of all objects of the sequence X1:r. with a recurrent network, which takes into account the confounders U. We cast this problem sequential prediction task, at each time step & predicting the residual position A w.r.t. to position t-1, i.e. the velocity vector. As in the rest of the model, this prediction is obtained object-wise, albeit with explicit modelling of the inter-object relationships through graph network. More precisely,
GCN
{lot w²1)), r = o rf
=
(3)
where rf is the hidden state of the GRU network denoted by vi, and W is the weight matrix of linear output layer. GCN is graph convolutional network as described in eq. (1) and thereafter.
At each moment of time, each object can either remain stationary or move under the influence of external physical forces or by inertia. The first task for the model is therefore detect which objects are moving (i.e. affected by the environment) and then estimate parameters of the motion if it occurs. This is aligned well with the concepts of whether-causation and how-causation defined in the field of perceived causality (Gerstenberg et al., 2015). In our work, the whether-cause is estimated in the form of a binary stability indicator of described below (for each object, updated at each time step) that is then leveraged to gate the object position predictor (how-cause estimator):
oft1 = + ( A
(4)
where (.) is the sigmoid function and À is a sparsifying temperature term.
Counterfactual estimation of stability estimation of object stability st is counterfactual problem, as stability depends on the physical properties, and therefore on the latent confounder representation ug. We combine the confounders U U=(u*) with the past after do-intervention (C), encoded in object states denoted as - at time step t=t. In particular, for each node we concatenate its object features with its confounder representation and we update the resulting object state with graph network to take into account inter-object relationships:
GCN w²]. [oo w'])). s =
(5)
where sf corresponds to the logits of stability of object & at time t and V is the weight matrix of linear layer (for simplifying notations we omit bias here and in the rest of the paper).
Training - The full counterfactual prediction model trained end-to-end in graph space only (i.e.
not including the de-rendering engine) with the following losses:
Leze Coe (st, sk
Σ Emoe (ot, of-r)
where Lee is the binary cross entropy loss between the stability prediction of object le at time and its ground truth value (calculated by thresholding applied to ground truth speed vectors), and Lmse is mean squared error between the predicted positions of and GT positions afte in the 3D space. A detailed description of the overall architecture is given in the Appendix.
7

---

Published as a conference paper at ICLR 2020
Scenurio
Top
Middle
Bottom
Mcan
Train-Test
Copy C Copy
NPE
CoPhyyNt
IN sign.
Human
CF
Non-CFF
108.89
99.98
53. 15
09.65
13.67
13.99
51.54
58.57
33.61
34.15
3-431
0.470
0.365
0.601
0.592
0.298
0.319
0.289
0,282
0.296
Copying
C-
B-+
65.41
92.60
35.85
25.51
18.54
7.36
49.00
52.76
N/A
N/A
3
0.754
0.846
0.524
0.523
0,482
0.467
1+4
0.235
0.852
0.521
0.528
0.453
0.48/
CoPhyNet
Non-CFF
2.39
37.81
4-+44
0.597
0.861
0.480
0.476
0.423
0.464
$7.10
23.26
4,40
28:25
N/A
0,480
0.618
0.342
0.350
0.381
0.297
Table 1:
Comparison with human per- formance in the BlockTowerCP scenario obtained with AMT studies. We report 2D pixel error for each block, as well global mean and variance a (reference resolution 448 x 448) on the test set with 3 blocks.
Table 2: BlocktowerCF: MSE on 3D pose av- erage over time. IN sup. methods in the last col- umn exploit the ground truth confounder quantities as input and thus represent a soft upper bound (are not comparable). 'Tsst confounder configurations not seen during training (50/50 split).
Figure 5: Visual examples of human performance on the ill-posed task of feedforward, i.e. non- counterfactual, dynamic prediction from single image (in the BlockTower scenario). The image shows the initial state C. Small dots correspond to human estimates of the objects' final positions. Larger circles indicate ground truth final positions of each block. We note that this task is ill-posed by construction, as the dynamics of each experiment is defined by physical properties of each block (e.g. masses) which cannot observed from a single image.
5 EXPERIMENTS
Training details. All models were implemented in PyTorch. We used the Adam optimizer (Kingma & Ba, 2015) and learning rate of 0.001. For training the de-rendering pipeline 200k frames were sampled for each of the three scenarios (see the appendix for more details).
Human performance. We empirically measured human performance in the lookk CCF sce- nario with crowdsourcing (Amazon Mechanical Turk/AMT), For this study, we have collected predic- tions from 100 participants, where each subject was given 20 assignments in both non-counterfactual (Fig. 5) and counterfactual (Fig. 6) settings. The human subjects were given 10 sec to click on the final positions of each block in the image C after the tower has fallen (or remained stable). The obtained quantitative results for both settings are reported in Table 1. We compare against copying baselines (i.e. predicting block positions in the frame D by either copying them from C or from B).
In conclusion, observe that humans perform slightly better in the counterfactual setup after having observed the first dynamic sequence (A, B) together with C compared to the classical prediction where only C is shown. This behavior has also been previously observed in experiments on intuitive physics in cognitive psychology (Kubricht et al., 2017) that revealed poor human abilities to extrapolate physical dynamics from single image. Similar human studies have also been conducted in (Battaglia et al., 2013) in more simplistic setup of predicting the direction of falling, where the authors also reported that the task appeared be challenging for human subjects.
The empirical results indicate that the participants decisions may have been however driven by simple inductive biases, e.g. "observed (in)stability in (A, B)" "prrdictt (in)stability (C, D)". The evidence for this approach is demonstrated qualitatively in Fig. 6: the variance in predictions after having observed a stable sequence is decreased (first row), after having observed falling case increased (second row). In all cases, human performance remains inferior w.r.t. the copying baselines.
The last part of Table shows results of our model (denoted by CoPhyNet in the rest of the discussion) after projecting the estimated 3D positions of all objects back into the 2D image space. CoPhyNet significantly outperforms both human subjects and copying basclines.
Performance and comparisons. We evaluate the counterfactual prediction performance of the proposed CoPhyNet model against various basclines (shown in Tables 2-4 separately for each of the three scenarios of the CoPhy benchmark). The evaluated Network Physics Engine (NPE) (Chang et al., 2017) and Interaction Network (IN) (Battaglia et al., 2016), are both non-counterfactual baselines,
8

---

Published as a conference paper at ICLR 2020
Fedioorand:: C
Counterfartual: (A. C D
Feddoowwadi: C-+
Cousterfactual (), C
imagee
Emage A
image B
image
iinagee
imaage A
image
Image
experiinent
confoumderss Pec)
experiment
(po Porht
Visual examples of human performance on the task of counterfactual dynamic prediction (in the BlockTowerCE scenario). Each participant has been shown both (A, B) and C. Small dots correspond to human estimates of the objects' resting positions (outcome after do-intervention). Larger circles indicate the ground truth final positions. The images show state C.
Figure 6:
Fretfforand:
Countrrfartzal: (A, C-
Reelforrndd:
fecttoe (A, B)
lo
tmaggn C
image A
iinage
Image C
imagee
Imagee A
Image R
image C
(
conlignntio
(A)
collggration (l)
Figure 7: Visual examples of the counterfactual predictions produced by CoPhyNet (in the BlocktowerCE scenario). Circles denote GT position and crosses correspond to predictions.
that predict future block coordinates from past coordinates after do-intervention without taking the confounders into account. More details for IN and NPE are given in the appendix A.3. Our method consistently outperforms NPE and IN by large margin in all scenarios. The CoPhyNet model also usually (but not always) outperforms the augmented variants of these methods that include the GT confounder quantities as input (a not comparable setting).
Fig. illustrates several randomly sampled experimental setups and corresponding counterfactual predictions by the CoPhyNet model in the Blocktowerch scenario.
Generalization. We evaluate the ability of the CoPhyNet model to generalize to new physical setups which were not observed in the training data. In Table 2 we show model performance on unseen confounder combinations and on unseen number of blocks in the BlacktowerCh scenario (lines marked with t). Our proposed solution generalizes well under unseen settings compared to other methods. In Table we also demonstrate that our method outperforms the baselines by a large margin unseen numbers of balls in the BallsCF setup. Finally, in the Collis MilissinCCF scenario (Table 4) we train on one type of moving objects and test on another type (spheres vs cylinders). In this case also show that our method is able generalize to the new object types when it has not static-object> before. Our method is able to estimate the
such a combination of <moving- object properties when an object is moving or initially stable.
Impact of the confounder estimate. Our model does not rely on any supervision of the confounders; do, however, explore what effect supervision could have performance, shown in Table (Middle). Adding the supervision increases the performance of the model for K=3 but the difference seems marginal (0.004 for K=3 and 0.020 for K44). Directly feeding the confounder quantities as input leads to better performance, which is expected (but not comparable).
ablated in Table 6 (Left) fully illustrate the impact of each submodule. Estimating the stability once for the whole sequence D decreases the performance by 0.020 for K=3 and 0.018 for K=4 compared predicting the stability per object at each time step. Replacing the GCN by a MIP (i.e. concatenating the object representation) hurts the performance of the overall system by increasing the MSE by 0.286 when tested in the K=4 setting. approach against single-step counterfactual prediction. Non-surprisingly, predicting the future autoregressively predicting the whole sequence once.
Model architecture. All design choices of CoPhyNet
Finally compare
step-by-step fashion turns out to be more effective than
Confounder estimation. After training for predicting the target CF sequences, we evaluate the quality of the learned latent representation. In this experiment, predict the confounder quantities of each object (mass, friction coefficient) from their latent representation by training a simple linear classifier, freezing the weights of the whole network. The obtained results shown in Table 5. A

---

Published as a conference paper at ICLR 2020
Train
Test
[Copy
Copy B
CalPhyNet
sup
Train-Test
[Copy
Copy
IN
NTE
CoPPeyNel
IN
1 1-2 2
7,271
3.267
5,060
1.989
2.307
2.109
nall
4,370
0,668
0.701
0.697
0.173
0.332
6.820
2,865
4.895
4.901
1,990
1.886
sphere ncylinder
4.45
0.715
0.710
6.538
4.821
cylinder supbere
4.571
0.922
0.720
0.699
0.152
0.586
>
6.221
2,568
4.732
4.817
1,958
2.746
4 -
6,045
2,488
4.661
4,668
1,899
2,564
Table 4: CollisionCF: MSE on 3D pose av- erage over time. IN sup. methods in the last col- umn exploit the ground truth confounder quanti- ties as input and thus is not directly comparable (still showing inferior performance).
Table 3: BallsCF: MSE on 2D pose average over time. IN sup. methods in the last column exploit the ground truth confounder quantities as input and thus is not directly comparable.
Table 5:
Ablations on BlockTowerCF: confounder prediction (masses, friction coefficients) from the joint latent representation U. Metric: 4-way classification accuracy: Random=random classification.
Method
13 3
4
Random
25.0
25.0
CoPhyNet
65.7
68.9
Method
3
Feedforward
Counterfactual
Method
3
3-+-1
Static guting
0.305
0.496
Subsct
confounders:
confounders:
Copy C
71.0
69.8
GCN replaced
0.764
input
supervision
Copy B 69.9
68.5
Singlo-stop prodiction
0.295
0.492
K=4
0.248
0.349
0.281
0.285
GCN(C)
71.8
70.1
CoPPyyNet
0.478
K=3
0.410
0.552
0.458
CoPPyy
73.8
Table 6: Ablation study on BlockTowerCF: (Left) Impact of each component of our model (MSE on 3D pose average over time). (Middle) Impact of the confounder estimate (MSE on 3D pose average over time, validation set). Feedforward methods do not estimate the confounder, counterfactual methods do. We compare against soft upper bounds, which use the ground truth confounder as input or supervise its estimation. (Right) Stability prediction (accuracy per block). With the ground truth confounder values as input, graph convolutional networks (GCN(C)) reach performance of 85.4 and 77.3 in the 3 + 3 and 3 + settings respectively (a soft upper bound, not comparable).
prediction is correct if both the mass and the friction coefficients are correctly predicted. Our model outperforms the random baseline by a large margin suggesting that the confounder quantities are correctly encoded into the latent representation of each object during training.
Stability prediction. We studied the performance of the stability estimation module in the Block rCF scenario and compared to several baselines, as shown in Table (Right). Our method predicts stability of each block from the confounder estimate U and the frame C. It outper- forms the baselines estimating stability from a single input or from the sequence (A, B) by a large margin, further indicating the efficiency of the confounder estimation and the complimentarity of this non-visual information w.r.t. the visual observation C.
CONCLUSION
We formulated new task of counterfactual reasoning for learning intuitive physies from images, developed a large-scale benchmark suite and proposed a practical approach for this problem. The task requires to predict alternative outcomes of physical problem given the original past and outcome and an alternative past after do-intervention. Our challenging benchmarks cannot be solved by classical methods predicting by extrapolation, as the alternative future depends on confounder variables, which are unobservable from a single image of the alternative past. We train neural model by supervising the do-operator, but not the confounders. Our experiments show that the CF setting outperforms conventional forecasting, and that the latent representation is related to the GT confounder quantities. We report human performance on this task, show its challenging nature and importance of CF reasoning.
We believe that counterfactual reasoning in high-dimensional spaces is a key component of AI and hope that our task will spawn new rescarch in this area and thus contribute to bridging the gap between causal reasoning and deep learning. We also expect the benchmark to become a testbed in model based RL, which employs predictive models of environment for learning agent behavior. Forward models are classically used in this context, but we conjecture that CF reasoning will contribute to disentangling representations and inferring causal relationships between different factors of variation.
Acknowledgements. This work was funded by grant Deepvision JANR-15-CE22 15-21-30-2,, STPGP. 455. 5). joint French/Canadian call by ANR & NSERC.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Anton Bakhtin, Laurens van der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. In NeurIPS, 2019.
Alexander Balke and Judea Pearl. Counterfactual probabilities: Computational methods, bounds and applications. In UAI, 1994.
Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, 2018.
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation as an engine of physical scene understanding. In PNAS, 2013.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In NIPS, 2016.
P.W. Battaglia, J.B. Hamrick, V. Bapst, A. Sanchez- Gonzalzz, V.F. Zambaldi, M. Malinowski, A. Tacchetti, Raposo, A. Santoro, R. Faulkner, Ç. Gülçehre, F. Song, A.J. Ballard, J. Gilmer, G.E. Dahl, A. Vaswani, K. Allen, C. Nash, V/ Langston, C. Dyer, N. Heess, D. Wierstra, P. Kohli, M. Botvinick, O. Vinyals, Li, and R. Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv preprint wrXiv:1807.092444 2018.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach to learning physical dynamics. In ICLR, 2017.
S. Ehrhardt, A. Monszpart, N. Mitra, and A. Vedaldi. Unsupervised intuitive physics from past experiences. arXiv preprint, arXiv:1905.. 10793, 2019.
Tobias Gerstenberg, Noah D. Goodman, David A. Lagnado, and Joshua A. Tenenbaum. How, whether, why: Causal judgments as counterfactual contrasts. In Annual Conference of the Cognitive Science Society, 2015.
Rohit Girdhar and Deva Ramanan. Cater: diagnostic dataset for compositional actions and temporal reasoning. In ICLR, 2019.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, AdriaPuigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using neural network with dynamic external memory. Nature, 2016.
Oliver Groth, Fabian B. Fuchs, Ingmar Posner, and Andrea Vedaldi. Shapestacks: Learning vision- based physical intuition for generalised object stacking. In ECCV, 2018.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
R. Hu, J. Andreas, Rohrbach R., T. Darrell, and K. Saenko. Learning to reason: End-to-end module networks for visual question answering. ICCV, 2017.
D. A. Hudson and C. D. Manning. Compositional attention networks for machine reasooning. In ICLR, 2018.
Herbert Jaeger. Artificial intelligence: Deep neural reasoning. Nature, 2016.
Michael Janner, Sergey Levine, William T. Freeman, Joshua B. Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning about physical interactions with object-centric models. In ICLR, 2019.
J. Johnson, B. Hariharan, L. van der Maaten, J. Hoffman, L. Fei-Fei, C. Zitnick, and Girshick R. Inferring and executing programs for visual reasoning. ICCV, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICML, 2015.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
11

---

Published as a conference paper at ICLR 2020
Murat Kocaoglu, Christopher Snyder, Alexandros G. Dimakis, and Sriram Vishwanath. CausalGAN: Learning causal implicit generative models with adversarial training. In ICLR, 2018.
James R. Kubricht, Keith J. Holyoak, and Hongjing Lu. Intuitive physics: Current research and controversies. Trends Cognitive Sciences, 2017.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman. Building machines that learn and think like people. In Behavioral and Brain Sciences, 2017.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In ICML, 2016.
David Lopez-Paz. and Maxime Oquab. Revisiting classifier two-sample tests for gan evaluation and causal discovery. In ICLR, 2017.
David Lopez-Paz, Robert Nishihara, Soumith Chintala, Bernhard Scholkopf, and Leon Bottou. Discovering causal signals in images. CVPR, 2017.
J. Mao, C. Gan, P. Kohli, J. Tenenbaum, and J. Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. In ICLR, 2019.
G. Martin-Ordas, J. Call, and F. Colmenares. Tubes, tables and traps: great apes solve twofunctionally equivalent trap tasks but show no evidence of transfer across tasks. Animal cognition, 2008.
Michaël Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016.
M. McClooskey, A. Caramazza, and B. Green. Curvilinear motion in the absence of external forces: Naive beliefs about the motion of objects. Journal of Experimental Psychoology: Learning, Memory, and Cognition, 1980.
M. McClooskey, A. Washburn, and L. Felch. Intuitive physies: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition, 1983.
M. McCloskey and D. Kohl. Naive physics: The curvilinear impetus principle and its role in interactionos with moving objects. Journal of Experimental Psychoology: Learning, Memory, and Cognition, 911::466, 1983.
A. Michotte. The perception of causality. In Basic Books, 1963.
Joris M. Mooij, Jonas Peters, Dominik Janzing, Jakob Zscheischler, and Bernhard Schölkopf. Distin- guishing cause from effect using observational data: Methods and benchmarks. JMLR, 2016.
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2nd edition, 2009.
Judea Pearl. Theoretical impediments to machine learning with seven sparks from the causal revolution. arXiv preprint
2018.
Judea Pearl and Dana McKenzie. The Book of Why: The New Science of Cause and Effect. Basic Books, 2018
E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
Scott E Reed, Yi Zhang, Yuting Zhang, and Honglak Lec. Deep visual mnalogy-making In NIPS, 2015.
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer. Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Intphys: A framework and benchmark for visual intuitive physics reasoning. arXiv preprint, arXiv:1803, 100.006666,, 2018.
M. Rojas-Carulla M. Baroni, and D. Lopez-Paz. Causal discovery using proxy variables. In ICLR Workshop, 2018.
12

---

Published as a conference paper at ICLR 2020
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillierap. A simple neural network module for relational reasoning. In NIPS, 2017.
Adam Santoro, Felix Hill, David G. T. Barrett, Ari S. Morcos, and Timothy Lillicrap. Measuring abstract reasoning in neural networks. In ICML, 2018.
B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij. On causal and anticausal learning. In ICML, 2012.
Peter Spirtes. Introduction to causal inference. JMLR, 2010.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In ICLR, 2018.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Eric Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruy Batra. Embodied question answering in photorealistic environments with point cloud perception. In CVPR, 2019.
Jlajun Wu, liker Yildirim, Joseph J. Lim, Bill Freeman, and Josh Tenenbaum. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In NIPS, 2015.
Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In NIPS, 2017.
Tian Ye. Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In ECCV, 2018
Kexin Yi, Chuang Gan, Yunshu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B. Tenenbaum. Clevrer: Collision events for video representation and reasoning. In ICLR, 2019.
David Zheng, Vinson Luo, Jiajun Wu, and Joshua B. Tenenbaum. Unsupervised learning of latent physical properties using perception-predictio networks. UAI, 2018.
A APPENDIX
A.1 NEURAL DE-RENDERING
We train convolutional neural network (CNN) to detect the K blocks and estimate their positions in 3D, denoted as O=(o*), k= K-1 where ot corresponds to the position of object k. The network is inspired by classical region-based based methods for object detection (He et al., 2017) and takes as an input an RGB image of resolution 224x 224.
We define a double convolution module as a stack ReLu activation, repeated two times. The resulting CNN includes three such modules with 64, 128 and 256 channels respectively, separated by x 2 max pooling, which produces output feature maps of size 256 x 56.
convolutional layer, batch normalization and
We design K different heads by splitting the final feature maps channel-wise into K feature maps and perform double convolution module with output channel. Each head outputs feature map of size 56 which is transformed into vector to regress the object positions.
We first pre-train the de-rendering module alone without the model parts responsible for reasoning in the graph space. particular, we de-render images X randomly sampled from A, B, C and D into its object representations [= and train with the following supervised loss:
K
Ldsrender
Cmse ( ,
(6)
k=1
where kh are the ground truth 3D positions and Line corresponds to the mean square error. The rest of the model is trained end-to-end described in section 4.2, paragraph "Training".
13

---

Published as a conference paper at ICLR 2020
A.2 OTHER ARCHITECTURES
Below are the descriptions of other networks used in our pipeline:
f, gg are implemented as MLPs with 4 and 2 layers respectively, with hidden layers of size 32 and ReLu activations.
6, o are implemented as GRU modules with 2 layers and a hidden state of dimension 32. Confounders (mass and friction coefficients) are predicted with single fully connected layer on top of the "confounder" representation of each object denoted u².
A.3 FEEDFORWARD BASELINES
We compare our approach against two recent feedforward approaches, namely IN (Battaglia et al., 2016) and NPE (Chang al., 2017). Both methods assume that GT object positions are available as input at training and test time, so they directly work on GT positions. They both predict the next position of each object using Graph Convolution Network. IN is modeling object pairwise interaction between all objects in the scene, while NPE is taking into account only neighbouring objects for estimating the object interactions.
A.4 TRAINING FROM ESTIMATED POSITIONS
In Table 7 report the impact of the de-rendering module performance. In particular, we compare performance of our model (CoPhyNet w/o GT, as described in the main part of the paper) with version where we use ground-truth positions (CoPhyNet GT) for training. During training time, GT positions are fed to the main model. For testing, we do, however, use positions estimated by the de-rendering module in both versions.
We can see that training using the GT positions gives slighly better performance than training from estimated positions, which expected.
Copy C Copy
CoPhyNet GT
CoPhyNet wio GT (=ours))
3 3
0,470
0.601
0.294
0.309
0.592
3 3-44
0.754
0.846
0.482
0.504
Table BlocktowerCF: MSE on 3D pose averaged over time. We evaluate different types of training: from the ground-truth positions (GT) or from the estimated positions (w/o GT). t'Tsst confounder configurations not seen during training (50/50 split).
14


Reviews:
[{"review_id": "ryloApppFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: after revision, I have decided to keep the score unchanged.\n\nOriginal comments:\n\nIn this paper, the authors proposed a new method to learn physical dynamics based on counterfactual reasoning. \n\n1. As also summarized by the paper, over recent years, there has been increasing interest in the research community for the study of visual reasoning, intuitive physics and perceptual causality. This work provides an interesting framework that combines all the three domains together to solve the problem of learning physical dynamics. The experimental result also shows promise. \n\n2. This paper also provides a nice work that bridges the gap between the counterfactual reasoning and deep learning community.\n\nWith all of this said, I think overall the paper is an interesting addition to the causal inference and deep learning literature.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkMT-8Rtr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies counterfactual event prediction in physical simulation. The authors proposed a model that leverages object-centric scene representations and graph networks for modeling object interactions. The model also uses a recurrent network to encode and extract the confounder information for counterfactual prediction. In the experiments, the authors compared the proposed method, baselines, and human performance on pose estimation and counterfactual prediction.\n\nI reviewed an earlier version of the paper at another venue. Compared with that, the current manuscript has improved a lot. It's studying an important problem. The model builds upon SOTA techniques such as GCN. Experiments are conducted on multiple physical events with multiple confounders. There are also rich ablation studies. The writing is clear and easy to follow. My recommendation is weak accept.\n\nI think the paper can be improved by adding experiments on real data. The model involves 'de-rendering' which seems not easily generalizable to complex real scenes. Also, while the block tower scenario has been well studied, the discussion on ball and collision scenarios is quite limited. I encourage the authors to include more results on those datasets. The authors should also conduct human studies there, too.\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gVN4rCtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of what the paper claims and contributes\n---\nThis paper presents a new object forecasting task in the setting of \"intuitive physics\" that requires counterfactual reasoning and also presents method to perform this task. The task is to predict the alternative future trajectory of objects in 3D simulation given a video (A_frame,B_frames) of how the objects move in one situation and an alternative single frame (C_frame) that corresponds to modifying object position(s) at the first timestep of the input video. Due to unobserved confounding factors such as mass and friction that are not directly observable in either the video or the intervened frame, successful forecasting requires either implicitly or explicitly estimating these confounders. The motivation for this task is the hypothesis that counterfactual reasoning is a necessary component to forecast in general, unobserved situations. Prior work on explicit causal reasoning mainly concerns lower-dimensional problems, and prior work on high-dimensional \"intuitive physics\" has not evaluated the capability of models to perform counterfactual reasoning.\n\nThe paper proposes a neural network that learns to implicitly predict sufficient statistics of confounders of the situation. Training occurs in two stages. First, it is trained to estimate 3d positions of objects from images given pairs of (image, ground truth 3d positions). Then, it is trained to forecast future unobserved object positions with a graph convolutional network. Inference is performed by estimating the current object positions in all of the frames of the input (A_frame, B_frames), and then using the final representation of the input situation as the representation of confounders that is fed into the alternative-future prediction.\n\nThe experiments find:\n1) Humans are not very good at these specific tasks, as evidenced by a simple position-copying baseline that outperforms humans. Given that the simple copying baseline is so performant, the claim in the abstract of \"super human performance\"  is also true for the copying baseline, and therefore that claim isn't very meaningful and should be removed.\n2) The proposed approach outperforms position-copying and two non-counterfactual approaches that do not reason (explicitly or implicitly) about confounders,.\n3) The approach attempts to show that confounder representations are learned as the main evidence that the model can perform counterfactual reasoning. However, several aspects of this experiment are unclear, making the claim difficult to evaluate.\n\nFinally, although the abstract and the main text say that the task is to perform counterfactual forecasting in high-dimensional scenarios in a unsupervised way, the paper ends up using direct supervision of the object positions. With this knowledge, the problem is not that different than counterfactual forecasting in low-dimensional settings, as the positions of the objects at all frames corresponding to \\mathbf{A}, \\mathbf{B}, and \\mathbf{C} could simply be extracted with this detector. Thus, through this supervision the approach seems to circumvent most of the challenges imposed by high-dimensional forecasting.\n\nEvaluation\n---\n\n>Originality:\nAre the tasks or methods new?\nThe task of counterfactual forecasting from high-dimensional observations is new. The proposed method extends prior work to the new task.\n\nIs the work a novel combination of well-known techniques?\nYes.\n\nIs it clear how this work differs from previous contributions?\nYes.\n\nIs related work adequately cited?\nTo my knowledge, yes.\n\n>Quality:\nIs the submission technically sound?\nYes.\n\nAre claims well supported by theoretical analysis or experimental results?\nSome of the claims are supported, but the evidence for the counterfactual representation recovery claim is currently unclear. Additionally, the high-level motivation of counterfactual forecasting in high-dimensional settings is significantly undermined by the use of low-level supervision information that could feasibly be predicted at test time.\n\nIs this a complete piece of work or work in progress?\nIt seems relatively complete.\n\nAre the authors careful and honest about evaluating both the strengths and weaknesses of their work?\nWeaknesses are not explicitly explored. It would be good to include discussion that illuminates or hypothesizes when and why the method fails, and how it could be made to perform better.\n\n>Clarity:\nIs the submission clearly written?\nThe submission is mostly clear, however there are some significant ambiguities in the experiments.\n\nIs it well organized?\nYes.\n\nDoes it adequately inform the reader?\nMostly -- see ambiguities.\n\n>Significance:\nAre the results important?\nIt's currently unclear whether the results are significant, but it's clear the task is important (only when not supervised by object positions). However, justification with respect to other recent intuitive physics simulators is needed.\n\nAre others (researchers or practitioners) likely to use the ideas or build on them?\nOther researchers are likely to build on the task and data released.\n\nDoes the submission address a difficult task in a better way than previous work?\nN/A (new task)\n\nDoes it advance the state of the art in a demonstrable way?\nThere is no SOTA on this new task.\n\nDoes it provide unique data, unique conclusions about existing data, or a unique theoretical or experimental approach?\nYes, one of the main strengths of the paper is the task and dataset that will be released. This experimental approach could be very useful for future research.\n\nAdditional feedback\n---\nThe experiments to validate the cofounder estimation is unclear. Ambiguities:\n0) In the text, the classification of mass and friction is presented as follows: \"The obtained results are shown in Table 6 (middle)\", yet the results are in Table 5. The impact of confounder supervision is described as follows \"we do, however, explore what effect supervision could have on performance, as shown in Table 5\", yet the results are in Table 6. These typos, when combined with the layout of Tables 5 and 6, makes it very difficult to interpret these results.\n1) What is feedforward method? I can't find any description. Is this the IN or the NPE baseline, or something different? Why aren't both used? Without more details, its unclear that the proposed method actually learns confounder representations better than other methods.\n2) What is the \"random baseline level\"? Is this just chance results from a uniform distribution as a baseline classifier?\n3) What are the details of the classifier that predicts the mass and friction coefficients?\n\nIt is a bit odd to include comparisons to humans on a task on which the humans are outperformed by a very simple baseline. Either downplay the role of these comparisons, or remove these comparisons, or change the task such that humans cannot be outperformed by a simple baseline.\n\nMore details about the NPE and IN baselines are missing. Specifically, do they also leverage access to the ground-truth positions at training time? If so, how do they do this? This discussion is important for the paper to be self-contained. \n\nPage 9 \"The CophyNet model also consistently outperforms the augmented variants...\" the outperformance isn't present in all cases (i.e. not in 4->2 or 4->3); change to \"usually\".\n\nIt's not clear if there is any stochasticity in the dynamics of the created data (which would be relevant to modeling and evaluation).\n\nIt wasn't clear until page 7 that the training leveraged GT positions in 3d space. The training objective (equation after Eq (5)) has an undefined o^*, which the reader must infer corresponds to GT, or read appendix A.1, to understand. The description of the method was introduced in terms of A, B, C, and D which are images or videos. To me, the fact that GT 3D positions are used seems like it was \"buried\" later in the paper, rather than described up-front alongside the introduction of A, B, C, and D.\n\nThe meaning of the colors, if any, in Fig 2 is unclear. If they are meant to illustrate observed information, perhaps grayscale shading could be used instead, as is standard in PGMs.\n\nThe latent representation in Fig 4 appear to emerge just from the red object's RNN, rather than all of the objects RNNs.\n\nTables 1 and 2 appear very far from their description in the text.\n\nMore discussion is needed that relates the tasks in to the tasks in Riochet2018 and Bakhtin2019. It's not clear why a new benchmark is needed without this discussion. Specifically, could the proposed method and evaluation be applied to these benchmark simulators? If not, why not? The answers to these questions seem to be the main motivation for the proposed tasks.\n\nThe simplest baseline would be to simply stack (A,B,C) as contextual input to a learned regressor of the future positions in D. Another simple baseline is to use the learned position estimator to predict all of the positions in (A,B,C) and use these positions as input to a learned regressor.\n\nUse two backtick characters instead of the double-quote character for starting forward quotations.\n\nRecommendation\n---\nTaken together, the novelty of the task and dataset, the discrepancy between problem motivation (high-dimensional counterfactual reasoning vs. uses low-dimensional ground-truth), and the partial clarity of the experimental results, leads me to conclude that although the proposed task and data are quite interesting and promising, the paper needs significant work to clarify its motivation and justification of its claims. I would rate the paper borderline, however it appears I'm forced to discretize to WA or WR. I choose WA because I am optimistic that the authors could address my doubts.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are unanimous in their opinion that this paper offers a novel approach to learning naïve physics.  I concur.
Decision: Accept (Spotlight)