Physics-as-Inverse-Graphics: Unsupervised Physical Parameter Estimation from Video
Authors: Miguel Jaques
Michael Burke
Timothy Hospedales
Venue:  (2020)

Abstract: We propose a model that is able to perform physical parameter estimation of systems from video, where the differential equations governing the scene dynamics are known, but labeled states or objects are not available. Existing physical scene understanding methods require either object state supervision, or do not integrate with differentiable physics to learn interpretable system parameters and states. We address this problem through a \textit{physics-as-inverse-graphics} approach that brings together vision-as-inverse-graphics and differentiable physics engines, where objects and explicit state and velocity representations are discovered by the model. This framework allows us to perform long term extrapolative video prediction, as well as vision-based model-predictive control. Our approach significantly outperforms related unsupervised methods in long-term future frame prediction of systems with interacting objects (such as ball-spring or 3-body gravitational systems), due to its ability to build dynamics into the model as an inductive bias. We further show the value of this tight vision-physics integration by demonstrating data-efficient learning of vision-actuated model-based control for a pendulum system. We also show that the controller's interpretability provides unique capabilities in goal-driven control and physical reasoning for zero-data adaptation.

Published as a conference paper at ICLR 2020
PHYSICC-AS--IVERREEGRRAII UNSUPERVISED PHYSICAL PARAMETER ESTIMATION FROM VIDEO
Miguel Jaques
Michael Burke
School of Informatics University of Edinburgh
School of Informatics University Edinburgh Edinburgh. UK
Edinburgh, UK m. a. m. jaques@ams ed. ac. uk
mi chael burkeled. ac. uk
Timothy Hospedales
School Informatics University of Edinburgh
Edinburgh, UK
t. hospeda Leseed. ac.duaccuu uk
ABSTRACT
We propose model that is able perform unsupervised physical parameter esti- mation of systems from video, where the differential equations governing the scene dynamics are known, but labeled states objects are not available. Existing physi- cal scene understanding methods require either object state supervision, do not integrate with differentiable physics to learn interpretable system parameters and states. We address this problem through physicc-ass-nnersee gerrreisgg approach that brings together
and differentiable physics engines, enabling objects and explicit state and velocity representations to be discovered. This framework allows us to perform long term extrapolative video prediction, well uision-based model--prdicctee performs related unsupervised methods in long-term future frame prediction systems with interacting objects (such as ball-spring 3-body gravitational sys- tems), due to its ability to build dynamics into the model as an inductive bias. We further show the value of this tight rision-physien integration by demonstrating data-efficient learning of vision- aattuaac model- -ased control for a pendulum sys- tem. We also show that the controller's interpretability provides unique capabilities in doil-nriven control and physical reasoning for zero-data adaptation.
control. Our approach significantly out-
INTRODUCTION
System identification or physical parameter estimation is commonly required for control or state estimation for physical modelling, and typically relies on dedicated sensing equipment and carefully constructed experiments. Current machine learning approaches physical modeling from video either require training by supervised regression from video to object coordinates before estimating explicit physies (Watters et al., 2017; Wu et al., [0176] Belbute-Peree discover and segment objects from video in an unsupervised manner, but do not naturally integrate with physics engine for long-term predictions or generation of interpretable locations and physical parameters for physical reasoning (Xu et al.) 22119: van Steenkiste et al. 2018). In this work, bridge the gap between unsupervised discovery of objects from video and learning the physical dynamics of system, by learning unknown physical parameters and explicit trajectory coordinates.
al., 2018). or are able to
Our approach, called physics-as ainess-s--iierseeeee peverse-graphics. solves the physical modeling problem via a novel vision-as- inverse graphics secoder-decoder system that can render and de-render image components using Spatial Transformers (ST) (Jaderberg et al.) (015)) a way that makes it possible for the latent representation generate disentangled interpretable states (position/velocity). These can be used

---

Published as a conference paper at ICLR 2020
directly by differentiable physics engine (Degrave et al. 2016 Belbute- Peres et al. 2018) to learn the parameters of a scene where the family of differential equations governing the system are known (e.g. objects connected by spring), but the corresponding parameters are not (e.g. spring constant). This allows us to to identify physical parameters and learn vision components of the model jointly in an end-to-end fashion. Our contribution is solution to unsupervised learning of physical parameters from video, without having access to ground-truth appearance, position or velocities of the objects, a task that had so far remained unsolved (Wu et al. 2013 Belbute-Peres et al. 2018).
In addition to showing that our model can learn physical parameters without object or state supervision (a task with intrinsic scientific interest in and of itself), we show that incorporating dynamics priors in the form of known physical equations of motion with learnable parameters together with learnable vision and graphics can improve model performance in two challenging tasks: long term video prediction and visual model predictive control. We first evaluate physical parameter estimation accuracy and future video frame prediction on 4 datasets with different non-linear interactions and visual difficulty. We then demonstrate the value of our method by applying it for data-efficient learning of vision-based control of an under-actuated pendulum. Notably our unique ability to extract interpretable states and parameters from pixels without supervision enables end-to-end vision-based control to exploit goal- paaulpaaaatterieed policies and physical reasoning for zero-shot adaptation.
2 RELATED WORK
The ability to build inductive bias into models through structure is a key factor behind the success of modern neural architectures. Convolutional operations capture spatial correlations (Fukushimal 1980) in images, recurrency allows for temporal reasoning (Hochreiter & Schmidhuber 1997). and spatial transformers (Jaderberg et al., 2015) provide spatial invariance in learning. However, many aspects of common data generation processes are not yet considered by these simple inductive biases. Importantly, they typically ignore the physical interactions underpinning data generation. For example. it is often the case that the underlying physics of a dynamic visual scene is known, even if specific parameters and objects are not. Incorporation of this information would be beneficial for learning, predicting the future of the visual scene, or control. Physics-as as as-ineerse graphics introduces a framework that allows such high-level physical interaction knowledge to be incorporated into learning, even when ground-truth object appearance, positions and velocities are not available.
In recent years there has been increased interest in physical scene understanding from video (Fragki- adaki et al. 2016]] Finn et al.) 2016 Fraccaro et al. 2017 Chang et al.] 2017 Jonschkowski et al. 2017; Zheng et al., 2018 Janner et al. 2019). In order to learn explicit physical dynamics from video our system must discover and model the objects in scene, having position as an explicit latent variable. Here we build on the long literature of neural vision- sision-as-inverss aas-inverse-eeee -n--erse--graphccs (Hinton et al.
2011
Kulkarni et al. 2015 [Huang & Murphy| [016]] Ellis et al. 2018 Romaszko et al.) [2017] Wu
ct al. 2017a), particulan larly on the use of spatial transformers (ST) for rendering (Eslami et al. 2016
Rezende al. 2016 Zhu al., 2018).
There are several models that assume knowledge of the family of equations governing system dynam- ics, but where the individual objects are either pre-segmented or their ground-truth positions//eloccties are known (Stewart& & Ermon 2017 Wu et al. 2017b; Belbutee-eerss et al.) 2018). In terms of learning physical parameters, our work is directly inspired by the Galileo model and Physics 101 dataset (Wu) et al. 2015; 2016), which fits the dynamics equations to scene with interacting objects. However, the Galileo model makes use of custom trackers which estimate the position and velocity of each object of interest, and is incapable of end-to-end learning from video, thus bypasses the difficulty of recognizing and tracking objects from video using a neural system. To the best of our knowledge, our model is the first to offer end-to-end unsupervised physical parameter and state estimation.
Within the differentiable physics literature (Degrave et al. 2016), Belbute-Peres et al. (2018) observed that multi-layer perceptron (MLP) encoder-decoder architecture with physics engine was not able to learn without supervising the physics engine's 's output with position/velocity labels (c.f. Fig. 4 in Belbute-Peres et al. (2018)). While in their case 2% labeled data is enough to allow learning, the transition to no labels causes the model to not learn at all. The key contribution of our work is the incorporation of vision-as-invense -repppeeseee with physics, which makes the transition possible.
2

---

Published as a conference paper at ICLR 2020
Decoder
Physics engine
U-Not
1
network
adtion
adtoon
acTio
astimator
(x,y)
Caordinate
Decader
Content
Concit
ST
Encoder
+D
Σ
Softmax
image
Figure 1: Left: High-level view of our architecture. The encoder (top-right) estimates the position of N objects each imput frame. These are passed to the velocity estimator which estimates objects' velocities at the last input frame. The positions and velocities of the last input frame are passed as initial conditions to the physics engine. At every time-step, the physics engine outputs set of positions, which used by the decoder (bottom-right) to output predicted image. If the system is actuated, an input action is passed to the physics engine at every time-step. See Section 3 for detailed descriptions of the encoder and decoder architectures.
Another related area of increasing interest unsupervised discovery of objects and/or dynamics from video (Xu et al. 2019; van Steenkiste et al., 2018 Greff et al., 2019 Burgess al. 2019). Though powerful, such models do not typically use interpretable latent representations that can be directly used by physics engine, reasoned about for physical problem solving, or that are of explicit interest to model users. For example, Kosiorek et al. (2018) and Hsieh et al. (2018) use ST's to locate/place objects in scene and predict their motion, but this work differs from ours in that our coordinate-consiste design obtains explicit cartesian, angular or scale coordinates, allowing us to feed state vectors directly into a differentiable physics engine. Under similar motivation our work, but without an inverse-grasn graphics approach, EErhardt et al. (2018) developed an unsupervised model obtain consistent object locations. However, this only applies cartesian coordinates, not angles or scale.
Despite recent interest model-free reinforcement learning, model-based control systems have repeatedly shown to be more robust and sample efficient (Deisenroth & Rasmussen 2011 Mania cet al. 2018 Watters et al., 2019a). Hafner et al. (2019) learn latent dynamics model (PlaNet) that allows for planning from pixels, which is significantly more sample efficient than model-free learning strategies A3C (Mnih et al. 2016 and D4PG (Barth-Maron et al.) 2018). However, when used for control, there is often desire for visually grounded controllers operating under known dynamics, as these are verifiable and interpretable (Burke et al.) 2019). and provide transferability and generality. However, system identification is challenging in vision- et al. (2018) use supervised learning to segment objects, controlling these using known rigid body dynamics. PPenkov & Ramamoorthy (2019) learn feedforward models with REINFORCE (Williams) 1992) to predict physical states used by a known controller and dynamical model, but this is extremely sample inefficient. In contrast, we learn parameter and state estimation modules jointly to perform unsupervised system identification from pixels, enabling data-efficient rision-actuatel model-base control.
control settings. [Byravan
LEARNING PHYSICAL PARAMETERS FROM VIDEO VIA INVERSE GRAPHICS
In order to learn explicit physics from video, several components have to be in place. First. the model must be able to learn to identify and represent the objects in an image. order to perform dynamics prediction with physics engine, the position and velocity of the object must be represented
3

---

Published as a conference paper at ICLR 2020
as explicit latent states (whereas appearance can be represented through some latent vector or, in video prediction architecture
case, as a set of learned object templates). Our consists of 4 modules trained jointly: encoder, a velocity estimator, a differentiable physics engine, and graphics decoder. The architecture is shown in Figure
input and outputs vector Pt RND corresponding to the D-dimensional coordinates of each of N objects in the scene, P; = ip................ P For example, when modelling position in 2D space have D 2 and p - [ar, y/]; when modelling object angle we have D = and 'º [o.] The encoder architecture is shown in Figure S right).
Encoder The encoder net takes single frame
To extract each object's coordinates use 2-stage localization approact First, the input frame is passed through a U-Net (Ronneberger et al. 2015) to produce N unnormalized masks. These masks (plus a learnable background mask) are stacked and passed through a softmax produce N W+1 1 masks, where each input pixel is softly assigned to a mask. The input image is then multiplied by each mask, and a 2-layer location network produces coordinate outputs from each masked input component. For a 2D system where the coordinates of each object are its (I, position (the polar coordinates case is analogous) and the images have dimensions H x H. the encoder output represents (xc, v) coordinates with values in [0, H). To do this, the activation of the encoder's output layer is saturating non- non-lineanee linearity H/2- tanh(.) + H/2.
Velocity estimator The velocity estimator computes the velocity vector of each object at the L-th input frame given the coordinates produced by the encoder for this object at the first L input frames, hidden layer MLP with 100 tanh activated units.
V% f(p", ...., p...). We implement this
Differentiable physics engine The physics engine contains the differential equations governing the system, with unknown physical parameters to be learned such as spring constants, gravity, mass, etc. Given initial positions and velocities produced by the encoder and velocity estimator, the physics engine rolls out the objects' trajectories. In this work we use simple physics engine with Fuler integration, where Pt, V, is computed from PI-1, Vc-1 by repeating for i E [1..MM]
At
At M
(1) where At is the integration step, 0 are the physical parameters and F is the force applied to each object, according to the equations in Appendix A. We use M = 5 in all experiments. In principle,
P++ 1 = P 35
Ve+ &
Vet = V
F(P4+ V
M
more complex physics engines could be used (Chen et al. 2018 Belbute-Peres et al. 2018).
Decoder The decoder takes as input the positions given by the encoder or physics engine, and outputs a predicted image T,. The decoder is the most critical part of this system, and is what allows the encoder, velocity estimator and physics engine to train correctly in fully unsupervised manner. We therefore describe its design and motivation in greater detail.
Coordinate-Consiste
While an encoder with outputs in the range [0, H] can represent coordinates in pixel space, it does not mean that the decoder will learn to correctly associate an input vector (zz, y) with an object located at pixel (z, 3). If the decoder unconstrained, like standard MLP, can very easily learn erroneous, non-linear representations of this Cartesian space. For example, given two different inputs, (x1, v1) and (2), with y1 F y2, the decoder may render those two objects at different horizontal positions in the image. While having correct Cartesian coordinate representation is not strictly necessary to allow physical parameters of the physics engine to be learned from video, it is critical to ensure correct future predictions. This is because the relationship between position vector and pixel space position must be fixed: if the position vector changes by (Ax, Ay), the object's 's position in the output image must change by (Ax, Ay). This is the key concept that allows us improve on Belbute- Peres et al. 2018), in order to learn an encoder, decoder and physics engine without state labels.
In order to impose correct latent-coordinate to pixel-coordinate correspondence, we use spatial transformers (ST) with inverse parameters as the decoder's 's writing attention mechanism. We want transformer parameters w to be such that decoder input of p = [ar, w) places the center of the writing attention window at (I, y) in the image, or that a decoder input of p OF rotates the attention window by 0. In the original ST formulation (Jderberg et al. 2015), the matrix w represents the affine transformation applied to the output image to obtain the source image. This means that the elements of w in Eq. of Jaderberg et al. (2015) do not directly represent translation, scale or angle of
'Toough any other architecture capable of effectively extracting object locations from images would work.
4

---

Published as a conference paper at ICLR 2020
Input -
Interpolation
Extrapolation
True
Physics+ InverseGraphics (Ours)
InverseGraphics
DDPAE
VideoLSTM
b-16
t-24
t-40
seq.
Physics
InverseGrrapics
VideoLSTM
tm6
b-9
1-12
t-15
t-18
t=21
t-24
t-27
b-30
Figure 2: Future frame predictions for 3-ball gravitational system (top) and 2-digit spring system (bottom). IN: Interaction Network. Only the combination of Physics and unverse-Graphin maintains object integrity and correct dynamics many steps into the future.
the writing attention window. To achieve this representation, we use ST with inverse transformation parameters. For general affine transformation with translation (z, y), angle 0 and scale we want to modify the source image coordinates according to:
Io
cos
8 sin 0
x
Yo
sin
00s 0
y
ys
1
0
1
1
This transformation can be obtained with ST by inverting (2):
Ex
coss
sin 0 - cos + y sin 0
/
y
sin
cos -x sin 0 - cos
(3)
0
Therefore, to obtain decoder with coordinate-consister outputs, we simply use a ST with parameters given in (3)
Each object is represented by learnable content c E [0, 1 and mask m E R tensor, L. Additionally, we learn background content bas [0, = and mask m bky E #x that do not undergo spatial transformation. One may think of the content as an RGB image containing the texture of an object and the mask grayscale image containing the shape and z-order of the object. In order to produce an output image, the content and mask are transformed according to [er m, ST([c", .mmm, wpp) and the resulting logit masks are combined via softmax across channels, [m] .... mu m, bkgj softmax ( ...,, m, kk). The output image is obtained by multiplying the output masks by the contents:
I T m
bkg
bkg
Σ m @ cp.
(4)
The decoder architecture is shown in Fig. bottom-right. The combined use ST's and masks provides a natural way to model depth ordering, allowing to capture occlusions between objects.
Auxiliary autoencoder loss Using constrained decoder ensures the encoder and decoder produces objects in consistent locations. However, it is hard to lear the full model from future frame prediction alone, since the encoder's 's training signal comes exclusively from the physics engine. To alleviate this and quickly build good encoder/decoder representation, add static per-frame autoencoder loss.

---

Published as a conference paper at ICLR 2020
Training During training L input frames and predict the next pred frames. Defining the frames produced by the decoder via the physics engine as Io and the frames produced by the decoder using the output the encoder directly as ft the total loss is:
L+Tpred
L+Tpred
Ltotal = Ppeed + af
Σ 1=
(0) + a E C(I
(5)
where a is a hyper-paramettr. We use mean-squared error loss throughout. During testing we predict an additional Text frames in order to evaluate long term prediction beyond the length seen for training.
EXPERIMENTS
4.1 PHYSICAL PARAMETER LEARNING AND FUTURE PREDICTION
2-balls bouncing
2- 2-balls spring
3-balls gravily
Studtral Siniarty lodex
Extrazotation
Edtrapo an
Saucturs Sinitrity Index
1.0
Extrupoiation
0.9
0.9
0.8
0.8
0.8
0.6
0.4
0.5
0.5
0.4
0.4
25
20
20
Prediction
Prediction stop
Prediction stop
Physics eyverssiinverserrapniisss (Ours)
BW-inverseGrophic
DDPAE
ViceoLSTM
Figure 3: Frame prediction accuracy (SSI, higher is better) for the balls datasets. Left of the green dashed line corresponds to the training range, Tpred. right corresponds to extrapolation, Tezt- We outperform Interaction Networks (IN) (Watters et al. 2017), DDPAE (Hsieh et al., 2018) and VideoLSTM (Srivastava al. 2015) in extrapolation due incorporating explicit physics.
Dataset
2-balls spring (k, () (4.26, 6.17) (4.0, 6.0)
2-digits spring (k: ()
3-balls gravity 3-balls gravity m
Parameters
Learned value
(2.18, 12.24)
65.7
0.95
Ground-truth value
(2.0, 12.0)
60.0
1.0
Table 1:
Physical parameters learned from video are within 10% of true system parameters.
Setup To explore learning physical parameters and evaluate long-term prediction we train our model scenes with 5 different settings: two colored balls bouncing off the image edges; two colored balls connected by a spring; three colored balls with gravitational pull all on a black background; and test greater visual complexity,
also use 2 MNSIT digits connected by spring, on CIFAR background. We train using values of (L, Pred. Text) set to 7,,7. (3, 7, 20). (3, 7, 20), (4, 12, 24) and (3, 7, 20), respectively. For the spring systems the physical parameters to learned are the spring constant & and equilibrium distance I, and for the gravitational system it is the gravity constant g or mass of the objects m (when learning gravity the mass if fixed, and vice-versa). In all cases we use objects with mass = 1. We provide the exact equations of motion used in these systems and other training details Appendices A and B, respectively. All datasets consist of 5000 sequences for training. 500 for validation, and 500 for testing. We use learnable ST scale parameter initialized at s = in the balls datasets and s = the digits dataset. these datasets we set
We compare our model to 3 strong baselines: DDPAE (Hiih et al. 2018 which is a generative model that uses an anverse-graphie model with black dynamics; VideoLSTM SSriaastava et al. 2015), which uses black-box encoding, decoding and dynamics; Interaction Network + Inverse- Inverse-Grrahics, which uses the same encoder and decoder as our Physics-ass -nniees------ Inwe-nn--rrappiiiie model, but where the dynamics module is Interaction Network (Battaglia et al.) 2016). The latter
Baselines
Using the code provided by the authors.

---

Published as a conference paper at ICLR 2020
model allows to compare explicit physics with relational dynamies networks, in terms of their ability to correctly capture object interactions 3
Results Table shows that our model finds physical parameters close to the ground-truth values used to generate the datasets, and Figure shows the contents and masks learned by the decoder. This highlights the fact that the proposed model can successfully perform unsupervised system identification from pixels. Future frame predictions for two of the systems are shown in Figure 2 and per-step Structural Similarity Index [f the models on the prediction and extrapolation range are shown in Figure 3 While all models obtain low error in the prediction range (left of the green dashed line), our model is significantly better in the extrapolation range. Even many steps into the future, our model's predictions are still highly accurate, unlike those of other black- box models (Figure 2). This shows the value of using an explicit physics model in systems where the dynamics are non-linear yet well defined. Further rollouts are shown in Appendix C, and we encourage the reader to watch the videos for all the datasets at atthpt:: /tt///sitese wwweeieeessogoglooooooooo.ooeooe com/
http://hyyiccassiieeeeees
This difference performance explained in part by the fact that in some of these systems the harder-to-prediet parts of the dynamics do not appear during training. For example, in the gravitational system, whiplash from objects coming in close contact is seldom present in the first K + Tpred steps given in the training set, but it happens frequently in the Tewt extrapolation steps evaluated during testing. We do not consider this to be a failure of black-box model, but rather a consequence of the generality vs specificity tradeoff: model without sufficiently strong inductive bias on the dynamics is simply not able to correctly infer close distance dynamics from long distance dynamics.
Train using
Cered
Lre
only Lpred separate gradients
31.4
20.5
a/0m c. olml
28.1
0.22
joint Lped + aCee
1.39
0.63
omioc
black-box decoder, joint
30.9
2.87
1bject
2
Object
Object Object 2 Background
Figure 4: Contents and masks learned by the decoder. Object masks: (m)) Ob- jects for rendering: o (m) C c. Contents and masks correctly capture each part of the scene: colored balls, MNIST digits and CI- FAR background. We omit the black back- ground learned on the balls dataset.
Table 2: Test loss under different training conditions. Separate gradients: Train encoder/decoder on Lise. and velocity estimator and physics engine on freed- Black-box decoder, joint: Joint training using a standard MLP network as the decoder. Only joint training using our coordinate-consiseen decoder succeeds.
Ablation studies Since the encoder and decoder must discover the objects present in the image and the corresponding locations, one might assume that the velocity estimator and physics engine could be learned using only the prediction loss, and encoder/decoder using only the static autoencoder loss, i... without joint training. In Table 2 we compare the performance of four variants on the 3-ball gravity dataset: joint training using only the prediction loss; joint training using the prediction and autoencoder losses; training the encoder/decoder on the autoencoder loss and the velocity estimator and physics engine on the prediction loss; and joint training, but using an MLP black- box decoder.
We can see that only joint prediction and autoencoder loss obtain satisfactory performance, and that decoder is critical. The prediction loss is essential in
the use of the proposed coordinate-consiste order for the model to learn encoders/decoders whose content and masks can be correctly used by the physics engine. This can be understood by considering how object interaction influences the decoder. In the gravitational system, the forces between objects depend on their distances, so the objects swap locations, the forces must be the same. If the content/mask learned for each object are centered differently relative to its template center, rendering the objects at positions [ar, y] and [w, 2], or [w, 2] and [x. v) will produce different distances between these two objects in image space. This violates the
2 Thiis baseline also serves as strong proxy for comparison with recent relational models Watters al. 2017 van Steenkiste et al.] 2018), which due to their supervision method or input-output space cannot be directly compared our model.
We choose SSI over MSE evaluation metric it more robust pixel-level differences and alignment.
7

---

Published as a conference paper at ICLR 2020
Data efficiecy
Zero-shot transfer
Gool anglee
90
90
80-
80
2127
Reward
70
Reward
Curs Cradle
60
Durs
50
Rafeet
Cracle
00
Ours
7777
20
25
35 Reward
30
40
ipropris
40
Kandom agent
oand (VAP)
50
Dracle
Plaell
0.5
0.
0.7
08
1.0
12
14
1.5
1.8
225
138
Environment steps (x1000)
Gravity multipller
0rack
\
\
Palee
/
I
/
1
\
/
/
(gux/cond)
b-
t-10
t-15
7+20
t-25
1+3)
7-35
t-40
b-45
1+50
t-55
b-6]
t-65
t-70
t-75
1-80
t-55
t-90
1-50
T-300
Goal
Figure 5: Top: Comparison between our model and PlaNet Harner et al. (2019) terms of learning sample efficiency (left). Explicit physics allows reasoning for zero-shot adaptation to domain-shi in gravity (center) and goal-driven control to balance the pendulum in any position (right). DDPG (VAE) corresponds
DDPG agent trained on the latent space of autoencoder (trained with 320k images) after 80k steps. DDPG (proprio) corresponds to an agent trained from proprioception after 30k steps. Bottom:
The first 3 rows show zero-shot counterfactual episode with a gravity multiplier of 1.4 for an oracle, our model and planet, with vertical as the target position (as trained). The last row shows episode using a goal image to infer the non-vertical goal state.
permutation invariance property of the system. Learning the encoder/decoder along with the velocity estimator and physics engine the prediction loss allows the encoder and decoder to learn locations and contents/masks that satisfy the characteristics of the system and allows the physics to be learned correctly. In Appendix D we perform further ablations on the decoder architecture and its ability to correctly render objects in regions of the image not seen during training.
4.2 VISION- VISION-BASED PREDICTVV CONTROL (MPC)
Tasks One of the main applications of our method is to identify the (actuated) dynamical parameters and states of a physical system from video, which enables vision-based planning and control. Here we apply to the pendulum from OpenAl Gym (Brockman et al., 2016) one typically solved from proprioceptive state, not pixels. For training we collect 5000 sequences of 14 frames with random initialization (6, ~ Unif( -,, 6, 6)) and actions (u, ~ Unif( 2, 2)). The physical parameters to learn are gravity g = 10.0 and actuation coefficient a 1.0. We use K and Tpred 10. We use the trained MPC model as follows. At every step, the previous 4 frames are passed to the encoder and velocity nets to estimate [0r, 0z). This is passed to the physics engine with learned parameters g and a. We perform 100-step model-predictive control using the cross entropy method (Rubinstein 1997). exactly described in Halner et al. (219)), setting vertical position and zero velocity as the goal. Baselines We compare our model to an oracle model, which has the true physical parameters and access to the true pendulum position and velocity (not vision-based), as well as a concurrent state-of-the art model-based RL method (PlaNet (Hafner et al.) 2019)), and a model-free deep deterministic policy gradient (DDPG) agent (Lillicrap al. 2016). To provide an equivalent comparison to our model, we train PlaNet on random episodes.
Results In terms of system identification, our model recovers the correct gravity (g 9.95) and force coefficient (a = 0.99) values from vision alone, which is a prerequisite to perform correct planning and control. Figure (top-left) highlights the data efficiency of our method, which is comparable to PlaNet, while being dramatically faster than DDPG from pixels. Importantly, the
DDPG, TRPO and PPO learned from pixels failed to solve the pendulum, highlighting the complexity of the vision-hased pendulum control task and brittleness of model-free reinforcement learning strategies.
8

---

Published as a conference paper at ICLR 2020
interpretability of the explicit physies in model provides some unique capabilities. We can perform simple counter-factual physical reasoning such as 'How should I adapt my control policy if gravity was increased?", which enables zero-shot adaptation to environmental parameters. (top-middle) shows that our model can exploit such reasoning to succeed immediately over wide range of gravities with no re-training. Similarly, while the typical inverted pendulum goal is to balance the pendulum upright, interpretable physics means that this is only one point in space of potential goals. Figure (top-right) evaluates the poal-paramaterized control enabled by our model. Any feasible target angle specified can be directly reached by the controller. There is extrapolative generalisation across the space of goals even though only one goal (vertical) was seen during training. Importantly, these last two capabilities are provided automatically by our model due to its disentangled interpretable representation, but cannot be achieved without further adaptive learning by alternatives that are reward-based (Mnih et al. 2016) or rely on implicit physics (Hafner ct al. 2019).
Figure 5
LIMITATIONS
Alhough the approach presented here shows promising results in terms of physical parameter es- timation, long-term video prediction and MPC, a number of limitations need to be overcome for real-world application.
Templates as object representation Though the assumption that every scene in a dataset is combination of learnable templates common one in the literature (c.f. Tieleman (2014) for an extensive study on this), this is insufficient to model real-world scenes. For example, applying sanblss-s--ivvree-gree
to the Physics101 dataset would require representing objects using a latent appearance representation that could be used by the decoder (Fslami et al.] 2016). This would introduce new modelling challenges, requiring object tracking to keep correct object identity associations (Kosiorek al. 2018). In this work we simplify this problem by assuming that objects visually distinct throughout the dataset, though this does not detract from the essential contributions of the paper.
Rigid sequence to sequence architecture this work we used a sequence-to- toaeece-tt-sqeenne architecture, with a fixed number of input steps. This architectural choice (inspired by Watters et al. (2017)). prevents the model from updating its state beliefs if given additional input frames later in the sequence. Formulating the current model in probabilistic manner that would allow for state/parameter filtering and smoothing at inference time is promising direction of future work.
Static background assumption Many scenes of interest do not follow the assumption that the only moving objects in the scene are the objects of interest (even though this assumption is widely used). Adapting our model varying scene backgrounds would require additional components to discern which parts of the scene follow the dynamics assumed by the physics engine, in order to correctly perform object discovery. This a challenging problem, but we believe would greatly increase the range of applications of the ideas presented here.
6 CONCLUSION
Physics-as -nnsssrs--ieeeeeeee graphics provides valuable mechanism to integrate inductive bias about physical data generating processes into learning. This allows unsupervised object tracking and system identi- fication, in addition to sample efficient, generalisable and flexible control. However, incorporating this structure into lightly supervised deep learning models has proven challenging introduced a model that accomplishes this, relying on coordinate-consisten decoder that enables image reconstruction from physics. We have shown that our model able to perform accurate long term prediction and that it can be used to learn the dynamics of an actuated system, allowing us to perform aasion-based model-predictio control.
date. We
REFERENCES
Gabriel Barth-Maron Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap. Distributed distributional deterministic policy gradients. ICLR, 2018.

---

Published as a conference paper at ICLR 2020
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interac- tion Networks for Learning about Objects, Relations and Physics. In NIPS, 2016.
Filipe De A Belbute-Peres, Kevin A Smith, Kelsey R Allen, Joshua B Tenenbaum, and J Zico Kolter. End-to-End Differentiable Physics for Learning and Control. In NIPS, 2018.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. 2016.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerehner. MONet: Unsupervised Scene Decomposition and Representation. CoRR, abs/1901. 1111901111199,, 11390, 2019.
Michael Burke, Svetlin Penkov, and Subramanian Ramamoorthy. From explanation to synthesis: Compositional program induction for learning from demonstration. Robotics: Science and Systems (R:SS), 2019.
Arunkumar Byravan, Felix Leeb, Franziska Meier, and Dieter Fox. SE3-Pose-Netss Structured Deep Dynamics Models for Visuomotor Planning and Control. In ICRA, 2018.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A Compositional Object-Based Approach Learning Physical Dynamics. In ICLR, 2017.
Ricky Q Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural Ordinary
Differential Equations. In NIPS, 2018.
Jonas Degrave, Michiel Hermans, Joni Dambre, and Francis wyffels. A Differentiable Physics Engine for Deep Learning Robotics. 2016.
Mare Deisenroth and Carl Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In ICML, 2011.
Sebastien Ehrhardt, Aron Monszpart, Andrea Vedaldi, and Niloy Mitra. Unsupervised Intuitive Physics from Visual Observations. CoRR, abs/1805.08095, 2018.
Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Joshua B. Tenenbaum. Learning to Infer Graphics Programs from Hand-Drad Images. In NIPS, 2018
S M Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, Koray Kavukcuoglu, and Geoffrey Hinton. Attend, Infer, Repeat: Fast Scene Understanding with Generative Models. In NIPS, 2016.
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised Learning for Physical Interaction through Video Prediction. NIPS, 2016.
Marco Fraccaro, Simon Kamronn, Ulrich Paquet, and Ole Winther. A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning. In NIPS, 2017.
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning Visual Predictive Models Physics for Playing Billiards. In ICLR, 2016.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological cybernetics, 36(4):193-202, 1980.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. In /CML, 2019.
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. ICML, 2019.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural Networks for Machine Learning Lecture 6a: Overview of bacc- gradient descent. Technical report,
Geoffrey E. Hinton, Alex Krizhevsky, and Sida D. Wang. Transforming auto-encoders. In ICANN, pp. 44-51, 201 1.
10

---

Published as a conference paper at ICLR 2020
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to Decompose and Disentangle Representations for Video Prediction. In NIP,, 2018.
Jonathan Huang and Kevin Murphy. Efficient Inference in Occlusion-Aware Generative Models of Images. In ICLR Workshop, 2016.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial Transformer Networks. NIPS, 2015.
Michael Janner, Sergey Levine, William Freeman, Joshua Tenenbaum, Chelsea Finn, and Jiajun Wu. Reasoning About Physical Interactions with Object-Oriented Prediction and Planning. In ICLR, 2019.
Rico Jonschkowski, Roland Hafner, Jonathan Scholz, and Martin Riedmiller. PVEs: Position- Velocity Encoders for Unsupervised Learning of Structured State Representations, CoRR, abs//105500905. 2017.
Position-Velocity
Adam R Kosiorek, Hyunjik Kim, Ingmar Posner, and Yee Whye Teh. Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects. In NIPS, 2018.
Tejas D Kulkarni, Will Whitney, Pushmeet Kohli, and Joshua B Tenenbaum. Deep Convolutional Inverse Graphics Network. In NIPS, 2015.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2016.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search provides competitive approach to reinforcement learning. NIPS, 2018.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In /CML, 2016.
Svetlin Penkov and Subramanian Ramamoorthy. Learning programmatically structured representa- tions with perceptor gradients. ICLR, 2019.
Danilo J. Rezende, Shakir Mohamed, Ivo Danihelka, Karol Gregor, and Daan Wierstra. One-Shot Generalization in Deep Generative Models. In ICML, 2016.
Lukasz Romaszko, Christopher K Williams, Pol Moreno, and Pushmeet Kohli. Vision-as-Inverse Graphics: Obtaining a Rich 3D Explanation a Scene from a Single Image. In ICCV, 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In MICCAI, 2015.
Reuven Y. Rubinstein. Optimization of computer simulation models with rare events. EJOR, 1997.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised Learning of Video Representations using LSTMs. In /CML, 2015.
Russell Stewart and Stefano Ermon. Label-Free Supervision of Neural Networks with Physics and Domain Knowledge. In AAAI, 2017.
Tijmen Tieleman. Optimizing Neural Networks that Generate Images. PhD thesis, 2014.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions. In ICLR, 2018.
Nicholas Watters, Andrea Tacchetti, Théophane Weber, Razvan Pascanu, Peter Battaglia, and Daniel Zoran. Visual Interaction Networks: Learning Physies Simulator from Video. In NIPS, 2017.
11

---

Published as a conference paper at ICLR 2020
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander Lerchner. COBRA: Data-Efficient Model -ased RL through Unsupervised Object Discovery and Curiosity- Driven Exploration. CoRR, abs/1905.09275, 2019a.
Nicholas Watters, Loie Matthey, Christopher Burgess, and Alexander Lerchner. Spatial Broadcast Decoder: A Simple Architecture for Learning Disentangled Representations in VAEs. 2019b.
Ronald Williams. Simple Statistical Fradient-Following Algorithms for Connectionist Reinforce- ment Learning. Machine Learning, 3:229-256, 1992.
Jiajun Wu, llker Yildirim. J.J. Lim, W.T. Freeman, and J.B. Tenenbaum. Galileo Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning. In NIPS, 2015.
Jiajun Wu, Joseph J Lim, Hongyi Zhang. Joshua B Tenenbaum, and William T Freeman. Physics 101: Learning physical object properties from unlabeled videos. BMVC, 2016.
Jiajun Wu, Mit Csail, Joshua B Tenenbaum, and Pushmeet Kohli. Neural Scene De-rendering. In CVPR, 2017a.
Jiajun Wu, Erika Lu, Pushmeet Kohli, William T Freeman, and Joshua B Tenenbaum. Learning to See Physics via Visual De-animation. In NIPS, 2017b.
Zhenjia Xu, Zhijian Liu, Chen Sun, Google Research, Kevin Murphy, William T Freeman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised Discovery of Parts, Structure, and Dynamics. In ICLR, 2019.
David Zheng, Vinson Luo, Jiajun Wu, and Joshua B Tenenbaum. Unsupervised Learning of Latent Physical Properties Using Perception Perception-Prediction Prediction Networks. In UAI, 2018.
Guangxiang Zhu, Zhiao Huang, and Chongjie Zhang. Object-Oriented Dynamics Predictor. In NIPS, 2018.
A SYSTEM DESCRIPTIONS
In this section we describe the equations of motion used for each system.
2-balls and 2-digits spring The force applied on object by object j follows Hooke's 's law:
F.j - (f. 13)))
p. Pj
(6)
P. P1
Each step corresponds to an interval AL- = 0.3.
3-balls gravity The force applied on object by object j follows Newton's law of gravity:
P: P3 p, p 3
F.g --mmm
where the masses are set to 1. Each step corresponds to an interval At = 0.5. Pendulum The pendulum follows the equations used by the OpenAI Gym environment:
3 2 sin (0 + +T) + 3u
F
where u is the action. Each step corresponds to interval At 0.05. In the physics engine used by the model we introduce an extra actuation coefficient to be learned along with g:
3 2 sin (0 + ④) + a.a.
F
(9)
12

---

Published as a conference paper at ICLR 2020
B TRAINING DETAILS AND HYPERPARAMETERS
For all datasets we use RMSProp Hinton et al. 2012) with an initial learning rate of 10-1, For the balls and digits datasets we train for 500 epochs with a and divide the learning rate by 5 after 375 epochs. For the pendulum data we train for 1000 epochs using a = 3, but divide the learning rate by after 500 epochs. The image sizes are x 32 for the 2-balls bouncing and spring, 36 36 for the 3-balls gravity, 64 x 64 for the 2-digits spring, and 64 x 64 grayscale for the pendulum.
The content and mask variables are the output of neural network with constant array of Is as input and hidden layer with 200 units and tanh activation. We found this easier to train rather than having the contents and masks as trainable variables themselves.
13

---

Published as a conference paper at ICLR 2020
C ADDITIONAL ROLLOUTS FOR EACH DATASET
3 BALLS GRAVITY
Input
Interpolation
Extrapolation
True seq.
Phylins+ InverseGraphics (Qurs)
InverseGraphics
VideLLTM
I-4
tw12
I-16
1-20
1-24
tw28
tw32
Tw36
[-40
Input...
Interpolation
Extrapelation
True seq.
Physics+ InverseGraphicc [Ours)
I
InverseGraphics
DEPAF
VideOLSTM
t-4
bwB
b-12
t-16
t-20
1-24
t-28
t=32
t-36
t-40
2-BALLS SPRING
Input
Interpolation
Extrapolation
seq.
Physics+
GnverseGraphits
(Ours)
InverseGraphics
DOPAE
VIdeaL.SMM
t-3
t-6
t-9
t-12
t-15
t-18
t-21
t-21
t-27
t-30
Interpolation
Extrapolation
True seq.
Physics+ InverseGGrpphil
(Ours)
InverseGraphics
DOPAE
t-3
6-
t-
t-12
t-15
t-18
t-21
t-21
t-27
t-30
14

---

Published as a conference paper at ICLR 2020
2- BALLS BOUNCING
Input
Interpolation
Extrapolation
True
Physics Inversefinphics (Ours)
IN+
InverseGraphics
VideoLSTM
t=3
t=
t-
t=12
t=115
t-24
t-27
t-30
Input
Interpolation
Extrapolation
Physics*
InverseGraphics
(Ours)
InverseGraphics
DDPNE
VideoLSTM
2- E-DGITTS SPRING
True seq.
Physics+
InverseGraphics
VideoLSTM
L-3
L=6
Lm9
t=12
t=15
L=18
L-21
1-24
1-27
t-30
True seq,
Physics+ InverseGrapthic (Ours)
VideoLSTM TM
t=12
t=15
tw18
t=21
t=24
t=27
t=30
D EXTRAPOLATION TO UNSEEN IMAGE REGIONS
One limitation of standard fully-connected or deconvolutional decoders is the inability to decode states corresponding to object poses locations not seen during training. For example, if in the training set no objects appear in the bottom half of the image, fully-connected decoder will simply learn to output zeros in that region. If in the test set objects move into the bottom half of the image, the decoder lacks the inductive bias necessary correctly extrapolate in image space.
To test this hypothesis, we replaced our model's 's decoder with a Deconv and Spatial Broadcast (Watters al. 2019b) decoder, and compared them in spatial extrapolation experiment. In this experiments, objects never enter the bottom half the image in the input and prediction range, though in the extrapolation range in the test set objects move this region of the scene. In the
15

---

Published as a conference paper at ICLR 2020
rollouts shown in Fig. Broadcast performs better than Deconv, but they both fail to maintain object integrity when the balls move to the bottom half of the image in the extrapolation steps, validating our hypothesis that black-bot decoder has insufficient extrapolation ability. In contrast, our rendering decoder be able to correctly decode states not seen during training.
In the limit that renderer corresponds full-blown graphics-engine, any pose, location, color, etc. not seen during training can still be rendered correctly. This property gives models using rendering decoders, such efficiency. We note, however, that in general this advantage does not apply to correctly inferring the states from images whose objects are located in regions not seen during training. This is because the encoders used are typically composed simply of convolutional and fully-connected layers, with limited de-rendering inductive biases.
and Hsiih al. (2018). an important advantage terms of data-
Input
Interpolation
Extrapolation
True seq.
Graphics (Ours)
Broadcast
t-3
t-6
t-9
1-12
t-15
t-18
1-21
1-24
1-27
1-30
Figure 6: Comparison between graphics decoder and two black-box decoders, trained data where objects only appear in the top half of the scene. Only the graphics decoder is able to correctly render the objects in the bottom half of the scene test time. Broadcast: spatial broadcast decoder (Watters et al. 2019b); Deconv: standard deconvolutional network.
E INCORRECT NUMBER OF OBJECT SLOTS
The model proposed assumes we know the number of objects present the scene. Here we briefly explore how the model behaves when gravitational system, since interaction forces between objects are easy to generalize for any N. Fig. left, shows that when using only object slots, of the objects found, since the model does not have capacity to find more. Fig. 77 right, shows that when using more slots than the number of objects the scene, all objects are discovered, and extra slots are left empty. However, in both cases we found predictive performance to be subpar, since in one case there are objects missing to correctly infer interactions, and in the other there are interactions between object slots and empty slots, confusing the dynamics.
incorrect number of slots N. We use the
2 object slots
object
m(m)
Object
Object
Object
Object
Object
Object
Figure 7: Results for incorrect number of object slots in the physics engien for the 3-body gravitational system Left: Contents and masks learned for object slots. Right: Contents and objects learned for 4 object slots.
16


Reviews:
[{"review_id": "ryeX_6kAtH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an approach for unsupervised estimation of physical parameters from video, using the physics as inverse graphics approach. The approach uses a feedforward encoder for localising object positions from which a velocity is estimated. These are fed as inputs to an (action-conditioned) physics simulator which generates future predictions of object positions. This simulator has knowledge of the system dynamics apriori, needing estimation of a few physical parameters such as gravity and spring constants. The outputs of this simulator is fed to a co-ordinate consistent decoder, a neural network that uses a Spatial Transformer to render the corresponding output image. The whole system is trained end-to-end on videos of dynamical systems, in an unsupervised manner. Results on two and three body interaction settings and an MNIST digit motion dataset show promising performance. The system is able to recover the underlying physical parameters accurately while also making consistent long-term ex predictions. Additionally, the model is used for visual MPC on a simulated cartpole task where it outperforms state of the art model-based and model-free RL baselines.\n\nThis paper is well written and clearly motivated, albeit a bit incremental in its approach. Many of the building blocks have been explored in prior work, with the major component being the co-ordinate consistent decoder. The experiments are visually simplistic and it is not obvious if the system will scale to more complex settings. A few comments:\n1. A major limitation of the approach is the assumption that the equations governing the system are known. This makes it harder to generalise the system to novel tasks and more complicated settings with contact and object interactions. A potential way to overcome this could be to use ideas from prior work such as Interaction Networks where the dynamics are modelled as unary and binary interactions. While such dynamics models are black-box and not as interpretable compared to the current approach, they can easily generalise to novel tasks. Additionally, using positions and velocities as the latent state representation together with IN style transition models can be a sensible middle ground.\n2. It would be great if there is an additional ablation experiment where the known equations of motion are replaced with a black-box neural network (while still retaining the position and velocity representation). This can quantify the effect of known dynamics and make the contributions of the paper (with regards to the decoder) more clear.\n3. An alternative way of generating consistent object positions from the encoder is to compute a mask-weighted average of the image co-ordinates. This can be a nice way of adding additional structure to the networks that can regularise training.\n4. The approach uses a 3-layer MLP for generating velocity estimates — could this not be done via finite differencing? (e.g. higher-order backward differencing)\n5. Both the content and mask vectors are learnable but fixed for the entire task — i.e. it is not a function of the input image. This makes the approach not applicable to novel objects or even objects with minor color changes. \n6. It is not clear how the translations, rotations and scale parameters for the Spatial Transformer are estimated. I presume that the positions and orientations predicted by either the encoder or physics simulator are directly used. This needs to be clarified in the main paper. \n7. The paper mentions that the background masks are known when localising the objects via the encoder. If this is the case, the localisation problem becomes somewhat trivial. This should be clarified.\n8. There needs to be a clear discussion on the limitations of the current approach — it does not scale to novel objects, needs to know the number of objects apriori and has not been shown to estimate object properties such as mass, friction etc. \n\nOverall, the approach presents promising initial results towards an unsupervised method for modelling dynamical systems from video. There are several limitations that need to be made explicit and some additional experiments on more complicated systems and a few ablation studies can significantly improve the strength of the paper. I would suggest a borderline accept.\n\nTypos:\n1. Section 4.1, Setup: 5 values of (K, t_pred, t_ext) are given, need only 4\n2. “K” is not introduced till the results section\n\nAdditional citation:\nThe following paper on learning physically consistent position and velocity representations for dynamical systems (and for use in visual MPC settings) should be cited:\nJonschkowski, Rico, et al. \"Pves: Position-velocity encoders for unsupervised learning of structured state representations.\" arXiv preprint arXiv:1705.09805 (2017).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rkxP3xuEYB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to integrate model-based physical simulation and data-driven (deep) learning. In a nutshell, one deep network predicts the state variables of the physics simulation (such as objet location, shape and velocity) from an image. A second network does the inverse task, to render images given the state variables (and a background image). In this way, one can go from a video frame to a physical system state, modify the state with physics simulation, and then go back from the modified state to a video frame. Together with a differentiable physics engine, through which one can back-propagate, this makes it possible to use the un-annotated video itself as supervision. At the same time, the two neural networks can be seen as an auto-encoder, in which the latent state is explicitly constrained to correspond to the desired physical state variables.\n\nThe topic of the paper is hot: a proper integration of physical models with data-driven deep learning is, arguably, one of the big short- to mid-term themes of machine learning research. The way it is done in the present paper intuitively makes sense. The approach is fairly obvious at the conceptual level; but in the details poses a number of technical challenges especially for the decoder, which are nicely analysed and resolved.\n\nSome minor design choices are not well justified and at first sight appear a bit l'art pour l'art. While it is a sensible, pragmatic choice to first predict object masks, then extract their location ands and velocities in a second step; I do not quite see why one would have to do the latter with neural networks. it would seem that once the masks have been found, their location can be chosen as something like the (perfectly differentiable) mask-weighted centroid and does not need a multi-layer network; and similarly that deriving velocity from locations in adjacent frames can be hard-coded and does not need a 3-layer network.\n\nThe experiments are still at an early \"toy\" level, with synthetic videos where high-contrast, homogeneous objects move in front of a uniform or blurry background. The baselines are sensible and ablation studies are done with care. Still, it would have been nice to also run the method on some real video. To my understanding, this would be easily possible at least for future frame prediction, all one has to do is either annotate the objects in the target frame or measure success by comparing the predicted and true frames at the image level. It is also not clear whether the videos were synthesised with the same physics engine also use inside the system - which would be slightly questionable, in the sense that the learnable pipeline is then a-priori matched to the biases in the data.\n\nOne comment on the presentation: while the paper is generally well-written and easy to follow, the wording could at times be more careful. There is a slight tendency to identify the particular (simple) physical systems of the paper with physics as a whole. E.g., not all physics simulation must have objects - for instance, fluid dynamics or radiative transfer do not have individual objects, but are nevertheless relevant in the context of visual data. Similarly, even for defined objects, position and velocity are not always a sufficient state, for instance objects might deform, or have different elastic properties when colliding.\n\nOverall, I find the work interesting and well-executed. It is a natural step to take towards the important goal of integrated data-driven and physical models, including the associated theme of self-supervision via physical constraints. On the negative side the paper does make a slightly rushed and unfinished impression by not showing any, even qualitative, experiments on real video. Most people - rightly - use simple toy-like datasets for development and analysis. But showing only those gives me the impression that the paper was written too early, just to be the first and to make the deadline. Or that moving to real video poses a much greater challenge than expected - but then this should be stated and discussed.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "S1xGzyBJcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a method for jointly making physical predictions and inferring latent physical parameters (e.g. gravity) in an unsupervised manner from video. Specifically, the proposed architecture consists of an object-centric encoder which estimates dynamic properties of each object in the scene (e.g. position), a differentiable physics engine, and a decoder. \n\nI enjoyed reading this paper and think that it is a valuable contribution to the literature on physical reasoning, and thus lean towards acceptance. It elegantly combines several ideas that have been recently been investigated in the literature, though not yet put together. The experiments are well done and encompass not just prediction and inference but also control. The results look very impressive compared to existing models as well. However, my main critique would be that the evaluation domains are somewhat simplistic. If experiments in slightly more complex domains could be performed then I would be willing to increase my score to a full accept.\n\nIn the present paper, the scenes consist of only two or three objects, which is quite small compared to other papers in the literature which have evaluated on scenes with 6 or more objects. Similarly, given that the paper says it was inspired by the Physics 101 dataset, it is a bit disappointing that the proposed model was not evaluated on Physics 101 which would provide a more ecologically valid test of the model. At a minimum, I would like to see experiments with at least 6 objects if not more. Beyond that, including experiments on Physics 101 would change this from a good paper to an excellent paper.\n\nI had a number of additional questions and comments:\n\nI wondered how the proposed method would far at inferring object-specific latent parameters which cannot be inferred from images alone, such as friction or density. It seems like the velocity encoder could try to make these predictions as well, but it is not clear to me how well this would work with only a few frames.\n\nSomewhat relatedly, it seems like a limitation of the method is that it would not work if the objects were not visually distinct (i.e. if the balls were all the same color)---in other words, it cannot track objects over time but must learn a fixed mapping of visual property (color) to slot. This is fine to leave for future work, but merits discussion. In particular, I suspect this is also related to why the IN results are so poor; due to small errors in the IN the objects end up out of the frame, and then because the model has no memory component, it just forgets about them. If the model had access to a memory that could track objects that leave the scene, then I expect the IN would fare much better. Similarly, I expect that even the model with the perfect simulator would fail in scenes in which objects can be fully occluded. I would appreciate if some discussion of these limitations could be added to the paper.\n\nI am curious how well the model would perform if the number of object slots were not correct (i.e. if N is less then that actual number of objects, or more than the number of objects). It would be great if to include some experiments on this in the appendix.\n\nThe related work should probably mention the recent COBRA architecture [1], which also uses unsupervised scene decomposition combined with model-based RL.\n\nCan you clarify whether the interaction net baseline is pretrained, or trained end-to-end with the encoder and decoder?\n\nWhat are the errorbars over in Figure 5? Are they multiple seeds? If not, then I would like to see the figure updated with results from multiple training runs in order to properly assess variance.\n\nPage 4: can you give more details on what a “fixed background mask” is?\nPage 6: what is K? Is this supposed to be L (the number of frames used for velocity estimation?)\nPage 6: when describing the values of (K, T_pred, T_ext), why are there 5 different settings? \nThe paper states earlier in the paragraph that there are only 4 different systems so I am a bit confused what these settings correspond to.\n\n[1] Watters, N., Matthey, L., Bosnjak, M., Burgess, C. P., & Lerchner, A. (2019). COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration. arXiv preprint arXiv:1905.09275.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents an approach to estimating physical parameters from video. The approach is sensible and is presented fairly well. The main criticism is that the approach is only demonstrated in simplistic "toy" settings. Nevertheless, the reviewers recommend (weakly) accepting the paper and the AC concurs.
Decision: Accept (Poster)