Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a method for jointly making physical predictions and inferring latent physical parameters (e.g. gravity) in an unsupervised manner from video. Specifically, the proposed architecture consists of an object-centric encoder which estimates dynamic properties of each object in the scene (e.g. position), a differentiable physics engine, and a decoder. 

I enjoyed reading this paper and think that it is a valuable contribution to the literature on physical reasoning, and thus lean towards acceptance. It elegantly combines several ideas that have been recently been investigated in the literature, though not yet put together. The experiments are well done and encompass not just prediction and inference but also control. The results look very impressive compared to existing models as well. However, my main critique would be that the evaluation domains are somewhat simplistic. If experiments in slightly more complex domains could be performed then I would be willing to increase my score to a full accept.

In the present paper, the scenes consist of only two or three objects, which is quite small compared to other papers in the literature which have evaluated on scenes with 6 or more objects. Similarly, given that the paper says it was inspired by the Physics 101 dataset, it is a bit disappointing that the proposed model was not evaluated on Physics 101 which would provide a more ecologically valid test of the model. At a minimum, I would like to see experiments with at least 6 objects if not more. Beyond that, including experiments on Physics 101 would change this from a good paper to an excellent paper.

I had a number of additional questions and comments:

I wondered how the proposed method would far at inferring object-specific latent parameters which cannot be inferred from images alone, such as friction or density. It seems like the velocity encoder could try to make these predictions as well, but it is not clear to me how well this would work with only a few frames.

Somewhat relatedly, it seems like a limitation of the method is that it would not work if the objects were not visually distinct (i.e. if the balls were all the same color)---in other words, it cannot track objects over time but must learn a fixed mapping of visual property (color) to slot. This is fine to leave for future work, but merits discussion. In particular, I suspect this is also related to why the IN results are so poor; due to small errors in the IN the objects end up out of the frame, and then because the model has no memory component, it just forgets about them. If the model had access to a memory that could track objects that leave the scene, then I expect the IN would fare much better. Similarly, I expect that even the model with the perfect simulator would fail in scenes in which objects can be fully occluded. I would appreciate if some discussion of these limitations could be added to the paper.

I am curious how well the model would perform if the number of object slots were not correct (i.e. if N is less then that actual number of objects, or more than the number of objects). It would be great if to include some experiments on this in the appendix.

The related work should probably mention the recent COBRA architecture [1], which also uses unsupervised scene decomposition combined with model-based RL.

Can you clarify whether the interaction net baseline is pretrained, or trained end-to-end with the encoder and decoder?

What are the errorbars over in Figure 5? Are they multiple seeds? If not, then I would like to see the figure updated with results from multiple training runs in order to properly assess variance.

Page 4: can you give more details on what a “fixed background mask” is?
Page 6: what is K? Is this supposed to be L (the number of frames used for velocity estimation?)
Page 6: when describing the values of (K, T_pred, T_ext), why are there 5 different settings? 
The paper states earlier in the paragraph that there are only 4 different systems so I am a bit confused what these settings correspond to.

[1] Watters, N., Matthey, L., Bosnjak, M., Burgess, C. P., & Lerchner, A. (2019). COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration. arXiv preprint arXiv:1905.09275.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents an approach for unsupervised estimation of physical parameters from video, using the physics as inverse graphics approach. The approach uses a feedforward encoder for localising object positions from which a velocity is estimated. These are fed as inputs to an (action-conditioned) physics simulator which generates future predictions of object positions. This simulator has knowledge of the system dynamics apriori, needing estimation of a few physical parameters such as gravity and spring constants. The outputs of this simulator is fed to a co-ordinate consistent decoder, a neural network that uses a Spatial Transformer to render the corresponding output image. The whole system is trained end-to-end on videos of dynamical systems, in an unsupervised manner. Results on two and three body interaction settings and an MNIST digit motion dataset show promising performance. The system is able to recover the underlying physical parameters accurately while also making consistent long-term ex predictions. Additionally, the model is used for visual MPC on a simulated cartpole task where it outperforms state of the art model-based and model-free RL baselines.

This paper is well written and clearly motivated, albeit a bit incremental in its approach. Many of the building blocks have been explored in prior work, with the major component being the co-ordinate consistent decoder. The experiments are visually simplistic and it is not obvious if the system will scale to more complex settings. A few comments:
1. A major limitation of the approach is the assumption that the equations governing the system are known. This makes it harder to generalise the system to novel tasks and more complicated settings with contact and object interactions. A potential way to overcome this could be to use ideas from prior work such as Interaction Networks where the dynamics are modelled as unary and binary interactions. While such dynamics models are black-box and not as interpretable compared to the current approach, they can easily generalise to novel tasks. Additionally, using positions and velocities as the latent state representation together with IN style transition models can be a sensible middle ground.
2. It would be great if there is an additional ablation experiment where the known equations of motion are replaced with a black-box neural network (while still retaining the position and velocity representation). This can quantify the effect of known dynamics and make the contributions of the paper (with regards to the decoder) more clear.
3. An alternative way of generating consistent object positions from the encoder is to compute a mask-weighted average of the image co-ordinates. This can be a nice way of adding additional structure to the networks that can regularise training.
4. The approach uses a 3-layer MLP for generating velocity estimates — could this not be done via finite differencing? (e.g. higher-order backward differencing)
5. Both the content and mask vectors are learnable but fixed for the entire task — i.e. it is not a function of the input image. This makes the approach not applicable to novel objects or even objects with minor color changes. 
6. It is not clear how the translations, rotations and scale parameters for the Spatial Transformer are estimated. I presume that the positions and orientations predicted by either the encoder or physics simulator are directly used. This needs to be clarified in the main paper. 
7. The paper mentions that the background masks are known when localising the objects via the encoder. If this is the case, the localisation problem becomes somewhat trivial. This should be clarified.
8. There needs to be a clear discussion on the limitations of the current approach — it does not scale to novel objects, needs to know the number of objects apriori and has not been shown to estimate object properties such as mass, friction etc. 

Overall, the approach presents promising initial results towards an unsupervised method for modelling dynamical systems from video. There are several limitations that need to be made explicit and some additional experiments on more complicated systems and a few ablation studies can significantly improve the strength of the paper. I would suggest a borderline accept.

Typos:
1. Section 4.1, Setup: 5 values of (K, t_pred, t_ext) are given, need only 4
2. “K” is not introduced till the results section

Additional citation:
The following paper on learning physically consistent position and velocity representations for dynamical systems (and for use in visual MPC settings) should be cited:
Jonschkowski, Rico, et al. "Pves: Position-velocity encoders for unsupervised learning of structured state representations." arXiv preprint arXiv:1705.09805 (2017).
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper proposes to integrate model-based physical simulation and data-driven (deep) learning. In a nutshell, one deep network predicts the state variables of the physics simulation (such as objet location, shape and velocity) from an image. A second network does the inverse task, to render images given the state variables (and a background image). In this way, one can go from a video frame to a physical system state, modify the state with physics simulation, and then go back from the modified state to a video frame. Together with a differentiable physics engine, through which one can back-propagate, this makes it possible to use the un-annotated video itself as supervision. At the same time, the two neural networks can be seen as an auto-encoder, in which the latent state is explicitly constrained to correspond to the desired physical state variables.

The topic of the paper is hot: a proper integration of physical models with data-driven deep learning is, arguably, one of the big short- to mid-term themes of machine learning research. The way it is done in the present paper intuitively makes sense. The approach is fairly obvious at the conceptual level; but in the details poses a number of technical challenges especially for the decoder, which are nicely analysed and resolved.

Some minor design choices are not well justified and at first sight appear a bit l'art pour l'art. While it is a sensible, pragmatic choice to first predict object masks, then extract their location ands and velocities in a second step; I do not quite see why one would have to do the latter with neural networks. it would seem that once the masks have been found, their location can be chosen as something like the (perfectly differentiable) mask-weighted centroid and does not need a multi-layer network; and similarly that deriving velocity from locations in adjacent frames can be hard-coded and does not need a 3-layer network.

The experiments are still at an early "toy" level, with synthetic videos where high-contrast, homogeneous objects move in front of a uniform or blurry background. The baselines are sensible and ablation studies are done with care. Still, it would have been nice to also run the method on some real video. To my understanding, this would be easily possible at least for future frame prediction, all one has to do is either annotate the objects in the target frame or measure success by comparing the predicted and true frames at the image level. It is also not clear whether the videos were synthesised with the same physics engine also use inside the system - which would be slightly questionable, in the sense that the learnable pipeline is then a-priori matched to the biases in the data.

One comment on the presentation: while the paper is generally well-written and easy to follow, the wording could at times be more careful. There is a slight tendency to identify the particular (simple) physical systems of the paper with physics as a whole. E.g., not all physics simulation must have objects - for instance, fluid dynamics or radiative transfer do not have individual objects, but are nevertheless relevant in the context of visual data. Similarly, even for defined objects, position and velocity are not always a sufficient state, for instance objects might deform, or have different elastic properties when colliding.

Overall, I find the work interesting and well-executed. It is a natural step to take towards the important goal of integrated data-driven and physical models, including the associated theme of self-supervision via physical constraints. On the negative side the paper does make a slightly rushed and unfinished impression by not showing any, even qualitative, experiments on real video. Most people - rightly - use simple toy-like datasets for development and analysis. But showing only those gives me the impression that the paper was written too early, just to be the first and to make the deadline. Or that moving to real video poses a much greater challenge than expected - but then this should be stated and discussed.

review_title: Official Blind Review #3

Meta-review: The submission presents an approach to estimating physical parameters from video. The approach is sensible and is presented fairly well. The main criticism is that the approach is only demonstrated in simplistic "toy" settings. Nevertheless, the reviewers recommend (weakly) accepting the paper and the AC concurs.
Decision: Accept (Poster)