Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: UPDATE: I appreciate the authors' discussions and qualitative results. My main original concern was that the empirical evaluation only studies a single type of situation of inferring physical parameters. Given that the authors claim that the proposed method infers "on the fly" physical properties, I would expect that the authors demonstrate the the method works on other physical properties as well beyond just one setting. Because this concern was not sufficiently addressed, I maintain my original rating.

----
Summary: This paper tackle the question of learning to infer physical parameters of a novel physical scenario with which to predict future motion of objects. In particular, the authors propose a meta-learning framework for inferring parameters of physical objects from a few video observations of physical scenario, with the constraint that physical parameters are not apparent from appearance alone. The authors evaluate the approach on a domain where the learner needs to infer whether balls should pass above, pass below, or bounce off obstacles.

Recommendation: Borderline. The main limitation of the paper is that the empirical evaluation only studies a single type of situation of inferring physical parameters (whether balls should pass above, pass below, or bounce off obstacles) but does not consider other situations for inferring physical parameters for prediction in a new domain. This paper has great potential, but the lack of evaluation on a wider variety of physical phenomena makes it difficult to evaluate the generality of the claims made by this paper. I would highly consider increasing my score if the authors would be able to provide a thorough evaluation of two more types of physical phenomena.

Research Problem: The research problem this paper tackles is to learn a model of intuitive physics that can learn "on the fly" physical properties specific to new environments.

Approach: A meta-network compresses a set of video observations of a scenario using the dynamic image encoding. The meta-network produces context parameters m and a that parameterize the scenario-specific predictor \hat{Phi} and conditional generator g respectively. The state x is represented with a two-dimensional array. The authors use a perceptual loss as an image reconstruction metric. The author also enforce that the predictions of the state-space predictor and the encoder eta should provide close predictions.

Strengths
- The meta-learning formalization is an interesting and intuitive contribution.
- The experiments provide a proof-of-concept of the efficacy of the framework with thorough analysis.

Weaknesses
- While the work is well-motivated and the experiments provided show a proof-of-concept of the approach with thorough analysis, the paper could be significantly strengthened by considering more domains for inferring physical parameters - simply inferring whether balls will pass above, pass below, or bounce off obstacles is a good first step, but does not provide enough evidence to evaluate the generality of the proposed method to intuitive physics tasks. This I believe is the main limitation of the paper. 

Questions
- Would the authors be able to provide an empirical study (with qualitative analysis) or theoretical justification to explain the reason why the method can generalize to scenarios with more objects than observed during training? Prior work that does show generalization to more objects explicitly build in the locality constraint (e.g. van Steenkiste 2018) through the network architecture, so it is interesting to see that the proposed method also can generalize similarly.

Van Steenkiste, S., Chang, M., Greff, K., & Schmidhuber, J. (2018). Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint arXiv:1802.10353.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: 
Summary 
The paper proposes an architecture for few-shot video prediction in which a number of videos are summarized through global pooling operations and passed into a video predictor that learns to leverage them for adaptation, similar in spirit to the RNN-based meta-learning approaches such as Santoro’16, Duan’16. Due to global image and feature pooling operations, the proposed approach is computationally efficient. Proof-of-concept experiments are presented in a simple simulated physical prediction setting. It is claimed that the proposed model achieves generalization to longer sequences and larger board sizes, as well as a larger number of objects.

Decision
The work considers an interesting task and shows promise, but the paper lacks a substantial contribution at this time. I recommend weak reject. To improve the paper, I recommend considering harder prediction tasks and lifting some of the restrictions of the method.

Pros
The paper proposes and evaluates a model for few-shot video prediction, which is an interesting and well-motivated task. 

Cons
1. The method is only evaluated on a toy task. While a model is proposed that can successfully solve the presented toy task, certain assumptions, such as averaging directly in the image space, may prevent the findings of this paper to scale to more complex tasks, which demands further investigation.

2. It is claimed that the method generalizes to longer sequences and larger board sizes, however, the performance of the method seems to quickly deteriorate in both settings. Authors present anecdotal evidence that the method generalizes to larger number of objects, however, quantitatively the results again are much inferior to the method trained on a larger number of objects. While the presented qualitative generalization findings are interesting, they seem limited in scope, especially when taking into account the simplicity of the considered data.

Minor comments
1. On page 3, the paper states “We are not the first to consider a similar learning problem, although most prior works do require some form of external supervision.” While there is a substantial body of literature that uses extra supervision, the following foundational works do not: Finn et al, 2016a,b; Fragkiadaki, 2016. In addition, “a similar learning problem” is considered without supervision in many papers on video prediction, including: Ranzato et al., 2014; Srivastava et al., 2015; Mathieu et al., 2015; Denton et al., 2017, 2018; Villegas et al., 2017, 2019; Wichers et al., 2018; Castrejon et al., 2019. While it might not be necessary to cite all of these as they do not necessarily apply their models to intuitive physics, the statement in the paper seems rather unsubstantiated.
2. A similar meta-learning prediction problem is addressed in Nagabandi’19a,b, which are not cited.
3. The paper is somewhat cluttered with notation, which overall hampers the flow for the reader. The paper would benefit from slight reorganization to improve the flow. This could be done e.g. be structuring the paper to first present the computational task that it addresses, namely few-shot video prediction, and how each component of the model works to address that problem, and only later go into the detail of how the data were collected and how each individual module was implemented


Santoro et al, Meta-learning with memory-augmented neural networks.
Duan et al, RL2: Fast reinforcement learning via slow reinforcement learning.
Nagabandi et al, LEARNING TO ADAPT IN DYNAMIC, REAL-WORLD ENVIRONMENTS THROUGH META-REINFORCEMENT LEARNING,
Nagabandi et al, DEEP ONLINE LEARNING VIA META-LEARNING: CONTINUAL ADAPTATION FOR MODEL-BASED RL


--------------------- Update 11.19 -----------------------

I appreciate the newly provided qualitative generalization results (in response to R3). However, it is clear that the network has trouble maintaining the exact number of spheres in the scene, especially after collisions or interactions with obstacles. While this generalization finding is undoubtedly interesting, I am not convinced it is enough for a publication in ICLR.

Furthermore, to support the authors' claim that the prediction performance on long sequences is satisfactory, I suggest comparison with a hand-crafted baseline, such as one suggested by R1, or one that makes use of explicit physics.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: 
Paper summary:

The paper proposes a method to predict the future trajectory of a ball (or a set of balls) given the first few frames of the trajectory and a set of experience runs in the same environment. The model first learns to convert the set of images to a set of corresponding heatmaps that encode the location of the ball. Then, a recurrent network generates the future states given the first few locations and the output of a network that learns abstract information from the experience runs. The network is trained using reconstruction and perceptual similarity loss.

Paper strengths:

It is interesting that the model is able to learn from a set of experience runs. Learning abstractions from a set of experience runs is an interesting direction.

Paper weaknesses:

The paper has two main problems. (1) The model and the experiments are designed for a very simplistic scenario. I do not think that machinery is really needed to perform prediction in this simple setting. (2) The paper needs major re-rewriting. Some of the main parts of the paper are not clear. Please refer to the comments below for more details.

- It is unclear how the initialization function (Equation 4) is learned. It is very unlikely that the auto-encoder produces a heatmap of the object locations only based on the reconstruction loss that it receives at the end. This should be clarified.

- Equation 2 is very confusing. Is \phi-hat a function of experience I(E) or I_{0:T}(R). It seems \phi-hat produces an image that is compared with I(R), but it doesn't seem that M produces an image.

- The proposed machinery is overkill for the very simple experimental setup. A nearest neighbor baseline would probably perform as well. We can extract a set of frames in the experience runs that are closest to the first few frames using some simple distance functions. Then, we can predict the future movements by copying the rest of the trajectory from the experience run.

- The only part that seems unsupervised is the location of the ball, which can be easily obtained by image subtraction in this simple setting. What else is unsupervised in the proposed approach?

- The beginning of section 3.2 mentions w, but there is no mention of it afterwards.

- Why does the backprop for sixty 64x64 images require 12Gb of memory? It would be good to explain that.

- Regarding the evaluation, how does it know which prediction corresponds to the which groundtruth to compute the distance that is mentioned? The prediction can correspond to any of the balls since there is no explicit notion of object in the model.

Due to the issues mentioned above, especially extremely simple experimental setup and lack of clarity, I chose "Weak Reject". 


review_title: Official Blind Review #1

Meta-review: While the reviewers found the paper interesting, all the reviewers raised concerns about the fairly simple experimental settings, which makes it hard to appreciate the strengths of the proposed method. During rebuttal phase, the reviewers still felt this weakness was not sufficiently addressed.
Decision: Reject