Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: * Note: emergency review, done under a shorter time frame than good reviews require.

In this paper, the authors develop a highly structured model to predict motions of objects defined by segmentation masks and depths. The model trains a physics model (in the form of a slightly modified interaction network) and a renderer composed of a per-object renderer combined with an occlusion model which composes the per-object segmentation and depth into a scene segmentation and depth.

Positives:
- The jury is still out on the degree of structure required to do proper object processing (neural nets with large amounts of data, mildly structured nets like networks with attention, more structured nets like this, or a full fledged renderer-like probabilistic program); this work contributes novel work to the line of research which attempts to do object-level processing with structured models while still leveraging the power of neural networks.

- The experimental section appears very thorough and convincing, even if the dataset is relatively simple.

Negatives:
- The model requires highly privileged information (segmentation mask, depth) at training and test time. Given that the segmentation/depth data are not too far from the actual images, it would have been interesting to see if it were possible to work with pixels (a variant of the occlusion model would probably still work), at least at test time. 

- Regarding using segmentation/depth as input to the model: for a real dataset, segmentation is more relevant: it is both less informative than positions (due to significant occlusions) and easier to measure. In this highly synthetic dataset, this feels more debatable: objects are more entangled in the segmentation (which makes using segmentation more challenging), but only weakly, with many frames with no occlusion; furthermore, segmentation provides object shapes as information.

- The paper is generally well written, but could benefit from some reorganization - instead of defining each module separately, it would be better to describe the flow of information through different modules, then describe the module. I was wondering for a while how the initial positions were estimated (required as input to both the interaction net and the renderer), but this only comes at the end of the paper.

- Some ablation experiments felt missing, for instance, the importance of the refinement network (also unfortunate that the details of refinement were not given in the main body).

- The stochasticity of the interaction network appears a bit weak (simple Gaussians) - it would be interesting to display some data to see if the ground truth data is indeed Gaussian like .

- Missing potential references:
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
Learning to Decompose and Disentangle Representations for Video Prediction
MONet: Unsupervised Scene Decomposition and Representation
review_title: Official Blind Review #6

final_score: rating: 3: Weak Reject
review_content: The key contribution of this paper is a model that can predict the dynamics of pre-segmented image patches under multiple frames of occlusion. The input image is processed by a CNN, the dynamics are predicted by a recurrent interaction net, and the output image is generated by a (deconv) CNN. 

The key weaknesses I see are:

- The objects must be pre-segmented by some externally defined mechanism. Where does this mechanism come from? Segmenting the objects is challenging, and there are various recent methods that explore how to learn to do this (van Steenkiste et al., 2018). But if one has the segmentation masks, that simplifies things considerably and also offers a good estimate of the location and velocity (if there are 2+ frames). 

- During training, the error is computed on all frames, including occluded ones, and backpropagated into the weights. But if I understand this correctly, this means that for training you need access to ground truth rendered trajectories. It would be better if the model didn't require the ground truth segmentations for objects that are occluded. How would they be made available to a learning system?

- Generally the writing wasn't that clear and I struggled to understand some details of the model and training procedure.


Overall I don't believe this work is ready for publication, as there isn't that much novelty and the requirements are impractical.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a method to predict future trajectories by modeling partial and full occlusions. Although it is well-written and the topic sounds interesting, I failed to catch why this approach is required for this setting. So, to strengthen the message of this paper, I listed a couple of suggestions and comments below (from the most important to the least important):


1. It is a bit hard to catch how this model handles "diversity." Specifically, when predicting the futures, it should be able to generate stochastic outputs. However, I failed to find how diverse the output of the model is. If the output is not that stochastic, then it would be tough to believe that the model can "predict" the future; instead, it may "extrapolate" the current condition only. To reassure such concerns, I recommend reporting how diverse your output is. (One easy way is to report the variance of the predicted center mass values between multiple samples while reporting the l2 distance.)


2. For the future prediction task, it would be much better if it is compared with various state-of-the-art future prediction approaches [1, 2, 3, 4, 5, 6]. For some of the models, it could not be able to compare directly with this approach (e.g., lack of 'center of mass' information). However, it would be still okay once it is compared with other state-of-the-art results without feeding some 3D information (e.g., provide projected 2D video as an input). By doing so, I believe the readers can easily catch (1) why it is better to predict physical interaction in 3D space (instead of directly predicting from a 2D space), and (2) also why predicting occlusion is essential in this problem setting.


3. Minor comments:
(a) It is a bit hard to catch how the author computes the "aggregate pixel reconstruction error" in Table S1. I recommend adding an equation number there to make it clear.

(b) There are a couple of missing references: the last sentence on page 4, the first paragraph in Supplementary, the last sentence in Supplementary page 3, etc.

(c) \citep is often misused. Please replace some inappropriate \citep with \citet.

(d) Please check the format of the reference, as well; currently, it has various styles even for the same source/conference.




------------------------------------------------------------------

[Some comments based on the authors' rebuttal]


I thank the authors for their thorough comments and detailed explanations for each question. I carefully read the whole (not just my part), but it didn't change my mind; it would be much better if the claim comes with a more directly comparable result.

Some additional comments:
Q1-comment) I think the limitation of "learning to extrapolate"-style video prediction approach is partially presented in Reviewer #2's claim as well. Therefore, in this context, I recommend the author to show a better result to reassure the reader's concern. 

Q2-comment) I at least strongly recommend to add more experiments with other baselines, rather than relying mainly on the original model of the dataset. Although the input condition of a model could be different, I at least do believe that it will help the readers to catch the benefit of your setting more clearly.

I hope this review phase would make your paper more powerful. 




[1] Liang et al., Dual Motion GAN for Future-Flow Embedded Video Prediction, in ICCV, 2017
[2] Denton and Fergus, Stochastic Video Generation with a Learned Prior, in ICML, 2018
[3] Wichers et al., Hierarchical Long-term Video Prediction without Supervision, in ICML, 2018
[4] Wang et al., Video-to-Video Synthesis, in NeurIPS, 2018
[5] Heish et al., Learning to Decompose and Disentangle Representations for Video Prediction, in NeurIPS, 2018
[6] Minderer et al., Unsupervised Learning of Object Structure and Dynamics from Videos, in NeurIPS, 2019
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: Summary:
This paper proposes a method that combines a recurrent neural network that predicts values that are used as inputs to a rendered which interprets them and generates an object shape map and a depth map for every step of the dynamics predicted by the recurrent neural network. The proposed method is able to handle object occlusions and interactions. In experiments, the authors show improved performance against baselines for future prediction, object tracking, and object permanence.


Pros:
+ Rendering network used with RNN
+ Outperforms chosen baselines

Weaknesses / comments:
- Compositional Rendering Network has to be pretrained: Did the authors try to train the model end-to-end? It would be interesting to see if this can be done so the proposed network is more unified.

- Figure 3 is not self explanatory: It would be good if the authors add labels to the predicted and gt frames. It is not easy to parse this figure from just looking at it.

- Difference from Battaglia et al., 2016: It seems that the only difference between the proposed method and this baseline is the change of input/outputs (including output with variance), and training in full sequence (RNN)? This looks like a minor change to me and reduces the novelty of the proposed method.

- Table 1 (trained on ground-truth positions): The authors claim that their network performs similar to Battaglia et al., 2016, but it seems that the baseline is better than the proposed method for the short term predictions with a relative improvement of about 20% and for long term when the baseline is better (half of the tests)  it’s by a relative improvement of about 10%. Can the authors comment on this? Am I missing something?

- Implausibility score: What do the authors mean by “the maximum error through the whole sequence”? How is this defined?

- The authors compare with Riochet et al., 2018 in Table 4, but not in the rest of the evaluations. Can the authors comment on why this is the case? 


Conclusion:
The paper proposes an interesting method, dataset, and seems to perform baselines in the quantitative evaluation. To the best of my knowledge, the current state of the method is novel in the rendering network. However, the rest of components have limited novelty. In addition, I have some comments about the paper which stated above.

review_title: Official Blind Review #2

Meta-review: The paper studies the problem of modeling inter-object dynamics with occlusions. It provides proof-of-concept demonstrations on toy 3d scenes that occlusions can be handled by structured representations using object-level segmentation masks and depth information. However, the technical novelty is not high and the requirement of such structured information seems impractical real-world applications which thus limits the significance of the proposed method.
Decision: Reject