Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

This paper introduces a method for predicting future trajectories of sets of objects.
They generate features for each object in the scene using a CNN followed by ROIPooling for each object.
They employ Interaction Networks to implement an update function that uses the relations between the object states.
Because they use spatial features for each object, they update the original Interaction Networks to take spatial blocks as opposed to vectors with convolutions instead of mlp's.
Their models output predicted bounding boxes, but can also be trained to output a mask (21x21) for each predicted object bounding box.

Reason for Score:

I vote for accepting.
The method is clear, and the results are good.
I would be interested in applications of this method to real world video using off the shelf object detectors.

Pros:

ROIPooled object features from CNN outputs have been demonstrated to be packed with information about both the objects and their context.
The enhancement to the original Interaction Networks (using convolutions) seems to bring performance improvements (Table 1).
The videos (especially for PYHRE dataset) are impressive.
The benefits of being able to make long term predictions when planning are clear.

Cons:

I am a little worried about the size of the input images (eg. 128x128 for PYHRE ) and the size of the spatial output of the CNN that is used for ROIPooling of the objects. I cannot see this size in the paper (should be in A.1?). It seems it might be too small to properly use ROIPooling that discriminates between close objects.
I am not sure the baseline comparisons are very fair, since it is stated that they are either video based or do not have mechanism to cope with "background" objects.

 


review_title: A tidy model, with impressive results

final_score: rating: 7: Good paper, accept
review_content: The authors propose a novel architecture RPIN (region proposal interacion networks) to learn physical interaction dynamics directly from visual input using region of interest features and a convolutional interaction module which retains spatial information. The authors showcase improvements in 4 different datasets for long term prediction, planning and generalization.


Strengths:
* Simple and effective idea to encode spatial information and use environment information via convolutional and RoI features respectively.
* Improvements on RealB and Phyre-C seem to be very promising (esp in the longer than training horizon setting).


Weaknesses:
* Evaluation and insight into how changing feature representation for interaction networks impacts long term prediction seems to be missing from the paper.

Questions: 

1. How does Fp (equation 2) work for scenes with different number of objects as your number of channels would change with changing number of objects in a scene.


2. The authors mention that "In SimB, the improvement is not as significant because there is no environment information needed to infer future dynamics". This doesn't seem to be true as you need scene context to perform prediction even in the simulated environment?
review_title: Simple, effective method to incorporate spatial information in learning visual dynamics

final_score: rating: 6: Marginally above acceptance threshold
review_content: *Summary*: The paper presents a DNN for learning dynamical models using the SOTA in deep learning-based computer vision literature.

Strengths:
1. The main strength of the paper is that it uses the SOTA in computer vision (DNNs for image and object understanding) to achieve a different goal in an area in which research has not yet matured (at least to my understanding). By doing so, it helps stimulate and ease further research in the latter area .

2. There are clear experiments that effectively demonstrate the contribution of the paper.

Weaknesses:

1. I would like to have seen some qualitative comparisons with the baselines to get a better sense of the superiority of the method.

2. The technical novelty is low as it seems the main technical contribution is to only replace the MLP from related work with a ConvNet


Comments/Questions to authors:
1. In the introduction, you write *"... those methods operate in the state space, or ignore the environment information"*. Could you elaborate a bit on this point. Folks outside the field may not understand what it means to "operate in the state space" and why it that not good ?


review_title: Leaning towards accept, but low technical novelty

final_score: rating: 7: Good paper, accept
review_content: Paper Summary
-----------------------------------
The paper proposes a variation of interaction networks (IN) called region proposal interaction networks. The key idea is to have a richer object-centric feature representation using ROI-Pooling to encode the objects for prediction and use convolution operators to help the IN handle the change in the dimensionality of the feature representation. The paper is well-written and is evaluated on several popular benchmarks. The proposed variations seem to have a considerable effect on the performance to offer significant gains on challenging benchmarks.

Comments:
- The predictions for time t+1 is a function of features from t-k to t. What is the value of 'k' used in the experiments? How is this chosen? Does this change across datasets, environments, tasks, etc.? k has a direct effect on the number of parameters in the network W_p and it would be interesting to see how it affects the predictions.
- On that note, are the object features re-computed using ROI pooling at every time step, or are the predicted features used recursively? Have you tried generating frames based on the predictions to see if it helps capture the interaction/context between the different objects in the scene based on predictions? 
- How are the objects registered across time? In other words, is there some tracking used to map the object bounding boxes across time to ensure that consistency is enforced when modeling the interaction of the same object over time? If there is no explicit registration, then is the interaction modeled on a per-frame basis i.e. ground truth is used to ensure that the features of each object are used for its interaction modeling? If the GT is used, then I am not sure how would the model fare without access to such explicit information when encountering unseen objects?
- I am not convinced about the use of the term 'spatial information' when using ROI pooling. From what I can see, ROI pooling results in a 3-d representation of the object as opposed to 1-d representations obtained when using cropped images. This offers a richer representation that captures object-centric features that is invariant to the spatial dimensions of objects. But I do not think that the spatial information is explicitly coded into the features are any point. 
- While the paper is well written, there are some inconsistencies in notations that are hard to follow. For example, in Equation 2, the parameters to the function f_Z has 'h' which in the writing refers to the height of the object-centric representation vector. Following consistent notations will greatly improve the readability.
- The paper would benefit from a more thorough ablation study. There are several factors that are used in the proposed framework that could have a considerable effect on the performance but not discussed such as the temporal parameter 'k' mentioned above.
review_title: Promising approach with object centric representations

Meta-review: This paper was reviewed by four experts in the field. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance to ICLR 2021. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes and include the missing references.
Decision: Accept (Poster)