Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a method which takes as input sequences of video frames of scenes from which it is able to understand the dynamics of objects present in the scene and their interactions and transfer this to several downstream tasks. The model consists of two modules, one to distill individual object dynamics and a second relation module to understand interactions between objects.  The model is shown to have SoTA results on several downstream tasks including video understanding and reasoning, video prediction, reconstruction, and segmentation. The representations learnt by the model are shown improve upon prior work especially on tasks which require knowing object dynamics such as predicting the next frame in a sequence where collisions or occlusions can occur. 
main_review: # Strengths

### Novelty 
- The two key contributions are clearly described:
    - use explicit dynamic features learnt by matching objects’ latents between the current and previous frames as opposed to only learning static scene properties as done in prior work. 
    - explicitly model interactions between objects

### Experimental verification 
- Small ablation on the effect of dynamic features in Table 2 
	 - Even if dynamics features are absent the model achieves better results than IODINE which means that ODDN can both model object dynamics and generates overall higher quality object static representations. 

- Gives SoTa improvement across a variety of tasks
    - Table 1 and Table 2 show a consistently significant improvement over prior work
    -  ODDN Model representations are fed into ALOE instead of using static Monet features which ALOE originally used. This ODDN-ALOE model improves on state of the art on CLEVER for various tasks 
    - Interestingly the gains are greatest on predictive and counterfactual questions which are related to predicting how objects will move.
    -  ODDN also improves over using object representations from IODINE or PROVIDE (in Table 2). 
    - Qualitatively, in Fig 4, the generated images for the next frames are better for ODDN as it is able to preserve the color of the objects as well as position whilst PROVIDE fails to preserve color after a few frames
    - ODDN also does better at reconstructing images and generates more compact  segmentation masks than IODINE or PROVIDE (in Fig 2).

# Weaknesses

### Clarity of explanations and figures could be greatly improved
- Although the ideas are clear, the method and implementation could be made clearer e.g. write out the algorithm in pseudo code 
    - In addition, the notation is slightly confusing given there are two meanings for t - inference model time step and time step in the video 
- Fig 7 (c) + section titled “Disentanglement” is not clearly explained
    - The caption is incorrect - (b) is actually referring to (c). (a) is not explained.
    - What is a dynamic dimension?
    - You could add some arrows to the diagram to explain what is happening across the rows/columns. It’s not clear how to interpret the images.
    - In addition where does “velocity magnitude” come from? This was not explained previously. 
- Figure 4 is also not very clear 
    - Explain in the caption what is happening in the 6 frames i.e. blue and brown cylinder collide, with ODNN this interaction is predicted but with PROVIDE it loses information about the blue cyclinder. It would be useful to explicitly mention the colors to look for in the images in the caption or in Section 4 in the text.
    - The images are also really small - hard to see what is happening 
- Figure 3 
    - Is it showing MSE from 6 frames in total or averaged over some number of runs?
- Table 3
    - For each metric add an arrow to indicate if lower or higher is better 
- Figure 6 is also not clear
    - The caption indicating the colors for the frames and timestamps is really tiny 

### Lack of implementation details
- No details about how the model was implemented or how to reproduce the results
- Equation 2 and 3 explain what IODINE does but the ODDN method is not fully explained. 

### Typos:
- Section 4.3: “weather” -> “whether”

summary_of_the_review: The paper presents a method with two key insights/contributions that are not present in prior work. The ideas are validated quantitatively and qualitatively. The technical details, figures and tables could be improved with better explanations and descriptions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an unsupervised distillation of disentangled object dynamics from raw video inputs. The distilled dynamics model is capable of causal reasoning and future frame prediction. Extensive results on tasks of segmentation and reconstruction show its favorable performance.
main_review: [Strengths]
The authors provide a novel distillation method to understand object dynamics. The proposed system shows state-of-the-art performance on CLEVRER.

[Weaknesses]
- I would like to see more qualitative results on the attention module. Visualized coefficients in Fig. 6 only cover one single scenario. I think the paper needs more validations on whether the attention module converges well.

- The proposed method is validated only in CLEVRER, which does not show its generalization capability. In addition, it is hard to find the differences in the qualitative results (Fig. 4) compared to the previous methods. In Fig. 5, IODINE cannot represent the shadow areas, while the proposed ODDN can reconstruct them. Could you elaborate on the reason how ODDN has a representation ability on the shadow areas?

- I would like to see the comparison of the computational complexity with the previous methods, e.g., ALOE, IODINE, etc.

- Please discuss the limitations of the proposed method. It is required to discuss whether there are any other aspects that can be further improved (as future works) for this task.

summary_of_the_review: Overall, the proposed architecture is designed simple and shows state-of-the-art performance. There are minor issues as commented in [Main Review], but I am leaning towards positive at this moment. Since the topic is out of my scope, I would like to see other reviewers' opinions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a framework that distillates explicit object dynamics (e.g., velocity) from the discrepancies between objects’ static latents of different frames, called the Object Dynamics Distillation Network (ODDN). The approach is built upon recent works which decompose static scenes into independent latents by randomly assigning different objects to a fixed number of slots, which share weights to obtain latents with a common format. And this paper makes use of the attention mechanism of the transformer to align objects’ latents of two input images and encode the aggregated representation into a low-dimensional vector to obtain disentangled object dynamics. Afterward, the paper builds a relation module to model object interactions based on NPE and R-NEM.
main_review: Strengths:

- The paper presents a very interesting idea of using transformer structure to align latent features and distillate object dynamics, and it seems very effective.

- Experimental results show superior performance compared to the SOTA on video understanding and reasoning.

Weaknesses:

- The paper is built upon many existing techniques, including the IODINE, NPE, and R-NEM. Especially, the core part of it is the use of IODINE latents and aligning them by a transformer structure. The reasoning module is a combination of NPE and R-NEM.

summary_of_the_review: Though the paper is built upon many existing techniques, including the IODINE, NPE and R-NEM, and the novel part of this paper is using a transformer structure to align the IODINE latents and find discrepancies, I still consider this paper very interesting and strong. It may provide some insight on how to model individual object dynamics from videos. Moreover, the proposed method is very effective as experimental results show superior performance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a method of extracting the latent space that describes objects in a video. The encoder part, Inference model, encodes an input image frame-by-frame from a video based on IODINE. The relative information between the latent vectors of three consequent frames is extracted and updated based on Transformer. The input image is reconstructed from the latent vectors by the decoder part, Generative model. The proposed method is tested for various tasks on CLEVRER dataset. 

main_review: The proposed method is compared to the previous methods, IODINE and PROVIDE, and the better results are shown. But, the problem is that the detail of the method is not described and difficult to understand.

In th experiment of video understanding, the input for IODINE is a single frame, but that for the proposed method is multiple frame from video. How different the input data is? If the information extracted by the proposed method is the dynamics of a scene, the importance will depends on the scene. But, no example is shown in the paper to confirm the effectiveness of Dynamic Distillation. Although the method, PROVIDE, is chosen as a method that introduces temporal information, it may not be meaningful because the performance is worse than the base method, IODINE, which uses a single image.

In the experiment of prediction, it is difficult to recognize how is the motion of objects and interaction each other in Fig.4. Is the example appropriate to show if the proposed method can extract the dynamics from video?

In decomposition, what is the condition of the experiment? Is the reconstruction through the autoencoder, or reconstruction from the latent vectors given by user? If in the latter case, what and how are the input latent vectors given?

In the explanation of the proposed method, the indexing of the latent vector is ambiguous. A latent vector is indexed by the frame number i, latent index k, time t and the pixel coordinate. But, what is the difference between the frame number and the time? And the indexes are omitted in many parts of the explanation. It is difficult to understand which latent variables are considered to find the relationship.

The detail of Generative model is not explained. The color is estimated for each pixel. But, the relationship between a latent vector and a pixel is not explained.

summary_of_the_review: Although the proposed method seems to show the better results in the experiments, the explanation is not sufficient to convince if the experiment is appropriate and fair. Additionally, the explanation of the method is not clear to understand the detail. Additionally, it is not clear that what kind of information is extracted by the part, Dynamic Distillation, introduced in this paper. It seems no experiment is shown that object dynamics is distilled by the proposed method. Consequently, the reviewer thinks that this paper is below the acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at learning object dynamics from unlabelled videos for multi-object representations. This work builds upon the inference model from Greff et al. (2019) to obtain the multi-object representations. Using that, this paper learns latent dynamics by predicting the future frame given two previous observations. The proposed network is trained by maximizing the log-likelihood in the pixel space for reconstructions as well as for predictions, similar to standard work in video prediction. In addition to that, this paper optimizes the objective from Greff et al. (2019) to obtain multi-object representations. The main contribution of this work lies in the module which predicts the future multi-object representation given the two past ones by using i) dynamic distillation as well as ii) a relation module. 
The dynamic distillation module is motivated by the claim that the object-to-slot assignment switches inside videos. Therefore, the paper introduces a self-attention layer to match the slots of the specific objects across two frames. The output of this module after applying the feed-forward network is called dynamic representation. \
The relation module is proposed to model the interactions between object pairs. This is implemented (10-13) using self-attention between the dynamics (from above) and static representation of objects i and k (slots). The output is added to update the dynamic representation from the dynamic distillation module. Ultimately, the future state is predicted using a linear transformation of the static representation of time t and the updated dynamic representation. \
The paper evaluates their learned representation on the CLEVRER dataset for video reasoning, prediction, reconstruction, and segmentation.
main_review: Strength:
+ The introduced modules, dynamic distillation and relation modules, are well motivated and their design and realization seem reasonable.
+ The paper reports strong numbers on the task of video reasoning outperforming existing works on CLEVRER.

Weaknesses:
- Missing ablation experiment for dynamic distillation and relation module. The paper claims that the slot assignment switches unpredictably inside the video. It proposed the dynamic distillation attention mechanism to solve that. Yet, it is not evaluated if this module actually solves this problem. With that, the reader does not know if this module is effective. The same for the relation module. The paper should report numbers with and without the use of this module to assess its impact. 
- Missing baseline. To assess the significance of the proposed modules (dynamic distillation + relation module), the reader also needs results without them. This baseline can be created by replacing the proposed modules with a feed forward neural network (with roughly the same number of parameters as the proposed modules) that predicts the next static representation (t+1) based on the concatenation of the (static) representation at time t and t-1. The paper introduces some kind of baseline in Table 2 (ODDN w/o dyn) by reducing the 4 dimension dynamics. Thus, the network is left to predict the next state solely based on the current static representation (at time t) which is insufficient. That is why I urge the authors to implement the baseline as pointed out above and to report the resulting numbers in the rebuttal.
- The paper compares their method to PROVIDE which is the only other competitor that leverages temporal information. PROVIDE extends upon the static method of IODINE using a 2D-LSTM. The reported numbers in table 2 for PROVIDE are significantly (~20%) worse than for IODINE which does not make sense to me, especially when looking at the PROVIDE paper in which PROVIDE is on par with IODINE (on average). The same holds true for table 3. The MSE reported in PROVIDE is significantly better than IODINE whereas in this paper MSE is significantly worse (48% for MSE). This suggests to me that there is something wrong with how PROVIDE was trained/evaluated. The paper discusses a reason why ODDN is superior compared to PROVIDE in 4.3 but does not address why there is such a huge gap between IODINE and PROVIDE. Could the authors please clarify this for me? This shows and enhances the need for a baseline as discussed above.
- Missing implementation details. The paper does not provide any implementation details like e.g. optimizer, learning rate, \beta value (hyperparameters, etc). This makes the paper not reproducible and I urge the authors to add these details. Does the code get released after acceptance?
- The paper evaluates their proposed model only on the CLEVRER dataset. To better assess the impact of this work the method needs to be evaluated at least on one other dataset such as e.g. CATER.
CATER: Rohit Girdhar et al. CATER: A diagnostic dataset for Compositional Actions and Temporal Reasoning. ICLR 2020
-Missing related work. This paper misses a relevant related work from Veerapaneni et al. “Entity Abstraction in Visual Model-Based Reinforcement Learning” (CoRL 2019). This work extracts entity representations from images and learns their dynamics for model-based reinforcement learning.  Moreover, this paper only cites related work which operates on multi-object representations. Yet, there exists related work also for non-multi-object dynamics representation learning which should be cited such as: \
 Minderer et al. “Unsupervised learning of object structure and dynamics from videos” NeurIPS 2019 \
 Blattmann et al. “Understanding Object Dynamics for Interactive Image-to-Video Synthesis” CVPR 2021 \
Also, the work should cite major works in video prediction since it is the task the network gets trained on: \
Lee et al. “Stochastic Adversarial Video Prediction” (SAVP) \
Franceschi et al. “Stochastic Latent Residual Video Prediction” (SLRVP) \
Denton et al. “Stochastic video generation with a learned prior” (SVG) 
- Missing comparison to current non-multi object representation-based approaches for video prediction. It would have been nice if the paper would have compared to non-multi object video prediction approaches to put their work more into context.
- This work does not deal with the inherent ambiguity of the video prediction task itself. It deterministically predicts a next state given two past observations. Thereby, it cannot cover the different scenarios the future can hold. The work from Lee et al. actually showed that by accounting for this ambiguity, the learned model improves on the task of video prediction (since it does not average over multiple predictions) and can cover multiple different future scenarios.  I would like to hear the authors take on that and see this more as future work for multi-object dynamics learning.

General notes:
- Below (4) there is z^{dyn} twice I believe the second one should be z^{sta}
- Paper states that their approach is capable of future frames prediction in “complex” 3D scenes. I have to disagree with this since this dataset on which the method is evaluated is far away from any real-world video dataset that is why I would not call it “complex”. 
- Change the name from “Reported” to “ALOE w/o self-superv.” to make it easier to read in table 2.
typos: figure 7 caption it’s -> its; very young infants -> infants; line intensity -> Line intensity...
- Figure 7 has a,b and c but in the caption only a and b occur. In c) the x-axis goes from -2 to 2 but there are 10 images -> -3 to 3? 
- The paper mentioned that the “Object’s motions … has to be *continuous* through space” while in the next sentence explaining how the next *discrete* static object latent state is predicted. One would have to use e.g. Neural ODEs to actually model continuous object motion so continuous should be replaced by e.g. smooth.
- The writing of the paper needs some polishing since the wording in many sentences does not make sense e.g. “Object’s motion consists with the basic physics concept ...”. 
- In the caption of 7b it says: “line intensity indicates the magnitude of attention probability” while in the figure it is written that the score of blue is 0.99. To me, all the intensities look more or less the same while one of them is already 0.99 out of 1 (the other ones should be then ~0.01). How does that make sense?  Could the authors not show all the weights between all objects between two frames (also for multiple examples). This would provide much more insight to the reader. 
- It is difficult to spot the object movement in figure 7 c. It would be great if the authors could add some markers to help visibility.


summary_of_the_review: The modules proposed in this work are well motivated and designed and produce strong results on video reasoning. Yet, I am concerned about the insufficient evaluation. Without the ablations and the baseline I discussed in weaknesses it is difficult to quantify the actual impact of the proposed method and its underlying modules. Moreover, experimental details are missing which makes it difficult to reproduce its reported numbers. That is why my initial rating is below the acceptance threshold. But I am open to adjusting my rating if the authors do sufficiently address the points raised in weaknesses.
review_title: null

Meta-review: This work presents a novel method h to learn object dynamics from unlabelled videos and shows its benefits on causal reasoning and future frame prediction.  This paper received 4 positive reviews and 1 negative review. In the rebuttal, the authors have addressed most of the concerns. AC feels this work is very interesting and deserves to be published on ICLR 2022. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make other necessary changes.
Decision: Accept (Poster)