OBJECT DYNAMICS DISTILLATION FOR SCENE DECOMPOSITION AND REPRESENTATION
Authors: Qu Tang
Xiangyu Zhu
Zhen Lei
Zhaoxiang Zhang
Venue: ICLR 2022 Poster (2022)

Abstract: The ability to perceive scenes in terms of abstract entities is crucial for us to
achieve higher-level intelligence. Recently, several methods have been proposed
to learn object-centric representations of scenes with multiple objects, yet most
of which focus on static scenes. In this paper, we work on object dynamics and
propose Object Dynamics Distillation Network (ODDN), a framework that distillates explicit object dynamics (e.g., velocity) from sequential static representations. ODDN also builds a relation module to model object interactions. We verify
our approach on tasks of video reasoning and video prediction, which are two important evaluations for video understanding. The results show that the reasoning
model with visual representations of ODDN performs better in answering reasoning questions around physical events in a video compared to the previous state-of-the-art methods. The distilled object dynamics also could be used to predict
future video frames given two input frames, involving occlusion and objects collision. In addition, our architecture brings better segmentation quality and higher
reconstruction accuracy.

Published as a conference paper at ICLR 2022
OBJECT DYNAMICS DISTILLATION FOR SCENE DE- COMPOSITION AND REPRESENTATION
Qu Tang XiangYu Zhu 12 Zhen Lei 123
ZhaoXiang Zhang 123+
School of Artificial Intelligence, University of Chinese Academy of Sciences Institute of Automation, Chinese Academy of Sciences
aCentre for Artificial Intelligence and Robotics, Hong Kong Institute of Science & Innovation,
Chinese Academy of Sciences
(tangqu2020, zhaoxiang. zaaiiiinn....gggg...a. @ia. al.....cc.r (xiangyu. zhu, zlei @nneen1pppp pr. par.iaaac... an
ABSTRACT
The ability to perceive scenes terms of abstract entities crucial for us to achieve higher-level intelligence. Recently, several methods have been proposed to learn object-centric representations of scenes with multiple objects, but most of them focus static images. In this paper, we work object dynamics and propose Object Dynamics Distillation Network (ODDN) which distills explicit object dynamic representations (e.g., velocity) from raw video input. Further- more, we build relation module that calculates object-pair interactions and ap- plies it the corresponding dynamic representations of objects. We verify our approach on tasks of video events reasoning and video prediction, which are two important evaluations for video understanding. The results show that vi- sual representations of ODDN perform better in answering reasoning questions around physical events in a video compared representations of the previous scene representation methods. And, ODDN could generate reasonable future frames given two input frames, considering occlusion and objects collision. addition, using the object dynamic clues allows the model to obtain better scene decomposition quality in segmentation and reconstruction. Code is available at attps://githus wtt.:/cccccccoaeeee wtt.tt/uww.ccm////////////
INTRODUCTION
Humans learn to decompose the scene into in- dependent objects to understand the environ- ment at a very early age. This cognitive pat-
ODDN
tern promotes the generation of common of physics, for example, the speed of the object will change after collision. However, the neu- ral networks seem to lack the ability to acquire compositional understanding of the world in terms of structured objects, which is crucial for generalizing and higher-level tasks such as planning and reasoning(Greeff et al., 2020). To tackle this problem, many scene decomposi- tion approaches have been proposed to segment an image with ground-trutt segmentation labels inpervision(Ronneeeee
Impui Image
Downstream Tasks
Resonstructed
happee next
Reasoning
Prediction
Segmentation
Figure I: ODDN could be used to handle multiple downstream tasks.
et al., 2015 Long
et al. 2015 (Badrinarayanan et al.
2017 Chen al. 2017 Zhang et al. 2020a b). However, the ac-
quisition of semantic segmentation labels is costly. Besides, these methods do not provide structured representations of segmented areas. Recently, several unsupervised methods have become popular learning decomposition and object-centric representations of scenes with multiple objects(Hsich
et al. 20 g: Van Steenkiste et al. 2018 Burgess et al.) 2019 Locatello et al. 20200 Greff et al.
*Corresponding author

---

Published as a conference paper at ICLR 2022
2017
Eslami et al. 2016 Kosiorek et al. 2018), and some of those methods learn disentangled features which represent crucial properties (c.g., 'color', 'size') of object in the same format. The work of Van Steenkiste et al. (2019) has proved that disentangled representations do in fact lead to better downstream task performance.
Beyond static scenes (images), people understand dynamic physical events and predict how the fu- ture might (Bttaglia et al., 2013). Even infants have the intuition that temporarily occluded objects remain coherent wholes and follow spatially contiguous pathssBaillargeon, (987). This com- mon sense of physics is inseparable from the understanding of object dynamics. However, existing scene decomposition and representation work focus on images, which means that when interpreting videos, those approaches process one frame at a time, and do not encode objects' dynamic proper- ties. We argue that representing object dynamics significant for tasks such as video reasoning or predicting the future.
In this work, we aim to distill and disentangle object dynamics in an unsupervised manner and present ODDN (Object Dynamics Distillation Network). ODDN is built spatial mixtures frame- work(Greff et al. 2019), a prime example of VAB(Kingma & Welling 2013) based architecture of decomposing scenes into object-centric latent representations. Formally, the model clusters and decodes pixels into latent representations through an iterative amortized variational inference pro- cedure then reconstructs the latent vectors into input images. In this way, scenes segmented and represented as object-centric disentangled features. Based on this, our key insight is that the object dynamics hide in variations between objects' static latent representations of different frames, thus, we design our model to distill object dynamics from their representations of the current and previous frames. With the distilled object dynamics as part of the representation, further build a relation module to model object interactions.
We evaluate ODDN on different downstream tasks as shown in Figure || Results show that rep- resentations with object dynamics perform better in reasoning task than representations with static properties only, and the relation module endows ODDN with the capability of predicting the fu- ture. In addition, we notice that incorporating object dynamics and relations into the basic scene decomposition framework benefits the segmentation and reconstruction quality.
We highlight contributions of our approach:
Distillation of disentangled object dynamics from raw video input in unsupervised man- ner.
State-of-the-art performance on CLEVRER Yi et 2019) which focus on video under- standing and causal reasoning.
Capability of future frames prediction in 3D scenes including physical events (occlusion, collision).
Better scene decomposition quality for spatial mixture architecture in terms of segmenta- tion and reconstruction.
RELATED WORK
Object representation of static scenes. Representation learning is an important topic in deep learning. Several recent lines of work making breakthroughs in this area learns representations without explicit supervision. NEM(Gr et al. 2017) builds spatial mixture model based on EM algorithm to discover compositional object representation but fails to cope with colored or 3D data. AIR(Eslami et al. 2016) achieves scene interpretation by using a recurrent neural network that attends to one element at a time and disentangles positions and appearances explicitly in latent variables. AIR can decompose 3D scenes, however, it generates partial images separately and adds them together to form the final image, which leads to its restriction: it can't be applied to scenes with occluded objects. MoNet(Burgess et al.) (219)) uses multiple attention steps to tackle complex 3D scene, which is the first being capable of discovering object masks and completing partially occluded objects. Grefff et al. 2019) does the same thing as MoNet by using iterative variational inference to refine the inferred latent representation in each encoding step.
Object representation of dynamic scenes. Methods mentioned above are designed to de- compose static scenes, hence they do not encode object dynamics in the latent space. R-
2

---

Published as a conference paper at ICLR 2022
inlmnd jane
Inference model
Generative model
Match Desii!
Dedied Oyanno
Pedicad
antetion
+
Syrenk
Prodiceer
Iftrndoo
200
Embedking
Figure 2: Overview of ODDN. The inference model assigns inputs into K slots per image, each slot encodes object or background into latent space, shown in different colors. We take use of the attention mechanism of transformer to align objects' latents of two input images and encode the aggregated representation into a low-dimensional vector to obtain disentangled object dynamics. Object latents (sta) and dynamics (dyn) are used to model interactions between object-pairs, we show the relation module with three slots as example, which compute the effects 22 and 23 have on 21- We concat the updated object dynamics and object latents to predict the next frame object latents. The Generative model outputs both reconstructions of input images and prediction of future frame images.
NEM(Van Steenkiste et al. 2018) applies RNN to NEM to encode object dynamics implicitly, and models relations between objects to learn physical interaction which enables the model to predict future frames for video input. Similarly, COAIR(Kosiores et al.) 2018) extends AIR to image se- quence to discover and track objects throughout the video frames. Both R-NEM and SQAIR bear the drawbacks of their predecessors. ViMON(Weis et al. 2020) is video-extension of MoNet by building temporal connections between latent representations with GRU et al. 2014). ViMON learns linear transformation to predict next time- step image, encouraging dynamics encoded. PRO- VIDE(Zablotsaiia et al.) 2020) applies 2D-I STTSS et al. 2007) to IODINE and is capable of simulating future video frames. Although ViMON and PROV IDE encode object dynamics, they do not model interactions between objects, thus, they can't 't handle datasets with object collisions. Besides, object dynamics of both methods are spread in latent space implicitly, which lacks inter- pretability. It has been proved that disentangled representations allow one to learn models with better performance and fewer samples for downstream reasoning tasks WW al. 2021 Chen et al., 2021bb Gan et al., 2017 Ding et al., 2021b). Therefore, in this work, we aim to represent object dynamics explicitly. Instead of using RNNs, we match objects' latent representations between the current and previous frames and distill their dynamics by applying a single-layer transformer encoder.
METHOD
Our goal is to decompose dynamic scenes into object-centric representations purely based visual observations. And the representation contains interpretable object dynamics. Recent works decom- pose static scenes by randomly assigning objects fixed number of slots, which share weights to obtain latent vectors with a common format. However, these works do not encode object dynamics for objects of sequential input. Our key idea is that building upon these successful multi-object scenes decomposing and representation approaches, object dynamics could be distilled from the corresponding latent representations of two consecutive frames.
3.1 ОBJECT DYNAMICS DISTILLATION NETWORK
We illustrate the framework of ODDN in Figure Formally, ODDN takes input a video, then the inference model processes all frames Ti:N independently to obtain object latent vectors which encode object static properties. For frames J2.N distill objects' dynamics from their latent
3

---

Published as a conference paper at ICLR 2022
vectors of the current and the previous frames. The relation module takes the object latent vectors and the object dynamics of frames 22:N-1 as input, calculates the interaction between objects and predicts object latent representations of frames 23.N-
The imput image x IRD could reconstructed by K latent vectors 2 E RM each of which corresponds to one object background area after training, and each latent vector is also called slot. Firstly, latent 25. is decoded into pixel-wise mean μ4r and pixel-wise mask logit ma Then, the K mask logits are normalized with softmax function to ensure that the of K mask logits for each pixel is 1. At last, and m parameterize the final spatial Gaussian mixture distribution:
K II
2(x|2)
i=1 k=1
where is the same and fixed for all and k. The input image can be reconstructed á Σ myHk..
The inference model of ODDN follows previous work (Greff et al. 2019) by using amortized vari-
ational inference infer and refine posterior (x(2)z) iteratively where {12z The main reason to use such an iterative process is that the standard variational inference framework leads to the multi-modal problem, which means that any pixel could captured by any slot. JODINE
updates posterior as follows:
z' ~ (x)),,)),
(2)
X+1 + X' fo(=",
(3)
where the posterior parameters À start with an arbitrary guess, and the first latent representations 2 sampled from gx- In each refinement step, input together with set of variables a(means 14, masks m, mean gradient V,L., mask gradient C.. which are computed from 2' are fed into a refinement network J to obtain the additive updates for the posterior.
ODDN predicts future objects state in latent space given first two frames:
dyn Ank W [reeed(= 1)1 (n-1)x)) 1);
2,
where zdyn is the distilled object dynamic representation and ste refers to the object latent repre- sentation output by the inference model. The next-frame latent vectors are predicted with a learnable transformation W pred. details will be introduced Section 3.2 and Section 3.3. The whole network is trained end-to-end and the final loss consists of prediction loss L, and reconstruction loss C,, in detail:
N
T
C,(x) =
| +
(5)
N
(x(x) = Σ
Hogg(p(zn(2))]
(6)
n=3
where T is the maximum refinement step number during the inference stage. We measure and optimize the KI. divergence between the variational posterior and the prior Lr.
3.2 DYNAMIC DISTILLATION
In this section, we describe how ODDN derive object dynamics from sequential object latent repre- sentations. For existing scene decomposition and representation approaches, object-to-slot assign- ments usually switch unpredictably match object latent representations in different frames correctly and further distill object dynamics.
video. Thus, we draw support from the Transformer to
Revisit attention mechanism of Transformer. The key component of the Transformer is the attention module which could be mathematically described as:
M
Attention K,
Σ
as W.y: as x erp (W (Waks)),
(7)
4

---

Published as a conference paper at ICLR 2022
where q Ra; K consists of [ elements (2, .. kx)) with K Ra; V consists of [V] elements (vi; 02, vlv)) with E IRg-
This attention function maps the input query q and set of key(k))-value()) pairs to an updated representation, Wg Wx Wr are learnable parameters. Besides, oz 2 0, EE = 1, and usually
| = |.
Model architecture. The inference model decomposes the input video into N x K latent vectors, where K is the slot number for each frame and N is the number of frames of the video. We put these sequence consisting latent vectors {211.-.. ... S1K.... 2N1. encoder layer to match the corresponding slots which usually switch their assignments with objects several times in video. After finding object correspondence in different frames, the Transformer aggregates latent information and distills object dynamies which are encoded into low-dimensional representation space. Mathematically, the dynamic distillation process for the k-th slot in the n-th frame could be described as:
(NNK] into one single Transformer
nkk Allention( (ukk; {Fn-1; Zn) 1, > 1,
(8)
dyn Fyk
FFN (No(NNr + knk));
where zd0u low-dimensional vector transformed by the fully connected feed-forward network (FFN). This learning process is performed in parallel by pre-defining an attention mask to ensure that latent representations of the current frame only attend to ones one frame before.
3.3 RELATION MODULE
ODDN models interactions between object pairs by computing all object-pair effects and letting them act on and update object dynamics. ODDN learns a soft attention coefficient for each effect. The object dynamics are updated
emb & Wemm (ladyn,
Wemmm stte dym, sata)); 2k, 2/,
(10)
attki,k, Sigmoid(Wattann (embre)
(11)
of fk = Wass (embkrkk) Ek. =
Σ
attkiik * ef fkuky: $
(12)
2xk:
dym'
2nk
4ym
+ W (FRs):
(13)
where W Womb first embeds representations of object-pair, then War learns the attention coefficient based on the object-pair embedding, and Wafs learns the effect zky has on zk: At last, the summed effects output an update value with Wayn.
We draw inspiration from NPE(Chhag al. 2016) and R-NEM Van Steenkiste al. 2018). both architectures model object relations. NPE has access to ground truth representations of scenes and computes effect for each neighborhood object-pair with a neighborhood mask based on ground-truth object position, and finally. the summed effects act on object velocity. R-NEM represents objects in an unsupervised fashion and implicitly encodes sequential information with an RNN(Hochreiter & Schmidhuber] 1997). Instead of using neighborhood mask, R-NEM learns soft attention for all object-pairs. Our approach learns to represent object dynamics explicitly. Thus, we compute all object-pair effects and let it act on object dynamics NPE, and learn attention coefficient for each effect as R-NEM.
After distilling and updating object dynamics, predict next-frame object latents by simply learn- ing transformation Wpred, and W Wpred: TR(++1)x gste
dyn' Wprea([222") ; ;2200)) where |
(n+1))
indicates the dynamics dimension (I (1=44 in our experiments).
To alleviate error accumulating when implementing multi-step step predicting in the inference stage, use student forcing strategy during training: instead of only using 2, to predict 2 & is also used to iteratively roll out predicted image sequence.
5

---

Published as a conference paper at ICLR 2022
Table I: Performance (per question/option accuracy) comparison with the state-of-the-ar methods
MAC (V+), NS-DR, DCL, and ALOE on CLEVRER, ALOE takes static scene representations from
Monet as visual input while ODDN-ALOE use representations of ODDN.
Model
Descriptive
Explanatory
Predictive
Counterlactual
opt.
ques.
opt.
ques.
per opt.
per ques.
MAC(V+) Yi et al.
2019
86.4
70.5
22.3
59.7
42.9
63.5
25.1
NS-DR Yi 2019
88.1
87.6
79.6
82.9
68.7
74.1
42.2
Chen
2021a
89.6
82.8
90.5
82.0
ALOE(Ding et al 2021a,
94.0
98.5
96.0
93.5
87.5
91.4
75.6
ODDN-ALOE
95.8
98.9
97.0
95.7
91.8
93.0
80.1
Table 2: Ablations of how different object representations perform CLEVRER, all experiments are conducted based on ALOE without self-supervision. "ODDN w/o relation" is ODDN trained without relation module (with only dynamic distillation module). - dyy means reducing distilled
dynamic representations (only static representations remained). "MoNet w/o ss" is the result of
ALOE without self-supervision from original paper with MoNet features.
Feature
Dim Dyn
Deserittive
Explanatory
Prodictive
Counterfactual
per opt.
per ques.
per opt.
ques.
per ques.
PROVIDE
16
75.2
91.6
77.4
80.5
65.4
76.4
39.4
MoNel w/o 88
91.0
92.8
82.8
68.7
IODINE IODINE
92.8
98.3
95.6
89.1
89.7
71.3
93.5
98.1
95.0
91.6
84.5
90.6
73.4
ODDN relation! dyn)
92.8
82.8
89.2
70.1
ODDN w/o relation
V
93.9
98.2
95.1
93.6
88.0
90.7
73.6
(DDN(=dp))
95.1
98.5
96.1
94.1
89.1
91.7
76.8
ODDN
20
95.8
98.9
97.0
95.7
91.8
93,0
80.1
4 EXPERIMENTS
In this section, we design experiments in the perspective of representation, prediction, and scene decomposition. In particular, we study how ODDN performs on tasks of video understanding and reasoning, video prediction, reconstruction, and segmentation. To quantify segmentation quality, measure the similarity between ground-truth (instance) segmentations and our predicted object masks using the Adjusted Rand Index(Rand, 1971) (ART) and The Foreground Adjusted Rand Index (F-ARI) which is modification of the ARI score ignoring background pixels.
Baselines. We compare our method with IODINE, PROVIDE and SRVP(Franceschi et al. 2020). IODINE is designed for static images, thus it neither encodes object dynamics nor considers object relations. PROVIDE is an extension of IODINE to sequential inputs. The major difference between PROVIDE and approach is that PROVIDE encodes object dynamics implicitly while we distill object dynamics explicitly. In detail, PROVIDE uses 2D-LSTM to achieve temporally conditioned and iterative amortized inference for posterior refinement, latents in every time-step contain dynamic information which could be used to predict the future. Besides, PROVIDE does not model object relations, which means when predicting the future, object latents are processed separately. SRVP is a dynamic latent model for stochastic video prediction which decouples frame synthesis and dynamies based on residual updates of small latent state. Unlike ODDN and PROVIDE, SRVP reperesents the scene with single latent.
result, the final sampled
4.1 VIDEO UNDERSTANDING AND REASONING
We study the representation power of ODDN on CLEVRER, which poses different types of ques- tions for each video in CLEVRER dataset. Those questions could test the model's understanding of a vidco comprehensively, including descriptive questions ("how many objects are moving"), ex- planatory questions("tha reason why certain collision happens"), predictive questions("what will happen next"), and counterfactual questions(" would happen if").
Table shows that ODDN-ALON outperforms ALOE all question types even without self- supervision, especially on predictive and counterfactual questions, and achieves statc-of-the-as per- formance. Both predictive and counterfactual types of questions related predicting how objects

---

Published as a conference paper at ICLR 2022
t=1
t=2
t-3
t=4
t=5
t=6
t=7
t=8
PROVIDE ODDN
Konditioning frames]
Ground truth
IODINE linear
w/o relation
SRVP
Figure 3: Illustration of video prediction results CLEVRER. All models take as input the first two frames and predict the next six frames. "IODINE linear" is implemented by simply concatenating latent features of the current and the previous frames and predicting the future frame with a feed- forward neural network. "ODDN w/o relation" is the version of ODDN without relation module. In the video, collision happens between the blue and brown eylinder from time step 6 to 8.
will move. This result proves that representations with object dynamics could help the neural model to do better understanding and reasoning around physical events. We also do ablation experiments to study how different representations affect the model's performance, results are shown in Table 2 We can tell that representations of PROVIDE lead to significant decline compared to IODINE, and the reason will be discussed in Section 4.3. On the contrary, representations of ODDN which encode object dynamies (dim=20) outperform IODINE (dim=16) by large margin, especially predictive (91.8 vs 84.5) and counterfactual (80.1 vs 73.4) questions. In addition, we obtain ODDN features without dynamics by reducing 4 dimension dynamics. Results show that ODDN without dynamics still performs better than IODINE, which means that ODDN not only distills useful ob- ject dynamics but also generates better static representations. static representations. "ODDN w/o relation" outperforms "ODDN w/o relation -dyn" which further shows the significance of our dis- tilled object dynamics. However, "DDN w/o relation -dyn" is no better than IODINE (dim=16), which proves that the relation module is the main reason for the quality increase of the object static representation.
4.2 PREDICTION
ODDN achieves predicting future frames in iterative fashion. Unlike PROVIDE, ODDN mod- els object dynamics and interactions explicitly. Results could be observed in Figure B where the blue and brown cylinder collides. ODDN could output reasonable predictions and handle objects collision. PROVIDE could only predict object's future state independently, and the reason is that PROVIDE does not cope with objects interaction, thus, objects blend when they suppose collide and change their dynamics. ODDN also generates more stable predictions comparing to PROVIDE which fails to capture the object's correct properties (shape, color) after a few frames. "IODINE lin- ear" and SRVP generate identical pictures, which means that both methods are incapable of learning object-level dynamics on CLEVRER. "ODDN w/o relation" learn some dynamics, but the gener- ated results are not reasonable, which may be caused the training noise: predicting the collision without modeling object interactions. We calculated the average MSE of each the predicted frames the entire test set, the quantitative results shown in Figure

---

Published as a conference paper at ICLR 2022
Input Image
IDDINE
PROVIDE
ORAN
Figure 4: Comparison of reconstruction results of ODDN with IODINE and PROVIDE, we highlight dense areas with dashed boxes.
4.3 DECOMPOSITION
We show how different architectures perform terms of reconstruction quality in Table 3 Comparing IODINE, PROVIDE brings some negative effects, shown in Figure PROVIDE occasionally fails to capture the right object shape. Instead, we notice that ODDN performs better than IODINE. Specifically, the reconstruction of close objects is more precise. We believe that this benefits from the relation module which relies on object surface details to process objects interaction (whether collision occurs and how object dynamics update). The reconstruc- tion performance is positively correlated to object representation quality which has been verified in Section 4.1.
Reconstruction.
Segmentation. ODDN and baseline models decode object latents into pixel val- ues and segmentation masks. The major contribution of PROVIDE is that mod- els temporal dependencies between latents across frames obtain better segmentation results.
Table 3: We compare ODDN with baseline models on scene decomposition quality, ODDN-I indicates train- ing with the long rangerangega prediction (6 frames).
Model
Dyn MSE
ARIT
F-ARI
IODINE PROVIDE
0.000100
0.096
95.55
0.000148
0.228
95.17
ODDN
<
0.000075
0.156
95.63
Table indicates that PROVIDE hurts the model's 8 reconstruction capability. Instead, we show that the future frame prediction task in our architecture also improves the segmentation performance and achieves good reconstruction quality at the same time.
ODDN-1
0.000103
0.233
95.36
The ARI further improves when increasing the steps future object states to predict. We believe that the long-range prediction supervision encourages ODDN to attend more foreground objects which may change its position and view over time. This temporal consistency similar to PRO- VIDE, which could be the main reason for both models achieve better segmentation performance than IODINE. Results are shown in Figure
4.4 REAL BLOCK TOWERS
Setup. We further evaluate ODDN on the real block tower video dataset[[eere et al., 2016). The block tower dataset has 493 real-world vidcos and each video contains a block tower which may or may not be falling. We train models on 393 randomly- videos and evaluate them on the rest 100 videos. We study how the models perform in reconstruction and predicting the future frames with the physical concept "falling".
Results. We show reconstruction results by ODDN and baseline models of images with block towers different inclination in Figure 0 We found that all models can reconstruct images of stable block towers precisely. However, for blocks with complex spatial position states (e.g., falling
8

---

Published as a conference paper at ICLR 2022
with interactions), IODINE and PROVIDE often fail to restore accurate block properties. ODDN performs the best and is able to retain the shape and inclination angle of the object.
In Figure present prediction results of ODDN, PROVIDE and SRVP. SRVP predicts reasonable object dynamics but is not accurate, and the generated images are not clear. The blocks predicted by PROVIDE tend be blurred and fused over time. ODDN models object interactions, thus predicted blocks do not blend easily.
Input
PROVIDE
image
IODINE
IODINE
ODDN
Input Image
Figure 5:
Unsupervised Segmentation re- sults of ODDN, PROVIDE, and IODINE on CLEVRER. Each color corresponds to one slot index whose assignment is unpredictable. Both PROVIDE and ODDN generate better segmen- tation masks than IODINE.
Figure 6: Qualitative comparisons of recon- struction of ODDN with IODINE and with PROVIDE on realistic block towers.
t=1
t=2
t=3
tm4
t=6
t=7
t=8
ODON
PROVIDE
SRVP
ODON wib relation
SNV
CO0N MRvEr
Figure 8: Qualitative comparison of predic- tion performance of ODDN with PROVIDE and SRVP on realistic block towers.
Figure 7: Comparison of MSE for 6 predicted frames of different models on CLEVRER.
5 DISCUSSION AND FUTURE WORK
We presented an unsupervised learning framework ODDN, novel approach to decompose tempo- ral scene of multiple objects with dynamics and interactions. ODDN distills disentangled object dynamics which are significant in downstream video reasoning task. ODDN also models object interactions which endows the model with the capability of predicting future frames. The incorpo- rating of object dynamics and interactions benefits the model's decomposing ability, reflecting in better segmentation mask and reconstruction quality.
ODDN also has limitations. The prediction quality decrease over time because of the error accumu- lation. ODDN can only generate deterministic predictions for dynamics obeying regular patterns. Besides, ODDN needs two frames to firstly derive object dynamics then predict the future. However, in the block tower case, humans can tell whether the tower will fall or stay stable with one frame by commonsense. We leave this for future work.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
This work was supported in part by the National Key Research Development Program (No. 2020YFC2003901), Chinese National Natural Science Foundation Projects 62176256, 61876178, 61976229, the Youth Innovation Promotion Association CAS (Y2021131), and the InnoHK pro- gram.
REFERENCES
Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder- decoder architecture for image segmentation. IEEE Iransactions on paltern analysis and machine intelligence, (9(11))2481-2455 2017.
René Baillargeon. Object permanence in 3 1/2- and 4 1/2-month-old infants. Developmental Psy- chology, 23(5):655-66) September 1987. ISSN 0012-1649. doi: 10. 103770002--169.22.5.666
Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):18327-18332 2013.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and represen- tation. arXiv preprint
11390, 2019.
Michael B Chang, Tomer Ullman, Antonio Torralba, and Joshua B Tenenbaum. A compositional object-based approach learning physical dynamics. arXiv preprint arXiv:1612.000: 2016.
Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4): 834-848, 2017.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Ye Yee Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reason- ing. arXiv preprint arXiv:2103 16564, 2021a.
Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. In ICLR, 2021b.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning. Advances in Neural Information Processing Systems, 34, 2021a.
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Josh Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physies models from video and language. NeurIPS, 34, 2021b.
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. Advances Neural Information Processing Systems, 29, 2016.
Jean-Y Yves Franceschi, Edouard Delasalles, Mickaël Chen, Sylvain Lamprier, and Patrick Gallinari. Stochastic latent residual video prediction. In International Conference on Machine Learning, pp. 3233-3246. PMLR, 2020.
Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boqing Gong. Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focused semantic segmen- tation. In ICCV, pp. 181 1-1800.. 2017.
10

---

Published as a conference paper at ICLR 2022
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Multi-dimensional recurrent neural networks. In International conference on artificial neural networks, pp. 549-558. Springer, 2007.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization.
Advances in Neural Information Processing Systems, 30, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-ohject representation learning with iterative variational inference. In International Conference on Machine Learning, pp. 2424-2433. PMLR, 2019.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Jun-Ting Hsieh, Binghin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. Advances in neural information processing systems, 31, 2018.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13/2.61 14, 2013.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, 31, 2018.
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In International conference on machine learning, pp. 430 410-433448 438. PMLR, 2016.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in Neural Information Processing Systems, 33:1 1525-115328 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241.. Springer, 2015.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. arXiv preprint /oXiv:/802.10333 2018.
Sjoerd Van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disen- tangled representations helpful for abstract visual reasoning? Advances in Neural Information Processing Systems, 32, 2019.
Marissa A Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander Ecker. Unmasking the inductive biases of unsupervised object representations for video sequences. arXiv preprint arXiv:2006. 07034, 2, 2020.
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning real-world videos. NewrIPS, 2021.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint /rXiv: 2019.
Polina Zablotskaia, Edoardo A Dominici. Leonid Sigal, and Andreas M Lehrmann. Unsupervised video decomposition using spatio-tempont iterative inference. arXiv preprint arXiv:2006.14777 2020.
11

---

Published as a conference paper at ICLR 2022
Xiaomei Zhang, Yingying Chen, Bingke Zhu, Jinqiao Wang, and Ming Tang. Blended grammar network for human parsing. In European Conference on Computer Vision, pp. 189-205. Springer, 2020a.
Xiaomei Zhang, Yingying Chen, Bingke Zhu, Jinqiao Wang, and Ming Tang. Part-aware context network for human parsing. In Proceedings of the IEEE/CVF Conference Computer Viston and Puttern Recognition, pp. 8971-8980, 2020b.
A DETAILS ON METHOD AND IMPLEMENTATION
A. ALGORITIMM
Here we detail our algorithm with pseudocode:
Algorithm 1: ODDN Pseudocode
Input: video 21 ... EN hyperparameters K,T
Input: trainable parameters (11), 0, d, p, 8,
Function Infer Inferreccnstzzuc (x):
for to do
(2 Zg (0) g=
Mk m Decodeero (t)
(t) (t)
m(t) t softmar(m (t)
((x) + - m
(1 N (0) o22)
(.) (z) + (
Blog(p(z|2(11)
as (z, (1) ())
X ((+1) hit + +1)++]=---- falax, (1)
10
end for
return Ly, (T)
11
12
for n
N 2 do
En., (x) + Infer fr Reconstruuc((za)
2(n+1).: L, (I((n1)) + Infer Reconstruu((a))
14
15
dym(n+1)- t Trans Transformery ormmrr
F(n+1)-)
dym(m+1)- Relations (d) ((m1))))))))))))))))
16
{(m+2)} + Prediclor (dyn (m+1)) ((n-))
17
(2(2))))) t -log (2) (2++)))
Output: Lp + Lr
A.2 DATASET
CLEVRER is synthetic video dataset of moving and colliding objects. Each video contains 128 frames at resolution 480 x 320. We pre-train ODDN on the entire CLEVRER training set, order to promote convergence of our Relation Module, we extract images every 4 frames and ensure that at least one collision event is included forming CLEvRER-colissonn and we fine-tune ODDN on CLEVRER-collision. For testing, use the validation set which has ground truth masks, we sample 1k sub-clips containing 6 objects, each sub-clip consists frames.
A.3 HYPER-PARE PARAMETERS
We Generally follow the setting of PROVIDE. We initialize the parameters of the posterior by sampling from U(0.5,0.5). In experiments in prediction tasks, use a latent dimensionality of 64 and downscale the image into 64x64 after a center-crop preprocess as IODINE and PROVIDE, such that dim(A) 128. And in experiments in video reasoning task, we latent dimensionality of
12

---

Published as a conference paper at ICLR 2022
Table 4: Computational complexity comparison between ODDN and IODINE and PROVIDE.
Model
GFLOPs
Parameters
Model Size
IODINE
17.2187
2780052
10.61
PROVIDE
17.2251
2977940
11.36
ODDN
17.2189
2925365
11.16
16 as ALOE which makes dim(A) = 32, and downscale the image into 64x96 without crop. The variance of the likelihood is set to 0.3 in all experiments. We keep the default number of iterative refinements at R = 5, and use K slots for both training and testing. Furthermore, we set 100.0 for all experiments.
A.4 TRAINING
We train our models on 8 GeForce RTX 3090 GPUs, which takes approximately two days per model. We use ADAM for all experiments, with learning rate 0.0003 and default values for all remaining parameters. During training, we gradually increase the number of frames per video, as have found this make the optimisation more stable. We train models with sequences of length 8 and the batch size is 12.
B MODEL COMPUTATIONAL COMPLEXITY
C MORE VISUALIZATION RESULTS
Ground truth
Reconstruction
Prediction
0008
0003
0.04
2.055
0.007
0.0007
Attention
coefficient
0am 0.001
0:35
0OO0C
0,00
Figure 9: Addition prediction results ODDN on CLEVRER and detailed attention coefficients visualization for relation module. We plot attention the scores all objects have the rubber sphere from the second frame. The collision happens between the rubber sphere and the metal sphere.
13

---

Published as a conference paper at ICLR 2022
(a)
Attenton Score0..59
Dalllld
Single latent travenal
Synaiics
Velecity
(b)
(c)
Figure 10: (a) Two consecutive frames with only the blue cylinder moving. (b) Visualization of the Transformer attention map of the blue cylinder in the matching phase of dynamics distillation. Latents of the second frame will update its representations by attending and merge information from latents of the first frame. Line intensity indicates the magnitude of attention probability. (c) Disentanglement visualization of the learned object dynamics. We adjust the feature value (from -2 to 2) of the corresponding dynamic representation dimension the blue cylinder, and visualize the effects the predicted next frame. We arrows show the velocity direction each dimension represents.
14


Reviews:
[{"review_id": "usD9YlZNw25", "reviewer": "Reviewer_pDGg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is compared to the previous methods, IODINE and PROVIDE, and the better results are shown. But, the problem is that the detail of the method is not described and difficult to understand.\n\nIn th experiment of video understanding, the input for IODINE is a single frame, but that for the proposed method is multiple frame from video. How different the input data is? If the information extracted by the proposed method is the dynamics of a scene, the importance will depends on the scene. But, no example is shown in the paper to confirm the effectiveness of Dynamic Distillation. Although the method, PROVIDE, is chosen as a method that introduces temporal information, it may not be meaningful because the performance is worse than the base method, IODINE, which uses a single image.\n\nIn the experiment of prediction, it is difficult to recognize how is the motion of objects and interaction each other in Fig.4. Is the example appropriate to show if the proposed method can extract the dynamics from video?\n\nIn decomposition, what is the condition of the experiment? Is the reconstruction through the autoencoder, or reconstruction from the latent vectors given by user? If in the latter case, what and how are the input latent vectors given?\n\nIn the explanation of the proposed method, the indexing of the latent vector is ambiguous. A latent vector is indexed by the frame number i, latent index k, time t and the pixel coordinate. But, what is the difference between the frame number and the time? And the indexes are omitted in many parts of the explanation. It is difficult to understand which latent variables are considered to find the relationship.\n\nThe detail of Generative model is not explained. The color is estimated for each pixel. But, the relationship between a latent vector and a pixel is not explained.\n", "summary_of_the_paper": "This paper proposes a method of extracting the latent space that describes objects in a video. The encoder part, Inference model, encodes an input image frame-by-frame from a video based on IODINE. The relative information between the latent vectors of three consequent frames is extracted and updated based on Transformer. The input image is reconstructed from the latent vectors by the decoder part, Generative model. The proposed method is tested for various tasks on CLEVRER dataset. \n", "summary_of_the_review": "Although the proposed method seems to show the better results in the experiments, the explanation is not sufficient to convince if the experiment is appropriate and fair. Additionally, the explanation of the method is not clear to understand the detail. Additionally, it is not clear that what kind of information is extracted by the part, Dynamic Distillation, introduced in this paper. It seems no experiment is shown that object dynamics is distilled by the proposed method. Consequently, the reviewer thinks that this paper is below the acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dwjTfPXv2Pw", "reviewer": "Reviewer_dfem", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n- The paper presents a very interesting idea of using transformer structure to align latent features and distillate object dynamics, and it seems very effective.\n\n- Experimental results show superior performance compared to the SOTA on video understanding and reasoning.\n\nWeaknesses:\n\n- The paper is built upon many existing techniques, including the IODINE, NPE, and R-NEM. Especially, the core part of it is the use of IODINE latents and aligning them by a transformer structure. The reasoning module is a combination of NPE and R-NEM.\n", "summary_of_the_paper": "The paper presents a framework that distillates explicit object dynamics (e.g., velocity) from the discrepancies between objects’ static latents of different frames, called the Object Dynamics Distillation Network (ODDN). The approach is built upon recent works which decompose static scenes into independent latents by randomly assigning different objects to a fixed number of slots, which share weights to obtain latents with a common format. And this paper makes use of the attention mechanism of the transformer to align objects’ latents of two input images and encode the aggregated representation into a low-dimensional vector to obtain disentangled object dynamics. Afterward, the paper builds a relation module to model object interactions based on NPE and R-NEM.", "summary_of_the_review": "Though the paper is built upon many existing techniques, including the IODINE, NPE and R-NEM, and the novel part of this paper is using a transformer structure to align the IODINE latents and find discrepancies, I still consider this paper very interesting and strong. It may provide some insight on how to model individual object dynamics from videos. Moreover, the proposed method is very effective as experimental results show superior performance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VP0dZX0HFkS", "reviewer": "Reviewer_YU79", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n+ The introduced modules, dynamic distillation and relation modules, are well motivated and their design and realization seem reasonable.\n+ The paper reports strong numbers on the task of video reasoning outperforming existing works on CLEVRER.\n\nWeaknesses:\n- Missing ablation experiment for dynamic distillation and relation module. The paper claims that the slot assignment switches unpredictably inside the video. It proposed the dynamic distillation attention mechanism to solve that. Yet, it is not evaluated if this module actually solves this problem. With that, the reader does not know if this module is effective. The same for the relation module. The paper should report numbers with and without the use of this module to assess its impact. \n- Missing baseline. To assess the significance of the proposed modules (dynamic distillation + relation module), the reader also needs results without them. This baseline can be created by replacing the proposed modules with a feed forward neural network (with roughly the same number of parameters as the proposed modules) that predicts the next static representation (t+1) based on the concatenation of the (static) representation at time t and t-1. The paper introduces some kind of baseline in Table 2 (ODDN w/o dyn) by reducing the 4 dimension dynamics. Thus, the network is left to predict the next state solely based on the current static representation (at time t) which is insufficient. That is why I urge the authors to implement the baseline as pointed out above and to report the resulting numbers in the rebuttal.\n- The paper compares their method to PROVIDE which is the only other competitor that leverages temporal information. PROVIDE extends upon the static method of IODINE using a 2D-LSTM. The reported numbers in table 2 for PROVIDE are significantly (~20%) worse than for IODINE which does not make sense to me, especially when looking at the PROVIDE paper in which PROVIDE is on par with IODINE (on average). The same holds true for table 3. The MSE reported in PROVIDE is significantly better than IODINE whereas in this paper MSE is significantly worse (48% for MSE). This suggests to me that there is something wrong with how PROVIDE was trained/evaluated. The paper discusses a reason why ODDN is superior compared to PROVIDE in 4.3 but does not address why there is such a huge gap between IODINE and PROVIDE. Could the authors please clarify this for me? This shows and enhances the need for a baseline as discussed above.\n- Missing implementation details. The paper does not provide any implementation details like e.g. optimizer, learning rate, \\beta value (hyperparameters, etc). This makes the paper not reproducible and I urge the authors to add these details. Does the code get released after acceptance?\n- The paper evaluates their proposed model only on the CLEVRER dataset. To better assess the impact of this work the method needs to be evaluated at least on one other dataset such as e.g. CATER.\nCATER: Rohit Girdhar et al. CATER: A diagnostic dataset for Compositional Actions and Temporal Reasoning. ICLR 2020\n-Missing related work. This paper misses a relevant related work from Veerapaneni et al. “Entity Abstraction in Visual Model-Based Reinforcement Learning” (CoRL 2019). This work extracts entity representations from images and learns their dynamics for model-based reinforcement learning.  Moreover, this paper only cites related work which operates on multi-object representations. Yet, there exists related work also for non-multi-object dynamics representation learning which should be cited such as: \\\n Minderer et al. “Unsupervised learning of object structure and dynamics from videos” NeurIPS 2019 \\\n Blattmann et al. “Understanding Object Dynamics for Interactive Image-to-Video Synthesis” CVPR 2021 \\\nAlso, the work should cite major works in video prediction since it is the task the network gets trained on: \\\nLee et al. “Stochastic Adversarial Video Prediction” (SAVP) \\\nFranceschi et al. “Stochastic Latent Residual Video Prediction” (SLRVP) \\\nDenton et al. “Stochastic video generation with a learned prior” (SVG) \n- Missing comparison to current non-multi object representation-based approaches for video prediction. It would have been nice if the paper would have compared to non-multi object video prediction approaches to put their work more into context.\n- This work does not deal with the inherent ambiguity of the video prediction task itself. It deterministically predicts a next state given two past observations. Thereby, it cannot cover the different scenarios the future can hold. The work from Lee et al. actually showed that by accounting for this ambiguity, the learned model improves on the task of video prediction (since it does not average over multiple predictions) and can cover multiple different future scenarios.  I would like to hear the authors take on that and see this more as future work for multi-object dynamics learning.\n\nGeneral notes:\n- Below (4) there is z^{dyn} twice I believe the second one should be z^{sta}\n- Paper states that their approach is capable of future frames prediction in “complex” 3D scenes. I have to disagree with this since this dataset on which the method is evaluated is far away from any real-world video dataset that is why I would not call it “complex”. \n- Change the name from “Reported” to “ALOE w/o self-superv.” to make it easier to read in table 2.\ntypos: figure 7 caption it’s -> its; very young infants -> infants; line intensity -> Line intensity...\n- Figure 7 has a,b and c but in the caption only a and b occur. In c) the x-axis goes from -2 to 2 but there are 10 images -> -3 to 3? \n- The paper mentioned that the “Object’s motions … has to be *continuous* through space” while in the next sentence explaining how the next *discrete* static object latent state is predicted. One would have to use e.g. Neural ODEs to actually model continuous object motion so continuous should be replaced by e.g. smooth.\n- The writing of the paper needs some polishing since the wording in many sentences does not make sense e.g. “Object’s motion consists with the basic physics concept ...”. \n- In the caption of 7b it says: “line intensity indicates the magnitude of attention probability” while in the figure it is written that the score of blue is 0.99. To me, all the intensities look more or less the same while one of them is already 0.99 out of 1 (the other ones should be then ~0.01). How does that make sense?  Could the authors not show all the weights between all objects between two frames (also for multiple examples). This would provide much more insight to the reader. \n- It is difficult to spot the object movement in figure 7 c. It would be great if the authors could add some markers to help visibility.\n\n", "summary_of_the_paper": "This paper aims at learning object dynamics from unlabelled videos for multi-object representations. This work builds upon the inference model from Greff et al. (2019) to obtain the multi-object representations. Using that, this paper learns latent dynamics by predicting the future frame given two previous observations. The proposed network is trained by maximizing the log-likelihood in the pixel space for reconstructions as well as for predictions, similar to standard work in video prediction. In addition to that, this paper optimizes the objective from Greff et al. (2019) to obtain multi-object representations. The main contribution of this work lies in the module which predicts the future multi-object representation given the two past ones by using i) dynamic distillation as well as ii) a relation module. \nThe dynamic distillation module is motivated by the claim that the object-to-slot assignment switches inside videos. Therefore, the paper introduces a self-attention layer to match the slots of the specific objects across two frames. The output of this module after applying the feed-forward network is called dynamic representation. \\\nThe relation module is proposed to model the interactions between object pairs. This is implemented (10-13) using self-attention between the dynamics (from above) and static representation of objects i and k (slots). The output is added to update the dynamic representation from the dynamic distillation module. Ultimately, the future state is predicted using a linear transformation of the static representation of time t and the updated dynamic representation. \\\nThe paper evaluates their learned representation on the CLEVRER dataset for video reasoning, prediction, reconstruction, and segmentation.", "summary_of_the_review": "The modules proposed in this work are well motivated and designed and produce strong results on video reasoning. Yet, I am concerned about the insufficient evaluation. Without the ablations and the baseline I discussed in weaknesses it is difficult to quantify the actual impact of the proposed method and its underlying modules. Moreover, experimental details are missing which makes it difficult to reproduce its reported numbers. That is why my initial rating is below the acceptance threshold. But I am open to adjusting my rating if the authors do sufficiently address the points raised in weaknesses. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2-dYawISQpX", "reviewer": "Reviewer_xJFb", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n### Novelty \n- The two key contributions are clearly described:\n    - use explicit dynamic features learnt by matching objects’ latents between the current and previous frames as opposed to only learning static scene properties as done in prior work. \n    - explicitly model interactions between objects\n\n### Experimental verification \n- Small ablation on the effect of dynamic features in Table 2 \n\t - Even if dynamics features are absent the model achieves better results than IODINE which means that ODDN can both model object dynamics and generates overall higher quality object static representations. \n\n- Gives SoTa improvement across a variety of tasks\n    - Table 1 and Table 2 show a consistently significant improvement over prior work\n    -  ODDN Model representations are fed into ALOE instead of using static Monet features which ALOE originally used. This ODDN-ALOE model improves on state of the art on CLEVER for various tasks \n    - Interestingly the gains are greatest on predictive and counterfactual questions which are related to predicting how objects will move.\n    -  ODDN also improves over using object representations from IODINE or PROVIDE (in Table 2). \n    - Qualitatively, in Fig 4, the generated images for the next frames are better for ODDN as it is able to preserve the color of the objects as well as position whilst PROVIDE fails to preserve color after a few frames\n    - ODDN also does better at reconstructing images and generates more compact  segmentation masks than IODINE or PROVIDE (in Fig 2).\n\n# Weaknesses\n\n### Clarity of explanations and figures could be greatly improved\n- Although the ideas are clear, the method and implementation could be made clearer e.g. write out the algorithm in pseudo code \n    - In addition, the notation is slightly confusing given there are two meanings for t - inference model time step and time step in the video \n- Fig 7 (c) + section titled “Disentanglement” is not clearly explained\n    - The caption is incorrect - (b) is actually referring to (c). (a) is not explained.\n    - What is a dynamic dimension?\n    - You could add some arrows to the diagram to explain what is happening across the rows/columns. It’s not clear how to interpret the images.\n    - In addition where does “velocity magnitude” come from? This was not explained previously. \n- Figure 4 is also not very clear \n    - Explain in the caption what is happening in the 6 frames i.e. blue and brown cylinder collide, with ODNN this interaction is predicted but with PROVIDE it loses information about the blue cyclinder. It would be useful to explicitly mention the colors to look for in the images in the caption or in Section 4 in the text.\n    - The images are also really small - hard to see what is happening \n- Figure 3 \n    - Is it showing MSE from 6 frames in total or averaged over some number of runs?\n- Table 3\n    - For each metric add an arrow to indicate if lower or higher is better \n- Figure 6 is also not clear\n    - The caption indicating the colors for the frames and timestamps is really tiny \n\n### Lack of implementation details\n- No details about how the model was implemented or how to reproduce the results\n- Equation 2 and 3 explain what IODINE does but the ODDN method is not fully explained. \n\n### Typos:\n- Section 4.3: “weather” -> “whether”\n", "summary_of_the_paper": "The paper presents a method which takes as input sequences of video frames of scenes from which it is able to understand the dynamics of objects present in the scene and their interactions and transfer this to several downstream tasks. The model consists of two modules, one to distill individual object dynamics and a second relation module to understand interactions between objects.  The model is shown to have SoTA results on several downstream tasks including video understanding and reasoning, video prediction, reconstruction, and segmentation. The representations learnt by the model are shown improve upon prior work especially on tasks which require knowing object dynamics such as predicting the next frame in a sequence where collisions or occlusions can occur. ", "summary_of_the_review": "The paper presents a method with two key insights/contributions that are not present in prior work. The ideas are validated quantitatively and qualitatively. The technical details, figures and tables could be improved with better explanations and descriptions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1tMOkIvCC-", "reviewer": "Reviewer_icxn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\nThe authors provide a novel distillation method to understand object dynamics. The proposed system shows state-of-the-art performance on CLEVRER.\n\n[Weaknesses]\n- I would like to see more qualitative results on the attention module. Visualized coefficients in Fig. 6 only cover one single scenario. I think the paper needs more validations on whether the attention module converges well.\n\n- The proposed method is validated only in CLEVRER, which does not show its generalization capability. In addition, it is hard to find the differences in the qualitative results (Fig. 4) compared to the previous methods. In Fig. 5, IODINE cannot represent the shadow areas, while the proposed ODDN can reconstruct them. Could you elaborate on the reason how ODDN has a representation ability on the shadow areas?\n\n- I would like to see the comparison of the computational complexity with the previous methods, e.g., ALOE, IODINE, etc.\n\n- Please discuss the limitations of the proposed method. It is required to discuss whether there are any other aspects that can be further improved (as future works) for this task.\n", "summary_of_the_paper": "This paper presents an unsupervised distillation of disentangled object dynamics from raw video inputs. The distilled dynamics model is capable of causal reasoning and future frame prediction. Extensive results on tasks of segmentation and reconstruction show its favorable performance.", "summary_of_the_review": "Overall, the proposed architecture is designed simple and shows state-of-the-art performance. There are minor issues as commented in [Main Review], but I am leaning towards positive at this moment. Since the topic is out of my scope, I would like to see other reviewers' opinions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents a novel method h to learn object dynamics from unlabelled videos and shows its benefits on causal reasoning and future frame prediction.  This paper received 4 positive reviews and 1 negative review. In the rebuttal, the authors have addressed most of the concerns. AC feels this work is very interesting and deserves to be published on ICLR 2022. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make other necessary changes.
Decision: Accept (Poster)