Unsupervised Video Decomposition using Spatio-temporal Iterative Inference
Authors: Polina Zablotskaia
Edoardo Alberto Dominici
Leonid Sigal
Andreas Lehrmann
Venue:  (2021)
Keywords: Unsupervised Learning
Representation Learning
Scene Decomposition
Computer Vision

Abstract: Unsupervised multi-object scene decomposition is a fast-emerging problem in representation learning. Despite significant progress in static scenes, such models are unable to leverage important dynamic cues present in video. We propose a novel spatio-temporal iterative inference framework that is powerful enough to jointly model complex multi-object representations and explicit temporal dependencies between latent variables across frames. This is achieved by leveraging 2D-LSTM, temporally conditioned inference and generation within the iterative amortized inference for posterior refinement. Our method improves the overall quality of decompositions, encodes information about the objects' dynamics, and can be used to predict trajectories of each object separately. Additionally, we show that our model has a high accuracy even without color information. We demonstrate the decomposition, segmentation, and prediction capabilities of our model and show that it outperforms the state-of-the-art on several benchmark datasets, one of which was curated for this work and will be made publicly available.

Under review as a conference paper at ICLR 2021
UNSUPERVISED VIDEO DECOMPOSITION OPATIO-TEMPORE ITERATIVE INFERENCE
USING
Anonymous authors Paper under double-blind review
ABSTRACT
Unsupervised multi-object scene decomposition is a fast-emerging problem in representation learning. Despite significant progress in static scenes, such models are unable to leverage important dynamic cues present in video. We propose novel ppatio-tempora iterative inference framework that is powerful enough to jointly model complex multi-object representations and explicit temporal dependencies between latent variables across frames. This is achieved by leveraging 2D-LSTM, temporally conditioned inference and generation within the iterative amortized inference for posterior refinement. Our method improves the overall quality of decompositions, encodes information about the objects' dynamics, and can be used to predict trajectories of each object separately. Additionally, we show that our model has a high accuracy even without color information. We demonstrate the decomposition, segmentation, and prediction capabilities of our model and show that outperforms the state-of-the -ftth--rrr on several benchmark datasets, one of which was curated for this work and will be made publicly available.
INTRODUCTION
Unsupervised representation learning, which has long history dating back to Boltzman Machines (Hinton & Sejnowskil 1986) and original works of Marr (9770), has recently emerged as one of the important directions of research, carrying the newfound promise of alleviating the need for excessively large and fully labeled datasets. More traditional representation learning approaches focus on unsupervised (e.g., autoencoder- (Pathak al. 20166] Vincent et al. 2008) or self- supervised (Noroozi & Favaro 2016 Vondrick et al.) 2016 Zhang et al. 20166 learning of holistic representations that, for example, are tasked with producing (spatial (Norozzi & Favaro 2016).
temporal (Vondrick et al. 2016), or color (Zhang et al.] (0166) encodings of images or patches. The latest and most successful methods along these lines include ViLBERT (Lu et al., (019)) and others (Sun et al. 2019 Tan & Bansal (219)) that utilize powerful transformer architectures (Vaswani al.) (017)) coupled with proxy multi-modal tasks (e.g., masked token prediction or visua-lingual alignment). Learning of good disentangled, spatially granular, representations that are, for example, able to decouple object appearance and shape in complex visual scenes consisting of multiple moving objects remains elusive.
Recent works that attempt to address this challenge can be characterized as: (i) attention-basee methods (Crawford & Pineau| 2019b Eslami al. 2016), which infer latent representations for each object in a scene, and (ii) iterative refinement models (Greff et al.] 2019 2017). which decompose a scene into collection of components by grouping pixels. Importantly, the former have been limited to latent representations at object- or image patch-levels, while the latter class of models have illustrated the ability for more granular latent representations at the pixel (segmentation)-leve most refinement models learn pixel-level generative models driven by spatial mixtures (Greff et al. 2017] and utilize amortized iterative refinements (Marino et a 2018) for inference of disentangled latent representations within the VAE framework (Kingma & Welling 2014); a prime example is IODINE (Greff et al. 2019). However, while providing a powerful model and abstraction which is able to segment and disentangle complex scenes, IODINE (Greff et al. 2019) and other similar architectures are fundamentally limited by the fact that they only consider images. Even when applied for inference in video, they process one frame at time. This makes it excessively challenging to discover and represent individual instances of objects that may share properties such as appearance and shape but differ in dynamics.
Specifically,

---

Under review as a conference paper at ICLR 2021
Tnfenuce
7.
Decomposiion
Figure I: Unsupervised Video Decomposition. Our approach allows to infer precise segmentations of the objects via interpretable latent representations, that can be used to decompose each frame and simulate the future dynamics, all in unsupervised fashion. Whenever a new object emerges into a frame the model dynamical adapts and uses one of the segmentation slots to assign to the object.
In computer vision, it has been a long-held held belief that motion carries important information for segmenting objects (Jepson et al. 2002 Weiss & Adelson 1996). Armed with this intuition, we propose spatio-temporal amortized inference model capable of not only unsupervised multi-object scene decomposition, but also of learning and leveraging the implicit probabilistic dynamics of each object from perspective raw video alone. This achieved by introducing temporal dependencies between the latent variables across time. As such, IODINE (Greff et al. (019)) could be considered a special (spatial) case of our spatio-tempora formulation. Modeling temporal dependencies among video frames also allows us to make use of conditional priors (Chung et al. 2015) for variational inference, leading more accurate and efficient inference results.
The resulting model, illustrated in Fig. 1 achieves superior performance on complex multi-object benchmark datasets (Bouncing Balls and CLEVRER) with respect to state-of-the-arr models, includ- ing R-NEM (Van Steenkiste et al. 20188) and IODINE (Greff et al. 2019) in terms of segmentation, prediction, and generalization. Our model has a number of appealing properties, including tempo- ral extrapolation. computational efficiency, and the ability work with complex data exhibiting non-linear dynamics, colors, and changing number objects within the same video sequence. In addition, we introduce entropy prior to improve model's performance in scenarios where object appearance alone is not sufficiently distinctive (e.g., greyscale data).
RELATED WORK
Unsupervised Scene Representation Learning. Unsupervised scene representation learning can generally be divided into two groups: attention-based methods, which infer latent representations for each object scene, and more complex and powerful iterative refinement models, which often make use of spatial mixtures and can decompose a scene into collection of estimated components by grouping pixels together. Attention -baseddd-baddddd-e methods, such AIR (Eslami al., (016) (Xu et al. (019) and SPAIR (Crawford & Pineau 2019b), decompose scenes into latent variables representing the appearance, position, and size of the underlying objects. However, both methods can only infer the objects' bounding boxes and have not been shown work non-trivial 3D scenes with perspective distortions and occlusions. MoNet (Burgess et al. 2019) is the first model in this family tackling more complex data and inferring representations that be used for instance segmentation of objects. On the other hand, it is not probabilistic generative model and thus not suitable for density estimation. GENESIS (Engelcke al.) (2020) extends it and alleviates some of its limitations by introducing a probabilistic framework and allowing for spatial relations between the objects. DDPAE (Hsich) al. 2018 framework that uses structured probabilistic models decompose video into low -ww-dmmenionall temporal dynamics with the sole purpose of prediction. It is shown to operate on binary scenes with no perspective distortion and is not capable of generating per-object segmentation masks. Iterative refinement models started with Tagger (Greff et al.) 2016) that reasons about the segmentation of its inputs. However, it does not allow explicit latent representations and cannot be scaled to more complex images. NEM (Greff et al.) 2017). mixture model inside an expectation maximization framework, but is limited to binary data. Finally, IODINE (Greff et al. 2019) is notable example of model employing iterative amortized inference w.r.t. a spatial mixture formulation and achieves state-of- of-te-o--theart performance in scene decomposition and segmentation.
extension of Tagger, uses spatial
Unsupervised Video Tracking and Object Detection.
SQAIR
(Kosiorek et al. 2018). temporal extensions
SILOOT (Crawford & Pineau 2019a) and SCALOR (Jiang al. 2020
2

---

Under review as a conference paper at ICLR 2021
of the static titention-bbsed models that are tailored to tracking and object detection tasks. SQAIR is restricted to binary data and operates at the level of bounding boxes. SILOT and SCALOR are more expressive and can cope with cluttered scenes, a larger numbers of objects, and dynamic backgrounds, but do not work on colored perspective data; accurate segmentation remains a challenge.
Unsupervised Video Decomposition and Segmentation. Models employing spatial mixtures and iterative inference in temporal setting are closest to our method from a technical perspective. Notably, there are only few models falling into this line of work: RTagger (Prémont-SShwarc et al. 2017 is recurrent extension of Tagger and has same limitations as its predecessor. R-NEM ( (Van Steenkiste ct al., 2018) effectively learns the objects' dynamics and interactions through relational module and can produce segmentations but is limited to orthographic binary data.
Methods without Latent Modeling. GAN-based ReDO (Chen al. 2019) uses model built around the assumption that object regions are independent, guiding the generator by drawing objects' pixel regions separately and composing them after segmentation. Another model (Arandjelović & Zisserman 2019) employs the same principles but guide the generator by copying region of an image into another one. Both architectures are shown to operate on static images only and do not have a clearly interpretable latent space prediction capabilities.
Our method allows an effective use of temporal information in object-centric decompositions of colored video data. This places approach between methods like R-NEM, which strictly operates on binary data, and IODINE, whose usage of temporal information is ad-hoc and produces results of limited quality (Table 1). In practice, we leverage a 2D-LSTM and employ implicit modeling of dynamics by incorporating the hidden states into a conditional prior in the efficient runtime manner.
DYNAMIC VIDEO DECOMPOSITION
We now introduce our dynamic model for unsupervised video decomposition. Our approach builds upon generative model of multi-object representations and leverages elements of iterative amortized inference. We briefly review both concepts ($3.1) and then introduce our model ($3.2).
3.1 BACKGROUND
Multi-Object Representations. The multi-object framework introduced in Greff et al. (2019) decomposes a static image = (x;); E RR into K objects (including background). Each object is represented by latent vector (k) E RM capturing the object's 's unique appearance and can be thought of as an encoding of common visual properties, such as color, shape, position, and size. For each (R) independently. a broadcast decoder (Watters al. 2019) generates pixelwise pairs (m, (A) (M)) describing the assignment probability and appearance pixel for object k. Together, they induce the generative image formation model
D
K
2
1I
m
where (z (k)))) Ek=1 m (k) 1 and is the same and fixed for all and kt The original image pixels can be reconstructed from this probabilistic representation as 2, k== m M4
Iterative Amortized Inference. Our approach leverages the iterative amortized inference frame- work (Marino et al., 2018). which uses the learning to learn principle (Andrychowicz et al., 2016) to close the amortization gap (Cremer et al. (2017) typically observed in traditional variational inference. The need for such an iterative process arises due to the multi-modalin of Eq. which results in an order invariance and assignment ambiguity the approximate posterior that standard variational inference cannot overcome (Greff et al. 2019)..
The idea of amortized iterative inference is start with randomly guessed parameters X (k) for the approximate posterior qx (z (k) (x) and update this initial estimate through a series of R refinement
steps. Each refinement step E (1...., R) first samples latent representation from yx evaluate
Perspecttve videos more complex as objects occlude another and change in size over time.
3

---

Under review as a conference paper at ICLR 2021
the ELBO L and then uses the upproximate posterior gradients VxC to compute additive update fę. producing a new parameter estimate X (k)
A
qA
(k)
(x),
x + fo(a 1,1)),
where (() is function of z (2119). The function Sa consists of a sequence of convolutional layers and an LSTM. The memory unit takes as input hidden state b,dh) from the previous refinement step.
VxC, and additional inputs (mirrors definition in Greff al.
3.2 SPATIO-TEMPORA TEMPORAL ITERATIVE INFERENCE
Our proposed model builds upon the concepts introduced in the previous section and enables robust learning dynamic scenes through spatio-temporal iterative inference. Specifically, we consider the task of decomposing a video sequence x = (x+) = (zr,,)), T.D into K slot sequences (m, (/))
and K appearance sequences (1( To this end, we introduce explicit temporal dependencies into the sequence of posterior refinements and show how to leverage this contextual information during decoding with generative model. The resulting computation graph can be thought of as a 2D grid with time dimension / and refinement dimension r (Fig. 2ap. Propagation of information along these two axes is achieved with a 2D-L.STM (Graves et al. 2007) (Fig. (2b). which allows us to model the joint probability over the entire video sequence inside the iterative amortized inference framework. The proposed method is expressive enough to model the multimodality of our image formation process and posterior, yet its runtime complexity is smaller than that its static counterpart.
3.2.1 VARIATIONAL ORJECTIVE
Since exact likelihood training is intractable, we formulate our task in terms of a variational objective. In contrast to traditional optimization of the evidence lower bound (ELBO) through static encodings of the approximate posterior, incorporate information from two dynamic axes: (1) variational estimates from previous refinement steps; (2) temporal information from previous frames. Together, they form the basis for spatio-temporal variational inference via iterative refinements. Specifically, we train our model by maximizing the following ELBO objective
CELBO(X) (x Buy alage)) 1 r Blog (p (Xx1x R) (L,, Bets)) (
(3)
where the first term expresses the reconstruction error of single frame and the second term measures the divergence between the variational posterior and the prior. The relative weight between terms is controlled with hyperparameter 3 (Higgins al. 2017). Furthermore, to reduce the overall complexity of the model and to make it easier to train, set R == max R 4, 1) (see Fig. for an illustration). Compared to a static model, which infers each frame independently. reusing information from previous refinement steps also makes our model more computationally efficient. In the next sections, we discuss the form of the conditional distributions in Eq.() in more detail.
3.2.2 INFERENCE AND GENERATION
Posterior Refinement. Optimizing Eq. inside the iterative amortized inference framework (Sec- tion 3.1 requires careful thought about the nature and processing of the hidden states. While there is vast literature the propagation of single signal, including different types of RNNs (Hochreiter & Schmidhuberr 1997 Cho et al. 2014 Graves et al., 2005 Chung et al.) 2017) and transform- ers (Vaswani al.) (017). ), the optimal solution for multiple axes with different semantic meaning (i.e., time and refinements) is less obvious. Here, we propose to use a 2D) version of the uni-directional MD-LSTM (Graves et al., 2007) to compute our variational objective (Eq. 3 in an iterative manner. In order to do so, replace the traditional LSTM in the refinement network (Eq, 22) with 2D extension. This extension allows the posterior gradients to flow through both the grid of the previous refinements and the previous time steps (see Fig. 2ap. Writing 2t,r for the latent encoding at time t and refinement r. we can formalize this new update scheme as follows:
Ztr (2 )ettr)
A1,++1 Ae. fø(a, h h 1-1.
For simplicity, drop references to the object slot e (K) from now and formulate all equations per-slot hasis.
4

---

Under review as a conference paper at ICLR 2021
Ap.
Time
Wilkkalen slash
Gradictt Suntgling
A13
Tte feneeeent
Aua
hr
Rofiserent
STS
Special decodle
Ingipee
X1
IP2 27% 384 3t
Figure 2: Model Overview. (a) Inference in model passes through 2D grid in which light gray cell (r, t) represents the r-th refinement at time f, dark gray cells are where the final reconstruction is computed and refinement is needed. Each light gray cell receives three inputs: refinement hidden state h, a temporal hidden state h R- and posterior parameters A The outputs are a new hidden state her and new posterior parameters At.++1. (b) An example of the internal structure of the highlighted cell from Fig. (a). We process the inputs with the help of spatial broadcast decoder and 2D LSTM. The rest of the light gray cells have the same structure.
Note that the hidden state from the previous time step is always h 1, Rt i.e., the one computed during the final refinement at time - 1. Our reasoning for this is that the approximation of the posterior only improves with the number of refinements (Marino al. 2018).
Temporal Conditioning. Inside the learning objective we set the prior and the likelihood to be conditioned on the previous frames and the refinement steps. This naturally comes from idea that each frame is dependent on the predecessor's dynamics and therefore latent representations should follow the property. Conditioning on the refinement steps is essential to the iterative amortized inference procedure. To model the prior and the likelihood distributions accordingly we adopt the approach proposed Chung al. (2015) but tailor our iterative amortized inference setting. Specifically, the parameters of our Gaussian prior are now computed from the temporal hidden state h
p(ze|xee nece) (Zz ii, diag a?)).
[i,, ã. = Co (h,
(5)
where Eo is simple neural network with a few layers Please refer to the supplemental material for details. Note that the prior only changes along the time dimension and is independent of the refinement iterations, because we refine the posterior be as close as possible the dynamic prior for the current time step. Finally, to complete the conditional generation, we modify the likelihood distribution as follows
I1
(k) m ( o
(k)
(K) (mrrs Pi.dd go
(1)
(k)
kst.))
(K)
where Mi.r. M are mask and appearance of pixel i in slot k at time step t and refinement step r. go is a spatial mixture broadcast decoder (Greff et al. 2019) with preceding MLP to transform the
pair Zt.r 1 into single vector representation.
3.2.3 LEARNING AND PREDICTION
Architecture. From graphical point of view, we can think of the refinement steps and time steps as being organized on a 2D grid from Fig. 2a with light gray cell (r, t) representing the r-th refinement at
In practice, E predicts log ar for stability reasons.
"iinee likelihood is Gaussian mixture model, we now referencing the object slot again.

---

Under review as a conference paper at ICLR 2021
time t. According to Eq. 4 each such cell takes as input the hidden state from a previous refinement h the temporal hidden state h and the posterior parameters A,,- Outputs of each light gray cell are new posterior parameters xt,++1 and a new hidden state h At the last refinement R at time t, the value of the refinement hidden state htr is assigned to a new temporal hidden state R,
Training Objective. Instead of direct optimization of E3. we propose two modifications that we found to improve our model's practical performance: (1) similar to observations made by (Greff al. 2019), we found that color is an important factor for high-quality segmentations. In the absence (k) of such information, we mitigate the arising ambiguity by maximizing the entropy of the masks m i,, along the slot dimension k, i.e., we train our model by maximizing the objective
poservvttio
D
K
LELOO +
(K) m E.r
log(m, (r,e)
1=1
where y defines the weight of the entropy loss. (2) In addition to the entropy loss, we also prioritize later refinement steps by weighting the terms in the inner sum of Eq.33 with r It'
Prediction. On top of pure video decomposition, our model is also able to simulate future frames xI+1...., XT+T7. Because our model requires image data as input, which is not available during simulation of new frames, we use the reconstructed image X in place of x, to compute the likelihood p(xe ((xe(xer, (gt,r) in these cases. We also set the gradients VxL, W,C, and m to zero
Complexity. Our model's ability to reuse information from previous refinements leads to a runtime complexity of O(R2 which is much more efficient than the ( complexity of the traditional IODINE model (when each frame is inferred independently) in the typical case T >> R.
4 EXPERIMENTS
We validate our model on Bouncing Balls (Van Steenkiste et al. 2018). 3) an augmented version of
CLEVRER (Yi et al. 2020), and Grand Central Station (Zhou et al. 2012). Our experiments comprise quantitative studies of decomposition quality during generation and prediction, as well as an ablation study. Also see Appendices C Dnd
and F
4.1 SETUP
Datasets. Bouncing Balls consists of 50 frame, binary, 64 64 resolution video sequences. Each video shows simulated balls with different masses bouncing elastically off each other and the image border. We train our model on the first 40 frames of 50K videos containing 4 balls in each frame. We use two different test sets consisting of 10K videos with 4 balls and 10K videos with 6-8 balls. We also validate our model on a color version of this dataset that we generate using the segmentation masks.
CLEVRER contains synthetic videos of moving and colliding objects. Each video is 5 seconds long (128 frames) resolution 480 320, which we trim and rescale to 64 x pixels (see Appendix B). For training, we use the same 10K videos as in the original source. For testing,
compute ground truth masks for the validation set using the provided annotations and test on 2.5K instances containing 3-5 objects and on I.IK instances containing 6 objects. In the training, we set the number of slots K to for CLEVRER and to one more than the maximum number of objects in all other cases.
Grand Central Station is video feed from the main hall of a trafficked train station, containing high number of people moving at various paces in all different directions. It has a total 50010 frames in resolution of 720 x 480. In order make the dataset more manageable, portion of the feed of resolution 128 x 128 and segmented it into sequences of 20 frames each. Each sequence contains approximately 10 people. We set K to 8 during training and to 10 for testing,
have extracted
Baselines. We compare our approach to recent baselines: R-NEM (Van Steenkiste et al. 2018). IODINE (Greff et al.) 2019) and DDPAE (Hsieh al. 2018). R-NEM state-of-the-ard model for unsupervised video decomposition and physics learning. While showing impressive results on simulation tasks, it limited to binary data and has difficulties with perspective scenes. IODINE is more expressive but static in nature and cannot capture temporal dynamics within its probabilistic
'The method is robust enough handle 128x resolution as is built top of the TODINE.

---

Under review as a conference paper at ICLR 2021
Table 1: Quantitative Evaluation (Scene Decomposition). We show our model's ability to produce high-quality instance segmentations for sequences with varying length. We test on sequences with 4 balls and two different types of data (binary, colored) for Bouncing Balls and on sequences with 3-5 objects for CLEVRER. Note, R-NEM does not cope with color data; hence we only run it on binary.
Bouncing Balls
ARI (t)
F-ARI (1)
MSE (L)
Length
10
20
30
40
10
20
30
40
20
30
40
R-NEM
0,5031
0.6199
0.6632
0.6833
0.6259
0.7325
0.7708
0.7899
0.0252
0.0138
0.0096
0.0076
binury
JODINE
0,0318
0,9986
0.0018
SEQ-IODINE
0,0230
0.0223
0.0021
-0.0201
0.8645
0.6028
0.5444
0.4063
0.0385
0.0782
0.0846
0.0968
Our
0.7169
0.7263
0.7286
0.7294
0.9999
0.9999
0,9999
0.9999
0.0004
0.0004
0.0004
0.0004
color
IODINE
0.5841
0,9752
0.0014
SEQ-IODINE
0,3789
0.3743
0.3225
0.2654
0.7517
0.8159
0,7537
0.6734
0.0160
0.0164
0,0217
0.0270
Our
0.7275
0.7291
0.7298
0.7301
1,0000
1.0000
0,9999
0.9999
0.0002
0.0002
0.0002
0.0002
CLEVRER
ARI (t)
F-ARI (t)
MSE (L)
Length
10
20
30
40
10
20
30
40
10
20
30
40
color
JODINE
0.1791
0.9316
0,0004
SEQ-IODINE
0.1171
0.1378
0.1558
0,1684
0,8520
0,8774
0,8780
0,8759
0,0009
0,0009
0,0010
0,0010
Our
0.2220
0.2403
0.2555
0.2681
0,9182
0.9258
0.9309
0,9312
0.0003
0,0003
0.0003
0.0003
framework. However, as noted in Greff et al. (2019), can be readily applied to temporal sequences by feeding new video frame to each iteration of the LSTM in the refinement network. We call this variant SEQ-IODINE. Since we can perform simulation of short sequences, we include a comparison of the predictive power of our model against DDPAE (Hsieh et al. 2018).
4.2 EVALUATION METRICS
ARI. The Adjusted Rand Index RRan 1971 Hubert & Arabie 1985) is a measure of clustering similarity. It is computed by counting all pairs of samples that assigned the same or different clusters in the predicted and true clusterings. It ranges from -1 to 1, with score of 0 indicating random clustering and indicating a perfect match. We treat each pixel as one sample and its segmentation as the cluster assignment.
The Foreground Adjusted Rand Index is a modification of the ARI score ignoring background pixels, which often occupy the majority of the image. We argue that both metrics are necessary to assess the segmentation quality of a video decomposition method; this metric also used in (Greff et al. 2019 Van Steenkiste et al. 2018).
F-ARI.
MSE. The mean squared error between pixels of the reconstructed x and the ground truth frames x.
4.3
VIDRO DECOMPOSITION
We evaluate the models on a video decomposition task at different sequence lengths. As shown in Table our model outperforms the baselines regardless of the presence of color information, which further reduces the error. Our model performs at least 7% better than R-NEM on all metrics and at least 20% than IODINE on ARI and MSE. Since R-NEM cannot cope well with colored data or the perspective of scenes, it is only evaluated on the Bouncing Balls dataset (binary), producing high-error results in the first frames, a phenomenon not observed with our model. IODINE is not designed to utilize temporal information. On both datasets, IODINE's results are therefore computed independently on each frame of the longest sequence. By processing frames separately, IODINE does not keep the same object-slot assignment, which we ignore when computing the scores. SEQ-IODINE tends to perform even worse than IODINE in many experiments, which we attributed to exploding gradients caused by limited refinement steps and lack of dynamics modeling.
4.4 GENERALIZATION
We investigated how well our model adapts to a higher number of objects, evaluating its performance on the Bouncing Balls dataset (6 to 8 objects) and on the CLEVRER dataset (6 objects). Table 2 shows that our F-ARI and MSE scores are at least 50% better than those for R-NEM, and ARI scores are just marginally worse and only on the binary data. In comparison to IODINE we are at least 4% better across all metrics. For the Bouncing Balls dataset we have also investigated the impact of
7

---

Under review as a conference paper at ICLR 2021
Figure 3: Qualitative Evaluation (Bouncing Balls). Our model can generalize sequences with balls when trained on 4 balls. Top-to-bottom: output masks, reconstructions, and ground truth video.
Bouncing Balls
Bouncing Balls
CLEVRER
94
ABl Dut Ioaan
Cosine similariy
PAl Dut 1.2
ARI/FAA ARI
3.3
PAN KMM
ARIF-ARI
0.8
67
Simulation steps
Simulation steps
Simulation steps
Velocity predic- tions.Cosine similarity for simulated frames after 10 inference steps.
Figure 4:
Figure 5: Prediction. We show the (F-)ARI for 3, 5, and simulated frames after 20 inference steps.
changing the total number of possible colors 4 and 8. The former resulting in duplicate colors for different objects and the latter in unique colors for each object. The higher MSE scores for the 8 balls variant is due the model not being able to reconstruct the unseen colors. Sample qualitative results are shown in Fig. and 6 while more can be found in Appendix
4.5 PREDICTION
We compare the predictions of our model (Section 3.2.3) to those of R-NEM after 20 steps of inference on 10 predicted steps on the Bouncing Balls dataset (Fig. Fleff). As we can see from the results our model is superior to R-NEM on shorter sequences, however for the longer sequences we are outperforming R-NEM only on colored data. Our model is capable of more accurate frame prediction than R-NEM the Bouncing Balls dataset during the first few predicted frames (5-7), with predictions slowly diverging over time due to the temporal consistency. This behavior is also observable the CLEVRER dataset (Fig. right), albeit to lesser extent, likely because the objects dynamics are less, even if non-linear. In Figure 4 we compute velocity vectors between bounding box centroids and compare the cosine similarity to the predictions of DDPAE on the Bouncing balls dataset. As expected, our model outperforms DDPAE on the first three frames and then declines in quality. This behavior not surprising and in line with the results reported in Fig
4.6 ABLATION
The quantitative results of an ablation study on the binary Bouncing Balls dataset and CLEVRER are shown in Table We investigate the effects of the efficient grid, conditional prior and generation, length of training sequences and entropy term on the performance of our model; all contributions are necessary and important. Note that the base models are too large to be trained on 40 frames, which confirms the superiority of our model in terms of both runtime and memory. The CLEVRER dataset not binary, which is why we do not include the entropy term (see Section 3.2.3. We validate our choice of It and compare it to alternative options in a supplemental study discussed in Appendix F.2
CONCLUSION AND DISCUSSION
We presented novel unsupervised learning framework capable of precise scene decomposition in multi-object videos with complex appearance and motion. Our temporal component enables modeling of dynamics inside the amortized iterative inference framework but also improves the quality of the results overall. From our quantitative and qualitative comparisons with IODINE and SEQ-IODINE see that our model shows more accurate results the decomposition task. We detect new
8

---

Under review as a conference paper at ICLR 2021
Table 2: Generalization. At test time, we change the number of slots in the models from 5 to 9 for the Bouncing Balls test dataset (6-8 balls), and from to the CLEVRER test dataset objects).
Table 3: Ablation Study. A 2D-LSTM extension of IODINE trained on sequences of 20 frames is unstable and its output segmentation lacks precision and consistency. Our efficient version 2D-LSTM grid (Fig. 2a) and the condi- tional prior and generation increase both segmentation and reconstruction quality. By training these models on longer sequences of 40 frames we observe further improvements.
Bouncing Balls
R-NEM
(1)
0.4484
F-ARI (1)
0.6377
MSE (
0.0328
Base
Girid
CP4G
Entrupy
Length
Nnarr
IODINE
0.0271
0.9969
ARI (1)
F-ARI (1)
MSE (L)
SEQ IODINE
0,0263
0,8874
0.0521
0.0126
0.7765
0.0340
0,4453
0.9999
BB
20
0.2994
0.9999
0.0010
IODINE (4)
0,4136
0.8211
0.0138
40
0.3528
0.9998
0.0010
IODINE (8)
0.2823
0.7197
0.0281
0.7263
0.9999
0.0004
color
SEQ IODINE
0,2068
0.5854
0.0338
20
0,1900
0.8200
0.0011
SEQ JODINE (8)
0.1571
0.5231
0.0433
CLEVRER
0.1100
0,9000
0.0005
Our (4)
0.4275
0,9998
0.0004
0,2403
0.9258
0.0003
Our (8)
0,4317
0,9900
0.0114
40
0.1700
0,9100
0.0005
0.2681
0.9312
0.0003
CLEVRER
[Base: base model using 2D-LSTM; Grid: efficient
ART
F.ART (1)
MSF (3)
triangular grid structure (Fig. 2a CP+G:
conditional prior
color
LODINE
0,2205
0,9305
0.0006
and generation; Length: sequence length; Entropy: entropy
SEQ-IODINE
0,1482
0.8645
0.0012
term (Eq. 7
Our
0.2839
0.9355
0.0004
Figure 6: Qualitative Evaluation (CLEVRER). Our model can generalize to sequences with 6 objects. We also demonstrate the ahility to handle dynamically changing number of objects, ranging from 4 in the heginning at the end.
objects faster and are less sensitive to color, because our model can leverage the objects' motion cues. For our experiments, we have chosen a setup consistent with other SOTA methods and focus on the objects' dynamics. Our model is currently not targeting complex textured datasets, as they are not designed for unsupervised learning and impose additional challenges, such limited coverage of the input space as well as a superposition of the scene's ' intrinsic components (object location, articulation, motion, albedo, shading, etc.). We show and discuss the segmentation of real-world video stream in Fig- 7 Self-supervised methods for video segmentation, which attempt to learn image representations through features, motion or specific tasks, are an alternative but not capable of inferring disentangled representations or extracting interpretable features (e.g. appearance, color). They are typically also not robust object occlusions, object (dis)appearances, and object ordering (Fig. 14) We refer to Appendix E for an extended discussion and future work.
Sequences
Figure 7: Qualitative evaluation real-world data. Qualitative Evaluation (Grand Central Station). We observe that our method is very consistent separating the image regions belonging to different objects as they move in the scene. This dataset is particularly challenging for its background texture, complex lighting and shadows. Please zoom in to allow better clarity.

---

Under review as a conference paper at ICLR 2021
REFERENCES
M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, and N. de Freitas. Learning to learn by gradient descent by gradient descent. In NIPS, 2016.
Relja Arandjelović and Andrew Zisserman. Object discovery with copy copy-pasting gan. arXiv preprint vrXiv:1905.11366 2019.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:1901.11320 2019.
Mickaël Chen, Thierry Artières, and Ludovic Denoyer. Unsupervised object segmentation by redrawing. In Advances Neural Information Processing Systems, pp. 12726-12737, 2019.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mnn encoder-decoder for statistical machine translation. In EMNLP, 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio.
A recurrent latent variable model for sequential data. In NIPS, pp. 2980-2988, 2015.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. In ICLR, 2017.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. In AAAI, 2019a.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In AAAI, pp. 3412-3420, 2019b.
C. Cremer, X. Li, and D. Duvenaud. Inference suboptimality in variational autoencoders. In NIPS Workshop on Advances in Approximate Bayesian Inference, 2017.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In ICLR, 2020.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. NIPS, pp- 3233,, 2016.
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Bidirectional Istm networks for improved phoneme classification and recognition. In Włodzislaw Duch, Janusz Kacprzyk, Erkki Oja, and Slawomir Zadrožny (eds.), Artificill Neurul Networks: Formal Models and Their Applications ICANN 2005, pp. 799-8094 Berlin, Heidelberg, 2005. Springer Berlin Heidelberg. ISBN 978-3- 540-28756-8.
Alex Graves, Santiago Fernánde7, and Jürren Schmidhuber. Multi-dimensional recurrent neural networks. International conference on artificial neural networks, pp. 549-558. Springer, 2007.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In NIPS, pp. 4484-4492, 2016.
Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In NIPS, pp. 6691-6701, 2017.
Klaus Greff, Raphaöl Lopez Kaufmann, Rishab Kabra, Nick Watters, Chris Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning 2019.
with iterative variational inference. arXiv preprint
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
10

---

Under review as a conference paper at ICLR 2021
G.E. Hinton and T.J. Sejnowski. Learning and relearning in boltzmann machines. Parallel Distributed Processing: Explorations in the Microstructure of Cognition, 1:282-317, 1986.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8): 1735-1780, 1997.
Jun-Ting Hsieh, Bingbin Liu, De-An Huang, Li Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. CoRR, abs/1 886.041666 2018.
URL httthtttp/aaxa.t org/ wtwpg/abs//88..000 04166
Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of classification, 2(1):193-218, 1985.
A. Jepson, D. Fleet, and M. Black. A layered motion representation with occlusion and compact spatial support. In ECCV, pp. 692-706, 2002.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In ICLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In NeurIPS, pp. 8606-8616, 2018.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-land vision-and-language tasks. In NeurlPS, 2019.
Joseph Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In ICML, 2018.
D. Marr. A theory for cerebral neocortex. Proceedings of the Royal Society of London, Series B(176): 161-234, 1970.
M. Noroozi and P. Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In ECCV, 2016.
D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros. Context encoders: Feature learning by inpainting. In CVPR, 2016.
Isabeau Prémont-Schwarz, Alexander Ilin, Tele Hao, Antti Rasmus, Rinu Boney, and Harri Valpola. Recurrent ladder networks. In NIPS, pp. 6009-6019, 2017.
William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical association, 66(336):846-850 1971.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. simple neural network module for relational reasoning. In NIPS, pp. 4967-4976, 2017.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmidt. Videobert: joint model for video and language representation learning. In ICCV, 2019.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In Conference on Empirical Methods Natural Language Processing, 2019.
Sjoerd Van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In ICLR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, pp. 5998-6008, 2017.
11

---

Under review as a conference paper at ICLR 2021
P. Vincent, H. Larochelle, Y. Bengio, and P. A. Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008.
C. Vondrick, H. Pirsiavash, and A. Torralba. Anticipating visual representations with unlabeled videos. In CVPR, 2016.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. arXiv preprint arXiv:1901.07017, 2019.
Y. Weiss and E. Adelson. A unified mixture framework for mo- tion segmentation: Incorporating spatial coherence and estimating the number of models. In CVPR, pp. 321-326, 1996.
Kun Xu, Chongxuan Li, Jun Zhu, and Bo Zhang. Multi-objects generation with amortized struc- tural regularization. CoRR, (bss/1906.0922/. 2019. URL ttttp://arxiie wrg. /abs//////// /1906
03923
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. In ICLR, 2020.
R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In ECCV, 2016.
B. Zhou, X. Wang, and X. Tang. Understanding collective crowd behaviors: Learning mixture model of dynamic pedestrian-agents In 2012 IEEE Conference Computer Vision and Pattern Recognition, pp. 2871-2878, 2012. doi: 10. 1,111///9R22221222288011
A BASELINES
A.I R-NEM
We use the R-NEM (Van Steenkiste et al. 2018) authors' original implementattion and their publicly available models: https:://qtthii com/ http://www.....o.oooeeeetettette NEM
A.2 IODINE
Our IODINE experiments are based on the following PyTorch implementation: https: //qi thub com/ /in chae Weeeeeennellllleeeeeeeeeeeaealll MeelllyyIODIINN IODINE' We use the same parameters in this code, with the exceptions of 3 = 10 (weight factor) and, for the Bouncing Balls experiments. R = 6 (refinement steps). The majority of the hyperparameters shared between our own model and IODINE are identical.
A.3 SEQ-IODINE
In order to the sequential version of IODINE, we use the regularly trained IODINE model but change the number of refinement steps to the number of video frames during testing. During each refinement step, instead of computing the error between the reconstructed image and the ground truth image, we use the next video frame. Since the IODINE model was trained on R = 6 refinement steps, extending the number of refinement steps to the video length leads to exploding gradients. This effect especially problematic in the binary Bouncing Balls dataset with 20, 30 and 40 frames per video, because the scores of the static model are already low. We deal with this issue by clamping with max 10 and min - 10 the gradients and the ó refinement value in this experimen 5 SEQ-IODINE weak performance, especially w.r.t. the ARI, reflect the gradual divergence from the optimum as the number of frames increases.
B DATASETS
Bouncing Balls. Bouncing Balls is dataset provided by the authors of R-NEM (Van Steenkiste et al. 2018). Dataset contains balls with different masses corresponding to their radii. The balls are
'llease that clamping was done only when applied hinary Bouncing Balls for 20, 30 and 40 frames.
12

---

Under review as a conference paper at ICLR 2021
initialized with random initial positions, masses and velocities. Balls bounce elastically against each other(without occlusions) and the image window. We use the train and test splits of this dataset in two different versions: binary and color. For the color version, 4-balls (sub-)datased For the 6-8 balls test data, we color them in 2 different ways: 4 colors (same as train) and 8 colors (4 from train, 4 new ones). Note that the former results in identical colors for multiple objects, while the latter guarantees unique colors for each object.
randomly choose 4 colors for the
CLEVRER. Each video in the CLEVRER dataset contains least one collision and (dis)appearence event making occlusions possible and frequent. Objects' initial velocities are approximately +2.5 m/s². Each object has one of eight distinct colors and one of 38 two materials (metal or rubber). In addition, two objects can have the same color but different material.
The version of the CLEVRER dataset (Yi et al. 2020) used in this work was processed as follows:
Train split, validation split and validation annotations were obtained from the official website:
http://clevret csail wetaailmit.......mmmm .mi..edu/ We use the validation set as test set, because the test set does not contain annotations.
For training, we use the original train split. Our minimal preprocessing consists of cropping the frames along the width axis by 40 pixels on both sides, followed by uniform downscal- ing to 64x64 pixels. Since the length of each video is 128 frames and the maximum number of frames during training was 40, we split the videos into multiple sequences to obtain larger number of training samples.
For testing, we trim the videos to a subsequence containing at least 3 ob- jects and object motion. We compute these subsequences by running the script (slice_videos farecossfrom__ee annotations.py in the attached code) from the folder with the validation split and validation annotations.
The test set ground truth masks can be downloaded from here The masks and the prepro- cessed test videos will be grouped into separate folders based on the number of objects in a video.
C HYPERPARAMETERS
Initialization. We initialize the parameters of the posterior À by sampling from (.55, 0.5). In all experiments, we use latent dimensionality dim(z) 64, such that dim(A) 128. Horizontal and vertical hidden states and cell states are of size 128, initialized with zeros. qΛ is the posterior probability per slot of the likelihood p(x|z), which is Gaussian mixture model. The variance of the likelihood is set to σ = 0.3 in all experiments.
Experiments on Bouncing Balls. For this experiment, we have explored several values of R (refinement steps) and empirically found R = 6 to be optimal in terms of accuracy and efficiency. Refining the posterior more than times does not lead to any substantial improvement, however, the time and memory consumption is significantly increased. For the 4-balls dataset, we use K = 5 slots for train and test. For our tests on 6-8 balls, we use K = 9 slots. This protocol is identical to the one used in R-NEM (Van Steenkiste et al. 2018). Furthermore, we set 3 = 100.0 and scale the KL term by vi 10. The weight of the entropy term is set to y = 0.1 the binary case. As expected, the effect of the entropy term is most pronounced with binary data, so set y 0 in all experiments with RGB data.
Experiments CLEVRER. We keep the default number of iterative refinements at R 5, because we did not observe any substantial improvements from a further increase. We use = 6 slots during training, K = 6 slot when testing on 3-5 objects and K = 7 slots when testing on 6 objects.
D TRAINING
We use ADAM (Kingma & Ba 2014) for all experiments, with a learning rate of 0.0003 and default values for all remaining parameters. During training, gradually increase the number of frames per video, as we have found this to make the optimisation more stable. We start with sequences of length 4 and train the model until we observe a stagnant loss or posterior collapse. At the beginning
13

---

Under review as a conference paper at ICLR 2021
of training, the batch size is 32 and is gradually decreased negatively proportional to the number of frames in the video.
D.1 INFRASTRUCTURE AND RUNTIME
We train our models on 8 GeForce GTX 1080 Ti GPUs, which takes approximately one day per model.
D.2 CODE
We have attached the code and the pretrained models to reproduce the experimental results. Please see README file in the code folder help you with running.
E DISCUSSION AND FUTURE WORK
Introduction temporal component not only enables modelling of dynamics inside the amortized iterative inference framework but also improves the quality of the results overall. From our quantitative and qualitative comparisons with IODINE and SEQ-IODINE, we see that our model shows more accurate results on the decomposition task. We can detect new objects faster and are less sensitive to color, because our model can leverage the objects' motion cues. The ability to work with complex colored data, a property inherited from IODINE, means that we significantly outperform R-NEM. However, R-NEM is a stronger model when it comes to prediction of longer sequences, owing to its ability model the relations between the objects in the scene. Similar ideas were used in SQAIR (Kosiorek et al. 2018) and GENESIS (Engelcke et al. 2020) by adding a relational promising direction
RNN (Santoro al. 2017). Integration of these concepts into framework for future research. Another possible route is an application of our model to complex real-world scenarios. However, given that such datasets typically contain a much higher number of objects, as well as intricate interactions and spatially varying materials, we consider the resulting scalability questions separate line research.
F ADDITIONAL EXPERIMENTS
F. PREDICTION
Bouncing Balls
CLEVRER
3.0
2
1.2
10.3
Our
(cclor)
NEM
1.0
2.0
0.9
MSE
W
1.5
0.8
0.6
0.5
10
Simulation steps
Simulation steps
Figure 8: Mean Squared Error for the prediction experiment. We have computed MSE for the same experimental set up as on Fig. 5 As expected the MSE increases with number of simulation steps. Similarly to ARI and F-ARI scores our model outperforms R-NEM on first steps of simulation. however the error function of our model is grows faster comparatively to R-NEM and we sooner diverge from the accurate simulation.
14

---

Under review as a conference paper at ICLR 2021
F.2 ABLATIONS
The performance of our model is governed by the function R = max R - 1, 1), where R is the free parameter. In Table we explore values of R ranging from to 10. We performance saturation R = We also explore an alternative choice Rail (Table Sp. which shows decreased performance compared to R. The number of slots could be determined via cross-validation, but for comparability to other SOTA methods assume it to be given.
Table 4: Performance as a Function of Parameter R.
ARI (t)
F-ARI (1)
MSE ())x10 10 ")
R
2
4
8
10
2
8
10
2
4
8
10
BB bin.
0.34
0.71
0.73
0.73
0.93
0.99
0.99
0.99
424
6
5
BB col.
0.48
0.72
0.73
0.73
0.93
0.99
1.0
1.0
148
3
3
CLEVRER
0.21
0.24
0.23
0.22
0.84
0.92
0.93
0.94
11
3
3
3
Table 5:
Ablation on the Form of the Function R. Ru.. = R. when O. and Ran. 1.
when > 0.
ARI (t)
F-ARI (t)
MSE (L)(x10-4)
R
R
R
Ratt
R
Butt
BB bin.
0.73
0.43
1.0
0.95
4
33.2
BB col.
0.73
0.57
1.0
0.97
2
11.9
CLEVRER
0.24
0.21
0.93
0.88
F.3 ADDITIONAL QUALITATIVE RESULTS
Figure 9: Video decomposition using our model applied on Bouncing Balls dataset with balls.
m0
Figure 10: Video decomposition using our model applied on Bouncing Balls dataset with 6-8 balls.
15

---

Under review as a conference paper at ICLR 2021
Simnlation
9
Figure 11: Prediction on Bouncing Balls (colored) dataset.
Simmlaaiio
1g
&
Figure 12: Prediction on CLEVRER dataset.
30
30
(b)
Figure 13: Qualitative results for Ours IODINE vs. SEQ-IODINE decomposition experiment. (a) From the figure it is clear that our model can much sooner detect new objects emerging to the frame, while SEQ-IODINE struggles properly reconstruct and decompose them. And IODINE doesn't have any temporal consistence and reshuffles the slot order. (b). Here we can see that our model is much more stable with time and does fail detect objects, unlike IODINE and SEQ-IODINE.
16

---

Under review as a conference paper at ICLR 2021
Stall puppl splem
ble wato ribem
Oolor
10
70
70
70
10
10
10
1
10
10
Size
70
7o
a
io
a
10
10
10
49
70
70
10
10
70
Position
40
10
4
70
40
Puppi cylade
70
70
a
70
70
42
10
70
10
69
10
10
10
10
70
Cokor
Size
10
ap
ab
70
a
P
70
10
7B
7g
Poition
Color
i
a
49
70
Size
io
c
Position
2
1o
1o
9
70
7o
10 0
70
-25
2.5
-2.5
(1
2.5
(a)
(b)
Figure 14: Disentanglement of the latent representations corresponding to distinct interpretable features. CLEVRER latent walks along three different dimensions: color, size and position. We chose random frame and for each object's representation in the scene dimensions were traversed independently.
F.4 DISENTANGLEMENT
We demonstrate that introducing a new temporal hidden state and an additional MLP in front of the spatial broadcast decoder has not impacted its ability separate each object's representations and disentangles them based on color, position, size and other features, similar to results shown Greff
al. 2019).
F.5
A ANIMATIONS
Attached animations include the following files:
bb binary 4 balls.giff Animation of the segmentations of 4 binary Bouncing Balls. 50 frames. Here and everywhere else, unless explicitly specified, we also included full scene decomposition and each object's individual reconstruction.
bb jo__biarr__de 6 8 3_allls.gif Animation of the ability to generalize to 6-8 binary Bouncing Balls. 40 frames.
bb _b__ollred__ 4 balls.gif Animation of the 4 colored Bouncing Balls. 50 frames.
bb colored 6 8 balls.gif' Animation of the ability generalize to 6-8 colored Bouncing Balls. 40 frames.
bb co_collred__ prroreddpredic..gfff Prediction on the Bouneing Balls colored data. With 40 normal steps of inference and 10 predicted masks and frames. Here we only included predicted masks and ground truth masks.
Baorer_5obj..e Animation of the segmentations of 5 objects CLEVRER dataset. 50 frames.
17

---

Under review as a conference paper at ICLR 2021
clevrer clevrer_6obj.gif Animation of the ability to generalize to 6 objects CLEVRER dataset. 45 frames.
18


Reviews:
[{"review_id": "qRd1qEtUQZ-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a model for the unsupervised decomposition of videos into objects. It builds on\nprevious models operating on individual images such as IODINE and GENESIS, and extends them to the\nvideo domain by conducting iterative inference also across the time dimension, leading to a\ncomputation graph resembling a 2D grid. It is shown that the proposed model provides more accurate\nscene decompositions than previous ones, and generalizes better to different numbers of objects.\nPredictions of future frames appear somewhat less accurate.\n\n\nStrengths:\n 1. The proposed method closes an important gap in the landscape of unsupervised object-based\n    models: It is the first such method to decompose colored videos using pixel-wise segmentation\n    masks. The method is well motivated and clearly presented.\n 2. The experimental evaluation follows the precedent set by R-NEM, IODINE, and others. It appears\n    to have been rigorously conducted and shows encouraging results in terms of scene decomposition\n    and generalization. The reduced reliance on color information, which has plagued previous models\n    such as MoNET, is especially welcome.\n 3. The main technical contribution, the two-dimensional refinement method, seems to provide clear\n    benefits over previous ad-hoc implementations such as SEQ-IODINE, both in terms of runtime and\n    performance.\n\nWeaknesses:\n 1. As noted by the authors, predictive performance is somewhat lacking, which may be explained by\n    the lack of relational components. \n 2. Using the inference network as opposed to the generative model for prediction seems to run\n    counter to the idea of learning a generative model to accurately model the distribution of the\n    data. In particular, because the generative distribution $p(z_t | x_{<t}, z_{<t})$ seems ill-equipped\n    to model object interactions.\n 3. Like all unsupervised object-models, this work is still quite far away from working on natural\n    videos. \n\nOverall, I believe this paper fills an important gap that many members of the community were waiting\nto see filled. It is well executed and adds some interesting ideas to the toolset of unsupervised\nobject models. I therefore recommend acceptance.\n\nQuestions:\n 1. It seems that during inference, interactions between slots may be taken into account through the\n    gradients of the likelihood. But during prediction, those are set to zero, and the prior\n    over $z_t$ seems to operate slot-wise as well. So can object interactions be computed at all\n    during prediction, or are the slots predicted independently of one another?\n 2. The choice of $\\hat{R}$ seems to express the idea that once a good latent state has been inferred,\n    a single refinement step is enough to update it given new observations. Would additional steps\n    be beneficial when receiving unexpected input, such as frames with newly appearing objects on\n    CLEVRER? Could the number of steps perhaps even be selected dynamically based on e.g. the\n    reconstruction error?\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "jTZtCJoBra", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose to better explicitly utilize the sequential information in the video to improve the performance of unsupervised scene decomposition in video. Concretely, 2D LSTM is used to combine the advantages of iterative inference and temporal information. By appropriately using the inferred results in the previous time step, the number of interactive inference steps is decreasing, which finally results in the O(R^2+T) complexity. \n\nThe main concern about this paper is the novelty. First, unsupervised video decomposition, like Sqair and R-NEM all use the temporal information, and the iterative inference method has been used in previous work like IODINE. It seems that this paper extends the IODINE to the video setting with a modification that decreasing the iterative steps as time increases. The authors should better illustrate their novelty and contributions of this paper.\n\nThe presentation is generally clear in this paper. However, the figures can be improved. First, figure 1 seems meaningless. I don’t the relationship between the top row and the bottom row in the top panel of figure 1. The captions said about the emergence of new objects, but I cannot find them in the figure. Highlight them with some red bounding boxes is helpful. Second, panel (b) of Figure 2 is really chaotic. I don’t know what’s the main purpose of this panel. Again, highlight the part you want to emphasize.\n\nFinally, there are some related but not discussed work:\n[a] Chen, Mickaël, Thierry Artières, and Ludovic Denoyer. \"Unsupervised object segmentation by redrawing.\" Advances in Neural Information Processing Systems. 2019.\n[b] Arandjelović, Relja, and Andrew Zisserman. \"Object discovery with a copy-pasting gan.\" arXiv preprint arXiv:1905.11369 (2019).\n[c] Xu, Taufik, et al. \"Multi-objects generation with amortized structural regularization.\" Advances in Neural Information Processing Systems. 2019.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "a7qILVg9CW", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors extend previous work of Greff et al. on unsupervised, multi-object scene decomposition to incorporate temporal information.  In particular, they apply the LSTM defined for each candidate object not only over inference steps, but also over time. This allows the model to capture temporal cues, such as object motion, to better decompose the scene into objects. In addition, this allows to speed up the inference, since they perform fewer inference steps at each consecutive frame, capitalizing on temporal consistency in videos (LSTM state can be largely refused between consecutive frames, since the appearance doesn't change a lot). Experimentally demonstrate that their approach indeed outperforms prior work on two toy datasets (bouncing balls and CLEVERER), while being more computationally efficient.\n\nThe paper is relatively well written, although it is very dense and is hard to follow for someone not already familiar with the field.\n\nI'm not an expert in iterative amortized inferences, so can't properly judge the technical contribution of the paper. That said, the novelty is non-trivial to the best of my knowledge.\n\nExperimental evaluation is sufficient to compare to the closely related approaches Greff et al., and  Van Steenkiste et al., demonstrating that the proposed method outperforms them when evaluation setting (number of objects in a video) matches the training setting. In a generalization experiments, however, when the number of objects in increased at test time, the gap between the methods becomes much smaller. Moreover, the proposed method seems to generalize worse than Van Steenkiste et al. A reasonable ablation study is provided to justify the design decisions made by the authors.\n\nMy main issue is with the positioning of the paper. The authors claim that their goal is learning disentangled, multi-object representations, but all the evaluations are on the task of unsupervised video segmentation on toy datasets. As the authors mention in the conclusion, the problem of unsupervised video segmentation is well studied in the computer vision community, featuring both heuristic-based (Brox and Malik, ECCV'10), and learning-based methods (Xie et al., CVPR'19, Dave et al., ECCV'19 Workshops). Unlike the proposed method, those approaches work on real videos in the wild, though they indeed do not claim to learn an interpretable, disentangled representation. In might well be that this direction of research has merit, even if existing method cannot compete with dedicated video segmentation approaches yet, but this has to be experimentally demonstrated. If video segmentation is indeed not the end goal of your approach, when it shouldn't be used as the main evaluation metric. In particular, when comparing to  Greff et al., and  Van Steenkiste et al., you should demonstrate that the representation learned by your method is more disentangled and more interpretable than theirs, instead of showing that you are better at video segmentation. Indeed, there is no evidence that the two metrics are strongly correlated. In addition, this would allow you to compare your method to other, not object decomposition-based methods for learning disentangled representations. \n\nAs mentioned above, I cannot adequately judge the technical contribution of the paper, so, if other reviewers find it convincing in itself, I'm ready to increase my score. The experimental evaluation of the actual task studied in the paper (learning disentangled representations) is completely missing, however, thus I recommend to reject the paper in this round.\n\n\nSome minor comments: \n\n1. Why not use mask IoU for segmentation evaluation?\n\n2. Why are ARI scores higher on CLEVERER in Table 2 compare to Table 1 (generalization scenario with more objects)?\n\n3. Please discuss and compare to Hsieh et al., NIPS'18, who also claim to learn disentangled representations via unsupervised video decomposition.\n\n\n\n\nIn the rebuttal the authors have argued that their method studies a different problem from that of unsupervised video segmentation, and thus a comparison to those methods is unnecessary. Moreover, they mentioned that no established metrics exist for evaluating the degree of disentanglement of a representation, thus no new experiments can or should be added to the paper. I can see that there is a difference between image-based scene decomposition and unsupervised video segmentation. Once you move to video-based methods, however, the difference start to elude me. At least in the classical works, such as Brox and Malik, ECCV'10, the problem is defined in exactly the same way - decomposing a video into object/background regions in a fully unsupervised way. The only difference I can see is that in those works decomposition was the end goal, whereas this paper attempts to use it as a surrogate task for representation learning. This would be a valuable contribution if the authors could show that the resulting representations are superior to those learned with other unsupervised objectives (say, contrastive learning) at least for some tasks (say, object detection). Unfortunately, such evaluation is missing from the paper, thus I still find find that the benefits of the proposed approach are not convincingly demonstrated.", "extra_scores": {}, "extra_text": {}}, {"review_id": "1pQijHm_OQ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new model for unsupervised video decomposition based on the multi-object scene representation of IODINE. Basically, it makes use of 2D-LSTMs to provide the IODINE framework with a better ability to capture temporal dependencies.\n\n1.  My first concern is about the novelty of the proposed model. The most important contribution, if I understand it correctly, is the use of 2D-LSTMs in the iterative amortized inference for temporal modeling. It does extend IODINE into the spatiotemporal context, but it only provides limited new insight into this research field.\n\n2. Another concern is about the CLEVRER experiments. From Table 2, a counter-intuitive observation is that IODINE significantly outperforms Seq-IODINE. There might be two reasons: (a) In this dataset, the temporal dependencies between frames are relatively weak, because future frames depend not only on previous ones but also on a series of actions. (b) Seq-IODINE may not be a strong baseline model; If so, the authors might include other existing approaches specifically designed for video data, such as R-NEM and DDPAE [Hsieh et al., 2018].\n[Hsieh et al., 2018] Learning to Decompose and Disentangle Representations for Video Prediction.\n\n3. Although I am aware that R-NEM and IODINE were also evaluated on the synthetic datasets only, I recommend that the authors validate the model on real-world datasets of non-rigid objects and more complex structures.\n\n4. If the authors can provide a direct qualitative comparison with R-NEM and IODINE in Figures 3-4, it will be easier to understand the advantages of the proposed model.\n\n5. In Section 2, the authors could use a separate paragraph to describe the main differences between the proposed model and previous ones, in particular R-NEM and IODINE.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the spatio-temporal formulation of amortised iterative inference.
However, the paper does not clearly state what is the end goal: if the end goal is video object segmentation, it should compared against other unsupervised object segmentation methods. If the goal is representation learning, it should evaluate the merit of the recovered representations, e.g. by fine-tuning them on some downstream task. 
Decision: Reject