Benchmarking Unsupervised Object Representations for Video Sequences
Authors: Marissa A. Weis
Kashyap Chitta
Yash Sharma
Wieland Brendel
Matthias Bethge
Andreas Geiger
Alexander S Ecker
Venue:  (2021)
Keywords: Unsupervised learning
object-centric representations
benchmark
tracking

Abstract: Perceiving the world in terms of objects and tracking them through time is a crucial prerequisite for reasoning and scene understanding. Recently, several methods have been proposed for unsupervised learning of object-centric representations. However, since these models have been evaluated with respect to different downstream tasks, it remains unclear how they compare in terms of basic perceptual abilities such as detection, figure-ground segmentation and tracking of individual objects. To close this gap, we design a benchmark with three datasets of varying complexity and seven additional test sets which feature challenging tracking scenarios relevant for natural videos. Using this benchmark, we compare the perceptual abilities of four unsupervised object-centric learning approaches: ViMON, a video-extension of MONet, based on a recurrent spatial attention mechanism, OP3, which exploits clustering via spatial mixture models, as well as TBA and SCALOR, which use an explicit factorization via spatial transformers. Our results suggest that architectures with unconstrained latent representations and full-image object masks such as ViMON and OP3 are able to learn more powerful representations in terms of object detection, segmentation and tracking than the explicitly parameterized spatial transformer based architecture of TBA and SCALOR. We also observe that none of the methods are able to gracefully handle the most challenging tracking scenarios despite their synthetic nature, suggesting that our benchmark may provide fruitful guidance towards learning more robust object-centric video representations.

Under review as a conference paper at ICLR 2021
BENCHMARKING UNSUPERVISED OBJECT REPRESEN- TATIONS FOR VIDEO SEQUENCES
Anonymous authors Paper under double-blind review
ABSTRACT
Perceiving the world in terms of objects and tracking them through time is a crucial prerequisite for reasoning and scene understanding. Recently, several methods have been proposed for unsupervised learning object-centric representations. How- ever, since these models have been evaluated with respect different downstream tasks, it remains unclear how they compare in terms of basic perceptual abilities such as detection, figure-groumd To close this gap, we design a benchmark with three datasets varying complexity and seven additional test sets which feature challenging tracking scenarios relevant for natural videos. Using this benchmark, we compare the perceptual abilities of four unsupervised object-centric learning approaches: VIMON, a video-extension of MONET, based on recurrent spatial attention mechanism. OP3, which exploits clustering via spatial mixture models, as well as TBA and SCALOR, which use explicit factorization via spatial transformers. Our results suggest that architec- tures with unconstrained latent representations and full-image object masks such as VIMON and OP3 are able to learn more powerful representations in terms object detection, segmentation and tracking than the explicitly parameterized spatial transformer based architecture of TBA and SCALOR. We also observe that none of the methods are able to gracefully handle the most challenging tracking scenarios despite their synthetic nature, suggesting that our benchmark may provide fruitful guidance towards learning more robust object-centric video representations.
segmentation and tracking of individual objects.
INTRODUCTION
Humans understand the world in terms of objects. Being able to decompose our environment into independent objects that can interact with each other is an important prerequisite for reasoning and scene understanding. Similarly, an artificial intelligence system would benefit from the ability to both extract objects and their interactions from video streams, and keep track of them over time.
Recently, there has been an increased interest in unsupervised learning of object-centric represen- tations. The key insight of these methods is that the compositionality of visual scenes be used to both discover (Eslami et al.] 2016) Greff et al. 2019 Burgess et al., 2019) and track objects in videos (Greff al. [0117] 7 van Steenkiste et al.) 20188 Veerapaneni al. 2019) without supervision. However, it is currently not well understood how the learned visual representations of different models compare to each other quantitatively, since the models have been developed with different downstream tasks in mind and have not been evaluated using a common protocol. Hence, in this work, we propose benchmark based on procedurally generated video sequences to test basic perceptual abilities of object-centric video models under various challenging tracking scenarios.
unsupervised object-based video representation should (/) effectively identify objects as they enter scene, (2) accurately segment objects, as well as (3) maintain consistent representation for each individual object a scene over time. These perceptual abilities can evaluated quantitatively the established multi-object tracking framework (Bernardin & Stiefelhagen 2008) Milan et al. 2016). We propose to utilize this protocol for analyzing the strengths and weaknesses of different object- centric representation learning methods. independent of any specific downstream task, in order to uncover the different inductive biases hidden in their choice of architecture and loss formulation. We therefore compiled benchmark consisting of three procedurally generated video datasets of varying levels of

---

Under review as a conference paper at ICLR 2021
visual complexity and two generalization tests. Using this benchmark, quantitatively compared three classes of object-centric models, leading to the following insights:
All of the models have shortcomings handling occlusion, albeit to different extents.
OP3 VVeeaaanenni et al. 2019) performs strongest in terms of quantitative metrics, but exhibits surprisingly strong dependency on color to separate objects and accumulates false positives when fewer objects than slots are present.
Spatial transformer models, TBA (He al. 2019) and SCALOR (Jiang et al. 2020), train most efficiently and feature explicit depth reasoning in combination with amodal masks, but are nevertheless outperformed by the simpler model, VIMON, lacking depth or interaction model, suggesting that the proposed mechanisms may not yet work as intended.
We will make our code, data, as well as public leaderboard of results available.
2 RELATED WORK
Several recent lines of work propose to learn object-centric representations from visual inputs for static and dynamic scenes without explicit supervision. Though their results are promising, methods are currently restricted to handling synthetic datasets and as of yet are unable to scale to complex natural scenes. Furthermore, a systematic quantitative comparison of methods is lacking.
Selecting and processing parts of image via spatial attention has been prominent approach for this task (Mnih et al. 2014; Eslami et al. 2016; Kosiorek et al., 2018 Burgess et al. 2019 Yuan
et al. 2019 Crawford & Pineau 2019 Locatello et al. 2020). As alternative, spatial mixture
models decompose scenes by performing image-space clustering of pixels that belong to individual objects (Greff al., 2016 2017: 2019: van Steenkiste al. 2018). While some approaches aim at learning a suitable representation for downstream tasks (Watters et al. 2019a Veerapaneni et al.,
2019), others target scene generation (Engeleke et al. 2019 von Kügelgenn et al. 2020). We analyze
three classes of models for processing videos, covering three models based on spatial attention and one based on spatial mixture modeling.
Spatial attention models with unconstrained latent representations use per-object variational autoencoders, as introduced by Burgess et al.) (2019). yon Kügelgen et al. (2020) adapts this approach for scene generation. So far, such methods have been designed for static images, but not for videos. We therefore extend MONET (Burgess et al. 2019) to able to accumulate evidence over time for tracking, enabling to include this class of approaches in our evaluation. Recent concurrent work on AlignNet (Creswell et al.) 2020) applies MONET Trame-by-frame and tracks objects by subsequently ordering the extracted objects consistently.
Spatial attention models with factored latents use an explicit factorization of the latent repre- sentation into properties such position, scale and appearance (Eslami et al.. 2016;] Crawford & Pineau 2019). These methods use spatial transformer networks (Jaderberg et al. (2015) to render per-object reconstructions from the factored latents (Kosiorek al. 2018 He et al. 2019 Jiang et al. 2020). SQAIR (Kosiorek et al., 2018) does not perform segmentation, identifying objects only the bounding-box level. We select SCALOR (Jiang et al. 2020) for analyzing spatial transformer methods in our experiments, which explicitly disentangle object shape and appearance, providing access to object masks.
(TBA) (He et al.) 2019) and
Spatial mixture models cluster pixels using a deep neural network trained with expectation maxi- mization (Greff et al. 2017 van Steenkiste al. 2018). IODINE (Greff al. 2019) extends these methods with an iterative amortised variational inference procedure (Marino et al., (2088), improving segmentation quality. SPACE (Lin et al. 2020) combines mixture models with spatial attention to improve scalability. To work with video sequences, OP3 (Veerapaneni et al.) 2019) extends IODINE by modeling individual objects' dynamics as well as pairwise interactions. We therefore include OP3 in our analysis representative spatial mixture model.
3 OBJECT-CENTRIC REPRESENTATION BENCHMARK
To compare the different object-centric representation learning models on their basic perceptual abilities, we the well-established multi-object tracking (MOT) protocol (Bernardin & Stiefelhagen
2

---

Under review as a conference paper at ICLR 2021
Table 1: Summary of datasets and example video sequences. See Appendix B for details.
Objects
Background
Dataset
Shape
Motion
Count Over Sequence
Size Variation
Orientation
Color
Motion
Color
SpMOT
4 Templates (2D)
Lineur
Varies (0-3)
Minimal
Fixed
Colors
None
Black
VOR
2 Templates (SD) 3 Tomplates (2D)
Varies (0-4)
Moderate
Random
Colors
Moving Camera
Random
VMDS
Non-Linear
Fixed (1-4)
Moderate
Random
256' Colors
None
Random
SpMOT
Von
VMDS
2008 In this section, we describe the datasets and metrics considered in our benchmark, followed by brief description of the models evaluated.
3.1 DATASETS
Current object-centric models are not capable of modeling complex natural scenes (Burgess et al. 2019) Greff et al. 2019 Lin et al. 2020). Hence, we focus on synthetic datasets that resemble those which state-of- oh-the-art models were designed for. Specifically, we evaluate on three synthetic dataset (see Table D. which cover multiple levels visual and motion complexity. Synthetic stimuli enable to precisely generate challenging scenarios in controllable manner in order to disentangle sources of difficulty and glean insights on what models specifically struggle with. We design different scenarios that test complexities that would occur in natural videos such partial or complete occlusion as well as similar object appearances.
Sprites-MOT (SpMOT, Table left), proposed by He et al. 2019), features simple 2D sprites moving linearly on a black background with objects moving in and out frame during the sequence. Video-MultiidSpritee (VMDS, Table right) video dataset we generated based on colored, multi-object version of the dSprites dataset (Matthey et al. 2017). Each video contains one to four sprites that move non-linearly and independently of each other with the possibility of partial full occlusion. Besides the i.i.d. sampled training, validation and test sets of VMDS, we generate seven additional challenge sets that we use study specific test situations we observed to challenging. such as guaranteed occlusion, specific object properties, or out-of-distribun Video Objects Room (VOR, Table middle) Objects Room dataset (Greff et al. 2019), which features static objects in a 3D room with moving camera. For full details on the datasets and their generation, see Appendix
appearance variations. video dataset we generated based on the static
3.2 METRICS
Our evaluation protocol follows the multi-object tracking (MOT) challenge, standard and widely- used benchmark for supervised object tracking (Milan et al. 2016). The MOT challenge uses the CLEAR MOT metrics (Bernardin Stetfehaggnn 2008), which quantitatively evaluate different performance aspects of object detection, tracking and segmentation. To compute these metrics, predictions have be matched ground truth. Unlike Bernardin & Stiefelhagen (2008) and Milan al. (0116), we use binary segmentation masks for this matching instead of bounding boxes, which helps us better understand the models' segmentation capabilities. We consider an intersection over union (IoU) greater than 0.5 match (Voigtlaender et al.) 2019). The error metrics used the fraction of Misses (Miss), ID switches (ID S.) and False Positives (FPs) relative to the number of ground truth masks. In addition, we report the Mean Squared Error (MSE) of the reconstructed image outputs summed over image channels and pixels.
To quantify the overall number of failures, the fraction of all failure cases compared to the total number of objects present in all frames. model with 100% MOTA perfectly tracks all objects without any misses, ID switches false positives. To quantify the segmentation quality, we define MOT Precision (MOTP) as the average loU of segmentation masks of all matches. A model with 100% MOTP perfectly segments all tracked objects, but does not necessarily track all objects. Further, to quantify detection and tracking performance
the MOT Accuracy (MOTA), which measures
Datasets available this https URL
3

---

Under review as a conference paper at ICLR 2021
Flaicte nequential
X Iranstermer
requantaal
Moetnemerts
Siviill Thanlome
Reconstructon
Des
Keote-arddel
VAF
Slo1
2e12
SELK
my
WAE
MLF
NL
Ne
Latents 2,1
LSSI
Slot
2 Siot
Eirande
Image
(a) Common Principle
ViMON
(c) TBA
(d) OP3
(c) SCALOR
Figure I: Common principles of all models: Decomposition of an image into fixed number of slots, each of which contains an embedding =r.k and a mask Mr, (ideally) a single object. Dotted lines: temporal connections. Solid lines: information flow within one frame.
independent of false positives, we measure the Mostly Detected (MD) and Mostly Tracked (MT) metrics, the fraction of ground truth objects that have been detected and tracked for at least 80% of their lifespan, respectively. an ID switch occurs, an object considered detected but not tracked. For full details regarding the matching process and the evaluation metrics, refer to Appendix A
3.3 MODELS
We consider three classes of unsupervised object-centric representation learning models: (1) spatial attention model with unconstrained latents, VIMON, which is our video extension of MONET (Burgess et al. 2019); (2) spatial transformer-based attention models, TBA (He al. 2019) and SCALOR (Jiang et al. 2020); (3) scene mixture model, OP3 (Veerapaneni et al. 2019 At high-level, these methods share common structure which illustrated in Fig. [I| They decompose an image into a fixed number of slots (Burgess et al. 2019), each of which contains an embedding 2t,k and a mask m, of (ideally) a single object. These slots are then combined in a decoding step to reconstruct the image. Below, we briefly describe each method. Appendix C provides a detailed explanation
unified mathematical framework.
Video MONet (VIMON) is our video extension of MONET (Burgess et al., 2019) MONET recurrently decomposes a static scene into slots, using an attention network to sequentially extract attention masks m [0, 1 | x W of individual objects k. A Variational Autoencoder (VAE) (Kingma Welling 2014) encodes each slot into a latent representation Z₁ E R²_ of the corresponding object. We use MONET simple frame-by-frran frame baseline for detection and segmentation that does not employ temporal information. VIMON accumulates evidence about the objects over time to maintain consistent object-slot assignment throughout the video. This is achieved by (1) seeding the attention network the predicted mask m. E [0, Hx from the previous time step and (2) introducing a gated recurrent unit (GRU) (Cho al. 2014), which aggregates information over time for each slot separately, enabling it to encode motion information. For full details on MONET and VIMON, as
well as ablations to provide context for the design decisions, refer to Appendix C.I C.2 and E.3
Tracking- Baacking-by-Anmmatte by-Animation (TBA) He et al. 2019) is a spatial transformer-basee attention model. Frames encoded by convolutional feature extractor before being passed recurrent block g called Reprioritized Attentive Tracking (RAT). RAT re-weights slot input features based on their cosine similarity with the slots from the previous time step and outputs latent representations for all K slots in parallel. Each slot latent is further decoded into a mid-level representation yt.A consisting of pose and depth parameters, as well as object appearance and shape templates (see Fig. Tc). For rendering, Spatial Transformer Network (STN) (Jaderberg et al. 2015) is used with an additional occlusion check based on the depth estimate. TBA is trained on frame reconstruction with an additional penalty for large object sizes to encourage compact bounding boxes. TBA can only process scenes with static backgrounds, as it preprocesses sequences using background subtraction (Bloisi & Locchil 2012). For full details TBA, refer Appendix C.3
Object-centric Perception, Prediction, and Planning (OP3) /Veerapaneni et al., 2019) extends IODINE (Greff et al. 2019) operate videos. IODINE decomposes an image into objects and
4

---

Under review as a conference paper at ICLR 2021
Table 2: Analysis of SOTA object-centric representation learning models for MOT. Results shown as mean + standard deviation of three runs with different random training seeds.
Model
MOTA
M
MD
MT t
Match f
Miss
ID
PPs
MSE
SpMOT
MONET
70.2 =
89.6
92.4 006
50.4
75.3
4.4
20.3
5.1
13.0
VIMON
92.9
91.8
87.7
87.2
95.0
4.8 02
0.2 0.
2.1 :.
11.1
TBA
79.7 15.0
71.2 1 03
83.4 197
80.0 136
87.8
9.6 1:60
2.6 2:30
8.1 1:60
11.9
OP3
89.1 1:51
78.4
1 24
92.4 1.4.0
91.8 3.
95.9 2.
3,7 1:22
0.4 1:00
6.8 1:29
13.3 11.
SCALOR
94,9 0.5
80.2 1: 0.1
96.4
93.2
95.9 944
2,4 8:00
0.
1.0
3,4
VOR
MONET
37.0 ::
81.7 =: 0.5
76.9 2.2
37.3 =
64.4 = 5.0
15.8
19.8 3.
27.4 2:
12.2 1444
VIMON
89.0
89.5 #
90.4
90.0 04
93.2
6.5
4.2
6.4
OP3
65,4
89.0
88.0 +:
85.4
90.7 0.3
8.2 +:200
1.1
25.3
3,0
SCALOR
74.6 0:4
86.0 02
76.0 0.04
75.9
77.9 04
22.1 2:04
0.0 0.0
3.3 +
6.4 0.1
VMDS
MONET
49.4 3.6
78.6
74.2
35.7
66.7 0.7
13.6 10
19.7
17.2 3:1
22.2 22
VIMON
86.8 0:03
86.8 00
86.2 0.3
85.0 0.3
92.3 0.2
7.0 0:2
0.7 0.00
5,5 +0
10.7 =0.1
TBA
12.1
75.0 + 09
62.9 5.59
58.3 6.1
75.9 4.
21.0
0.3
21.4
28.1 =20
OP3
91.7 1.7
96.8 0.8
96.3 0.4
2.0
0.0
1:
4.3
SCALOR
74.1
87.6
67.9
66.7
78.4
20.7
0.8
4.4 1.04
14.0
represents them independently by starting from initial guess of the segmentation of the entire frame, and subsequently iteratively refines using the refinement network f (Marino et al. 2018 In each refinement step m, the image is represented by K slots with latent representations 2 OP3 applies IODINE to each frame X₁ to extract latent representations zt,m,kk which are subsequently processed by dynamics network (see Fig. [c). which models both the individual dynamies of each slot & as well as the pairwise interaction between all combinations of slots, aggregating them into prediction of the posterior parameters for the next time step + for each slot k. For full details on IODINE and OP3, refer to Appendix C.4land C.5 respectively.
SCALable Object-oriented Representation (SCALOR) (Tiagg et al. 2020) is a spatial transformer-bas model that factors scenes into background and multiple foreground objects, which are tracked throughout the sequence. Frames are encoded using convolutional LSTM f. In the proposal-rejection phase, the current frame & is divided into H x W grid cells. For each grid cell object latent variable br,w, w is proposed, that is factored into existence, pose, depth and appearance parameters, Subsequently, proposed objects that significantly overlap with propagated object are rejected. In the propagation phase, per object GRUs are updated for all objects present in the scene. Additionally, SCALOR has background module to encode the background and its dynamics. Frame reconstructions are rendered using a background decoder and foreground STNs for object masks and appearance. For full details on SCALOR, refer to Appendix C.6
RESULTS
We start with summary of our overall results across the three datasets and four models (Table before analyzing more specific challenging scenarios using variants of the VMDS dataset.
We first ask whether tracking could emerge automatically in an image-based model like MONET, which may produce consistent slot assignments through its learned object-slot assignment. This not the case: MONET exhibits poor tracking performance (Table 2 While MONET correctly finds and segments objects, does not assign them to consistent slots over time (Fig. E.2) In the following, we will thus focus the video models: VIMON, TBA, OP3 and SCALOR.
SpMOT. All models perform tracking well on SpMOT with the exception of one training run of TBA with poor results leading to high standard deviation (ep. best TBA model: 89.8% MT; Table E. SCALOR outperforms the other models on the detection and tracking metrics MD and MT, while VIMON exhibits the highest MOTP, highlighting its better segmentation performance SpMOT.
VOR. TBA is not applicable to VOR due to the dynamic background which cannot be resolved using background subtraction. VIMON and OP3 show similarly good performance detection (MD) and
5

---

Under review as a conference paper at ICLR 2021
segmentation (MOTP), while VIMON outperforms OP3 on the tracking metrics MOTA and MT. OP3 accumulates a high number of false positives leading to a low MOTA due to the splitting of objects into multiple masks well as randomly segmenting small parts of the background (Fig. EE.4) In contrast, SCALOR has almost no false positives or ID switches, but accumulates a high number of misses leading a poor MOTA. It often segments two objects one that occluding each other in the first frame, which is common in VOR due to the geometry of its scenes (Fig. F.11 last row).
VMDS. OP3 outperforms the other models on VMDS, on which TBA performs poorly, followed by SCALOR, which again accumulates a high number of misses. We will analyze the models on VMDS qualitatively and quantitatively in more detail in the following.
Accumulation of evidence over time. Recognition and tracking of objects should improve if models can exploit prior knowledge about the objects in the scene from previous video frames. To test whether the models exploit such knowledge, we evaluate their MOTA per-
Purformanc difforance [SQ
80
VIMON
TBA
0-
60-
VIMON
of
SCALOR
40-
-2
TBA
GP3
SCALDR
20
Warm start=2
formance on VMDS after warm starting with up to 10 frames which are not in- cluded in evaluation (Fig. 2 Note that the models were trained sequences of length 10, but are run for 20 frames in the case of a warm start of 10 frames. The performance of VIMON improves with longer warm starts, showing that the GRU accumulates evidence over time. TBA, in contrast, does not use temporal information beyond 2-3 frames, while SCALOR's 's performance slightly drops after 3 frames. OP3 appears to most strongly rely on past information and is able to integrate information over longer time scales; its performance does not even saturate with a warm start of 10 frames. However, the effect for all models is rather small.
9:10
Number
Namber of objects sideo
Figure 2: MOTA on frames 11-20 of the VMDS test set with warm starts of 1-10 frames (0 no warm start). Difference to performance of warm start 2 shown.
Figure 3: Distribution of failure cases dependent on number of objects in VMDS videos. Mean of three training runs. Error bars: standard deviation.
Challenging scenarios for different models. The number of objects in the sequence matters for VIMON, TBA and SCALOR: more objects increase the number of failure cases (Fig. (3). In contrast, OP3 does not exhibit this pattern: it accumulates higher number of false positives (FPs) in videos with fewer (only one or two) objects (Fig. I.T. as it tends to split objects into multiple slots if fewer objects than slots present.
Occlusion leads to failure cases for all models (Fig. 4). Partial occlusion can lead to splitting of objects into multiple slots (Fig. 4h) Objects that reappear after full occlusion are often missed when only small part of them is visible (Fig. 4p). In particular, S SCALOR tends to segment two objects as one when they overlap while entering the scene, leading
high number of misses.
Color of the object is important. TBA often misses dark objects (Fig. 4p). In contrast, VIMON, OP3 and SCALOR struggle with scenes that feature objects of similar colors as well as objects that have similar colors to the background (Fig, Fb.e)
False positives are more prevalent for OP3 and TBA than for VIMON and SCALOR (Table 2). FPs of OP3 are due to objects split in multiple masks (Fig. 4b) and random small parts of the background being individually segmented (Fig. 4e), while TBA tends to compose larger objects using multiple
smaller, simpler components (Fig.
Challenge sets. Based on the challenging scenarios identified above, we design multiple 'challenge sets': videos featuring (I) heavy occlusion, (2) objects with same colors, (3) only small objects and (4) only large objects (Fig. 5 top). For details, see Appendix (B.1.I).
Occlusion reduces performance of all models compared with the i.i.d. sampled VMDS test set, albeit
to different degrees (Fig. 5) for absolute performance see Table E.2 OP3 more robust to occlusion
than the other models.
Tracking objects with the same color is challenging for all models (Fig. 5). In particular, OP3 appears to rely on object color as a way to separate objects.

---

Under review as a conference paper at ICLR 2021
Grouns
Truth
Ground
MNON
TBA Racces
SSMON
VINON
TBA Sigg
WHYON Seggn
(b) TBA struggles with dark objects.
CP
Ground Trun
OR3
OP3 Segins
OPT Rocon
TBA
TBA
OP3 Saggm
TBA Segm
OP3 misses object with similar color to background.
TBA Segit
SOLDK
SEALOR
SCALER
SOALR
Models struggle with
Models strugple with similar
TTa Segm.
ooclusion.
object color.
(d)TBA splits large objects into multippe masks.
Example failure cases for models on VMDS. Segmentation masks are binarized and color-coded to signify slot assignment.
Figure 4:
OP3, VIMON and SCALOR not sensitive to object size (Fig. 5) They exhibit only slightly decreased performance on the large objects test set, presumably because large objects cause more occlusion (Fig. 5). TBA shows increased performance small objects but performs poorly the large objects set.
iut-of-distribution test sets. Next, eralization to -ittr--distribution (o.o.d.) changes in object appearance that not encountered during training. the training set of VMDS, object color, size and orientation are constant throughout video. To test o.o.d. generalization, we evaluate models trained on VMDS on three datasets that feature un- seen object transformations (Fig. [6 and Table E.3)
gen-
Dockusian
Same Color
t=1
t=4
[=7
t-10
D=1
t- 10
continuous changes in object color or size as well as continuous rotation around the object's centroid
Performance N
MOTA
MT
Test set
TBA
while moving. For details, see Appendix B.1.2
OP3
Ocdla son
Same color
Small Large Occku- ston
Same color
Small Large
Continuous changes in object size do not pose rious problem to TBA, OP3 and SCALOR, while VIMON's performance drops (Fig. 6). Surpris- ingly, continuous color changes of objects do not impact the performance of any model. Tracking performance of VIMON drops significantly for rotated objects, while OP3 and SCALOR are affected less. TBA's tracking performance is not as strongly influenced by object rotation (for absolute values, Table
objecs objects
Aijectsibbees
Figure 5: Performance on challenge sets relative performance VMDS test set (100%).
Stability of training and runtime. TBA and SCALOR train faster and require less memory than OP3 and VIMON (see Table E.4 for details). However, some training runs converge to suboptimal minima for TBA. Training OP3 is sensitive to the learning rate and unstable, eventually diverging in almost all experiments. Interestingly, it often reached its best performance prior divergence. hhper-paranen parameter settings in our experiments. For a more detailed analysis of the runtime, see Appendix E.2
VIMON and TBA are less sensitive to hyper-

---

Under review as a conference paper at ICLR 2021
5 DISCUSSION
Our experimental results provide insights into the inductive biases and failure cases of object- centric models that were not apparent from their original publications. Despite the positive re-
Size
Color
Rotation
L-1
L-4
t=7
t-10 [=1 tm4
bw10
tw
bw10
sults shown in each of the papers for the evalu- ated methods, a controlled, systematic analysis demonstrates that they do not convincingly suc- ceed at tracking, which is fundamentally what
MOTA
Performance %
100
Test set
VIMON
DP3
object-centric video methods should enable.
SCALOR
TBA has a significantly lower MOTP than the other models all datasets, suggesting that the simple rendering-based decoder using a fixed template might be less suitable to generate ac- curate segmentation masks (see also Fig. F.5 and Fig. F.4) compared to the VAE-based approaches of VIMON, OP3 and SCALOR.
Size
Color
Rota-
Size
Color
Rota-
tion
tion
Figure 6: Performance out-of-distribution sets relative to VMDS test set (100%).
Handling occlusion of objects during the video is a key property object-centric representations should be capable of. Qualitatively and quantitatively, OP3 more robust to occlusion than the other models, suggesting that its dynamics network which models interactions between objects is currently most successful modeling occluded objects. Surprisingly, TBA and SCALOR, which explicitly encode depth, do not handle occlusion more gracefully than VIMON, whose much simpler architecture has no explicit way of dealing with depth. Moving forward, ocelusion handling a key component that object-centric video models need master, which can be addressed by either equipping the model with a potent interaction module, that takes pairwise interaction between objects (including occlusion) into account, similar to OP3's dynamies model, ensuring that the depth reasoning of the models works as intended, which may be preferable, as explained below.
All models struggle with detecting objects that have similar color as the background (for TBA: dark objects, since background is removed and set to black in a pre- pre-processing step). Color is a reliable cue to identify objects in these datasets. However, the auto-encoding objective incurs little extra loss for missing objects with similar color as the background and, thus, the models appear to not to learn to properly reconstruct them. In order to scale to data with more visual complexity, one might want to replace the pixel-wise reconstruction with for instance a loss based in feature space in order to focus more on reconstructing semantic content rather than high- fre-frreyency texture, as done when using perceptual loss functions (Gatys et al.) 2015) Hou et al., 2017) or by using contrastive learning (Kipf al. 2020). Furthermore, the models particularly so OP3 struggle with separating objects of similar colors from each other. This result hints at mismatch between the intuitions motivating these models and what the models actually learn: it should be more efficient terms of the complexity of the latent representation to decompose two objects even of similar colors- into two masks with simple shapes, rather than encoding the more complicated shape of two objects simultancously in one slot. However, since none of the models handle occlusion with amodal segmentation masks ei including the occluded portion of the object) successfully, they learn to encode overly complex (modal) mask shapes. As consequence, they tend merge similarly colored objects into one slot, This result suggests that resolving the issues surrounding the depth reasoning in combination with amodal segmentation masks would enable much more compact latents and could also resolve the merging of similarly colored objects.
major difference between models the spatial transformer based model formulation of TBA and SCALOR, compared to VIMON and OP3, which operate on image-sized masks. The parallel processing of objects and the processing of smaller bounding boxes renders training TBA and SCALOR to be significantly faster and more memory efficient, enabling them to scale to a larger number of objects. On the downside, the spatial transformer introduces its own complications. TBA depends strongly on its prior object size and performs well only when this prior fits the data well as well as when the data contains little variation in object sizes, as in SpMOT (Table 2). However, not able to handle VMDS and its larger variation in object sizes and shapes. SCALOR performs tracking well in scenes where objects are clearly separated, but struggles to separate objects that partially occlude each other when entering the scene. This difficulty is caused by its discovery
8

---

Under review as a conference paper at ICLR 2021
mechanism which propose most one bounding box per grid cell. leading to high number of misses on datasets which feature significant occlusion (VOR and VMDS). Unfortunately, simply increasing the number of proposals does not provide simple solution, SCALOR's performance sensitive to properly tweaking the numher of proposals.
Choosing class of models is therefore dependent the dataset wants to apply it to well as the computational resources at one's s disposal. Datasets that feature a high number of objects (>10) that well separated from each other make method like SCALOR, which can process objects in parallel, advisable. On datasets with a lower number of objects per scene which feature heavy occlusion, methods like OP3 and ViMON will likely achieve better results, but require high computational budget for training.
In conclusion, our analysis shows that none of the models solve the basic challenges of tracking even for relatively simple synthetic datasets. Future work should focus on developing robust mechanisms for reliably handling depth and occlusion, additionally combining the transformer-basee efficiency of TBA and SCALOR with the stable training of VIMON and the interaction model of OP3. The key open challenges for scaling these models natural videos include their computational inefficiency, complex training dynamics, well as over-dependence on simple appearance cues like color.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: The clear mot metries. EURASIP Journal on Image and Video Processing, 2008
Domenico Daniele Bloisi and Luca locchi. Independent multimodal background subtraction. In CompIMAGE, 2012.
Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matthew Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representa- tion. arXiv.org, 1901. 1390, 2019.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv.org, 1406.1078, 2014.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. Proc. of the Conf. on Artificial Intelligence (AAAT), 2019.
Antonia Creswell, Kyriacos Nikiforou, Oriol Vinyals, Andre Saraiva, Rishabh Kabra, Loic Matthey, Chris Burgess, Malcolm Reynolds, Richard Tanburn, Marta Garnelo, and Murray Shanahan. Alignnet: Unsupervised entity aligmment. arXiv.org. 2007.08973, 2020.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. arXivorgg 1907.13052, 2019.
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2016.
Leon A. Gatys, Alexander Ecker, and Matthias Bethge. neural algorithm of artistic style.
arXiv.org, 1508.06576, 2015.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems (NeurIPS). 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Proc. of the International Conf. on Machine learning (ICMI), 2019.
Zhen He, Jian Li, Daxue Liu, Hangen He, and David Barber. Tracking by animation: Unsupervised learning of multi-object attentive trackers. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, doi: (a.162//cco.1199..98.117 URL https::/dois org. 11w./11121..... www.//eco..///////////// 1997 9.8.1735
(8):1735-1780, November 1997. ISSN 1899-7667,
X. Hou, L. Shen. K. Sun, and G. Qiu. Deep feature consistent variational autoencoder. In 2017 IEFF Winter Conference Applications of Computer Vision (WACV), pp. 1133-1141, 2017. doi: 10. 1.1100//ACV..21..111...
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. In Advances in Neural Information Processing Systems (NeurIPS). 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proc. of the International Conf. Learning Reprexentations (ICIR), 2017.
10

---

Under review as a conference paper at ICLR 2021
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In Proc. of the International Conf. on Learning Representations (ICLR), 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of the International Conf. on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proc. of the International Conf. on Learning Representations (ICLR), 2014.
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In Proc. of the International Conf. on Learning Representations (ICLR), 2020.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. arXiv.org, 2001.02407, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. arXiv.org, 2006. 15055, 2020.
Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In Proc. of the International Conf. on Machine learning (ICML), 2018.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset.
2017.
A. Milan, L. Leal-Taixee Reid, S. Roth, and K. Schindler. MOT16: A benchmark for multi-object tracking. arXiv.org, 1603.008 1000108 2016.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Detito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-perffrmance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In Proc. of the International Conf. on Learning Representations (ICLR), 2018. URL /ttttss://opeeree////
Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua B. Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based reinforcement learning. arXiv.org. 1910.12821 2019.
Paul Voigtlaender, Michael Krause, Aljosa Osep. Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
Julius von Kügelgen, Ivan Ustyuzhaninov, Peter Gehler, Matthias Bethge, and Bernhard Schölkopf. Towards causal generative scene models via competition of experts. arXivorrg, 12906, 2020.
11

---

Under review as a conference paper at ICLR 2021
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander Lerchner. COBRA: Data-efficient model-based RL through unsupervised object discovery and curiosity- driven exploration. arXiv.org, 1905.09275, 2019a.
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in VAEs. arXiv.org, 1901.07017, 2019b.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Generative modeling of infinite occluded objects for compositional scene representation. In Proc. of the International Conf. on Machine learning (ICML), 2019.
12

---

Under review as a conference paper at ICLR 2021
SUPPLEMENTARY MATERIAL FOR: BENCHMARKING UNSUPERVISED OBJECT REPRESENTATIONS FOR VIDEO SEQUENCES
In this supplementary document, we first discuss the metrics used (Section A and describe the data generation process (Section B). We then describe the methods MONET, VIMON, TBA, IODINE, OP3 and SCALOR (Section Cp. Dectionn information regarding the implementation details and training protocols. Finally, we provide additional qualitative and quantitative experimental results in Section
A EVALUATION PROTOCOL DETAILS
We quantitatively evaluate all models on three datasets using the standard CLEAR MOT metrics [Bernardin & Stiefelhagen] 2008). Our evaluation protocol is adapted from the multi-object tracking (MOT) challenge (Milan et al. 2016), a standard computer vision benchmark for supervised object tracking. In particular, we focus on the metrics provided by the py-motmetrics package
A.1 MAPPING
In each frame, object predictions of each model in the form of binary segmentation masks mapped to the ground truth object segmentation masks. We require that each pixel is uniquely assigned to most one object in the ground truth and the predictions, respectively. Matching is based the intersection over union (IoU) between the predictions and the ground truth masks (Voigtlaender al. 2019). A valid correspondence between prediction and object has to exceed a threshold in IoU of 0.5, Predictions that are not mapped to any ground truth mask are classified as false positives (FPs). Ground truth objects that are not matched to any prediction are classified as misses. Following (Bernardin & Stiefelhagen 2008), ground truth objects that are mapped to two different hypothesis IDs in subsequent frames are classified as ID switches for that frame.
A.2 MOT METRICS
MOT Accuracy (MOTA) measures the fraction of all failure cases, i.e. false positives (FPs), misses and switches compared to total number of objects present in all frames. MOT Precision (MOTP) measures the total accuracy in position for matched object hypothesis pairs, relative to total number of matches made. We use percentage Intersection over Union (IoU) of segmentation masks as the accuracy in position for each match. Mostly Tracked (MT) is the ratio of ground truth objects that have been tracked for at least 80% of their fifespan.(i.. 80% of the frames in which they are visible). MT as implemented by py-motmetries counts trajectories of objects as correctly tracked even if ID switches occur. We use strictly more difficult definition of MT that counts trajectories with ID switches as correctly detected but not correctly tracked. Consequently, we add the Mostly Detected (MD) measure which does not penalize ID switches. Match, Miss, ID Switches (ID S.) and FPs are reported as the fraction of the number occurrences divided by the total number of object occurrences.
ΣT 21-1 M. FP, + IDS, O,
MOTA- 1
where M,, FP, IDS; are the number of misses, false positives and ID switches, respectively, for time f, and O is the number of objects present in frame Note that MOTA can become negative, since the number of FPs is unbounded.
MOTP ΣT
EL
E
d
(2)
E1 ct
where dž is the total accuracy in position for the 5th matched object-hypothesis pair measured in loU between the respective segmentation masks and e is the number of matches made in frame f.
13

---

Under review as a conference paper at ICLR 2021
Note that we exclude the background masks for VIMON and OP3 before evaluating tracking based on IoU. The Video Object Room (VOR) dataset can contain up to three background segments, namely the floor and up two wall segments. In order to exclude all background slots regardless of whether the model segments the background as one or as multiple masks, we remove all masks before the tracking evaluation that have an IoU of more than 0.2 with one of the ground truth background masks; we empirically tested that this heuristic is successful in removing background masks regardless of whether the models segments it as one or as three separate ones.
B DATASET GENERATION DETAILS
B.1 VIDEO MULTI-DSPRITES (VMDS)
The Multi-DSprrtes Video dataset consists of 10-frame video sequences of 64x 64 RGB images with multiple moving sprites per video. In order test temporal aggregation properties of the models, the test set contains 20 frame-long sequences. Each video contains one to four sprites following the dataset proposed in (Burgess
2019) that move independently of each other and might partially or fully occlude one another. The sprites are sampled uniformly from the dSprites dataset (Matthey et al.) 2017) and colored with a random RGB color. The background is uniformly colored with a random RGB color. Random trajectories are sampled per object by drawing x and coordinates from Gaussian process with squared exponential covariance kernel cov [Ix, 27.] = exp(- (xr, xx) / and time constant 10 frames, and then shifted by an initial (z, --ossitoioonn of the sprite centroid, which is uniformly sampled from [10,54 54] to ensure that the object is within the image boundaries. Trajectories that leave these boundaries are rejected. In occlusion scenarios, larger objects are always in front of smaller objects to disambiguate prediction of occlusion. The training set consists of 10,000 examples whereas the validation set as well the test set contain 1,000 examples each. Additionally, we generated four challenge sets and three out-of-distribution test sets for VMDS that contain specifically challenging scenarios. Each test consists of 1,000 videos of length 10 frames, which we describe in the following.
B.1.1 VMDS CHALLENGE SETS
Occlusion test set. In each video, one or more objects are heavily occluded and thus often are not visible at all for a few frames. This is ensured by sampling object trajectories that cross path, i.e. at least in one video frame, two objects are centered on the same pixel. The time step and spatial position of occlusion is sampled randomly. Object trajectories are sampled independently as described above and then shifted such that they are at the sampled position of occlusion time t. Videos contain two to four sprites (Fig. 5 since at least two objects are necessary for occlusion.
Small Objects. Videos contain one four sprites with all sprites being of the smallest size present in the original dSprites (Matthey et al. (0117) dataset (Fig. 5). Other than that, it follows the generation process of the regular training and test set.
Large Objects. Videos contain one to four sprites with all sprites being of the largest size present in the original dSprites (Matthey et al. (2017) dataset (Fig. 5). Other than that, follows the generation process of the regular training and test set.
Same Color. Videos contain two to four sprites which are identically colored with a randomly chosen color. Other than that, it follows the generation process of the regular training and test set (Fig. 5
1.1 VMDS OUT-OF-DISTRIIUTTIN TEST SETS
Rotation test set. Sprites rotate around their centroid while moving. The amount of rotation between two video frames is uniformly sampled between 5 and 40 degrees, and is constant for each object over the course of the video. Direction of rotation is chosen randomly, Rotation is not included as a transformation in the training set (Fig. 6)
Color change test set. Sprites change their color gradually during the course of the video. The initial hue of the color is chosen randomly as well as the direction and amount of change between two frames, which stays the same for each object over the course of the video. Saturation and value of the color are kept constant. Color changes are not part of the training set (Fig. 6
14

---

Under review as a conference paper at ICLR 2021
Size change test set. Sprites change their size gradually during the course of the video. The original dSprites dataset (aathhey et al. (017)) contains six different sizes per object. For each object, its size is sampled either the smallest largest in the first frame as well as random point in time, at which it starts changing its size. At this point in time, will either become larger or smaller, respectively, increasing decreasing each frame to the next larger smaller size present in the original dSprites dataset, until the largest or smallest size is reached. Size changes are not part of the training set (Fig.
B.2 SPRITES-MON MOT (SPMOT)
SOri originally introduced by (He al., 2019). consists of video sequences of length 20 frames. Each frame is a 128x 128 RGB image. It features multiple sprites moving linearly on black background. The sprite can have information, refer to the original paper (He et al. 2019). We generate a training set consisting of 9600 examples, validation set of 384 samples and test set of 1,000 examples using the author-provided public e However, instead of using the default setting of 20 frames per sequence, we instead generate sequences of length 10, in order to facilitate comparison the other datasets in study which have only 10 frames per sequence.
of four shapes and one of six colors. For more
Frames are downsampled resolution of 64 for training VIMON, OP3 and SCALOR.
B.3 VIDEO OBJECTS ROOM (VOR)
We generate video dataset based on the static Objects Room dataset (Greff et al. 2019), with sequences of length 10 frames each at a resolution of 128x 128. This dataset is rendered with OpenGL using the gym-miniworlq"" reinforcement learning environment. It features a 3D room with up to four static objects placed in one quadrant of the room, and a camera initialized at the diagonally opposite quadrant. The objects either static cubes or spheres, assigned one of colors and random orientation on the ground plane of the room. The camera then follows one of five trajectories moving towards the objects, consisting of small fixed distance translation and optional small fixed angle of rotation each time step. The wall colors and room lighting are randomized, but held constant throughout& a sequence. The training set consists of 10,000 sequences whereas the validation set and the test set contain I, 1,000 sequences each.
Frames are downsampled a resolution of 64 for training VIMON, OP3 and SCALOR.
C
METHODS
In this section we describe the various methods in common mathematical framework. For details about implementation and training, please refer to Section
C.I MONET
Multi-Objeet-Netwool (MONET) (Burgess et al. 2019) is an object-centric representation model designed for static images. It consists of recurrent attention network that sequentially extracts attention masks of individual objects and a variational autoencoder (VAE) (Kingma & Welling 2014) that reconstructs the image region given by the attention mask in each processing step.
The attention network is a U-Net (Ronneberger et al. 2015) parameterized by vh. At each processing step ks, the attention network receives the full image x [0. 1] H>
Attention Network:
as input together with the scope variable Sş E [0. H| The scope sy keeps track of the regions of the image that haven't been attended in the previous processing steps and thus remain to be explained. The attention network outputs soft attention mask my E [0, 10 and the updated scope with the current mask subtracted:
15

---

Under review as a conference paper at ICLR 2021
m, 8g- 10--10kp (x, 82-1)
(3)
Sk+1 = Ss Cip (x, 8k))
(4)
where ag (x, Sk) E [0, [0x is the output of the U-net and 50 1. The attention mask for the last slot given by mx S to ensure that the image is fully explained, i.e. Σ my 1.
RLLx2 and decoder R² + [0, 1] x [0, 1]#WW which are two neural networks parameterized by ø and respectively. The VAE encoder receives input the full image x and the attention mask my and computes (Mgr log JR), which parameterize the Gaussian latent posterior distribution 9o X. m&) NH,, IzI). Using the reparametriiztioo trick (Kingma & Welling 2014). Zy E A is sampled from the latent posterior distribution, Zy is decoded by the VAE decoder into a reconstruction of the image component & E [0. 1| x3 and mask logits, which are used to compute the reconstruction of the mask m. E [0. >| via a pixelwise softmax across slots. The reconstruction of the whole image is composed by summing over the K masked reconstructions of the VAE:
VAE: The VAE consists of encoder g [0, 1| [0, 1|
X
Σ
Lk=1 m Xx.
Loss: MONET is trained end-to-end with the following loss function:
K
K m&pe (x|zx) + DDLL II
L(0;0;v:x) -log
(z x ma)llp(z)
k=1
(5)
K
DKL (m)/x)/lpo(mpa
where px(x(zx) is the Gaussian likelihood of the VAE decoder and Zy E IR is the latent representa- tion of slot
The first two loss terms are derived from the standard VAE objective, the Evidence Lower BOund (ELBO) (Kingma & Welling 2014), i.c. the negative log-like log-likelihood of the decoder and the Kullback-Lcibl divergence between the unit Gaussian prior p(z) (0, and the latent posterior distribution (2x x, my) factorized across slots. Notably, the decoder log-likellnn liglliillhood term po (x (zx) constrains only the reconstruction within the mask, since it is weighted by the mask m, Additionally, as a third term, the Kullback-Leibler divergence of the attention mask distribution qu (m, (x) with the VAE mask distribution pe ( (zx) is minimized, to encourage the VAE to learn good reconstruction of the masks.
K slots sequential
V
VAE Dec.
ZLA
h h
GRU
It 1,k
C.2 VIDEO MONET
VAE Encoder
We propose an extension of MONET (Burgess et al. 2019), called Video MONet (VIMON), which accumulates evidence over time about the objects in the scene (Fig. C.T).
m1.6
Attn.
VIMON processes video recurrently by reconstructing frame at a time and predicting the next frame of the video. The processing of each frame follows logic similar MONET with some notable differences. In the following, we use / to indicate the time step in the video and k indicate the processing step within video frame.
Net
X;
Attention Network: The attention network of VIMON outputs an attention mask m E [0, #xW in each step & conditioned on the full frame x, [0, []HXXxx the scope S [0, =| and additionally the mask m E [0, H>W that predicted by the
Figure C.I: VIMON. Atten-
tion network followed by VAE encoder and GRU computes latent Zt,k-
16

---

Under review as a conference paper at ICLR 2021
VAE in the previous time step, in order to provide it with information about which object it should attend to this specific slot
M = 10kp (xe, -,, m.,k)
(6)
VAE: The VAE of VIMON consists of encoder g(x,, mi Ø) and a decoder h(z 0). In contrast to MONET, the encoder in VIMON followed by a gated recurrent unit (GRU) (Cho et al.) 2014) with separate hidden state he per slot k. Thus, the GRU aggregates information over time for each object separately. The GRU outputs (Mr,. log ttk) which parameterize the Gaussian latent posterior distribution qa (Zi,a Xx, mr.k) where Zr,h E I& is the latent representation for slot k at time I:
41, g(x,, m(,i
(7)
(11,8) log s,,,,, h S(GRU he Io (2e, Xx, m.,k) = -N
(8)
Vt, k
where 9 the VAE encoder and f is linear layer. The latent representation Zr.6 is sampled from the latent posterior distribution using the reparametrization trick (Kingma & Welling. 2014). Subsequently, Zr,x is linearly transformed into 2,+1,x via a learned transformation A RIXT.
22+1.1 AZz, with 22+1. being the predicted latent code for the next time step i + 1. Both Zi.k and 21+1. are decoded by the shared VAE decoder ha into reconstruction of the image RL E 1),1)xxX*3 and a reconstruction of the mask m.,A [0, 1] H> as well R and m respectively.
Loss: VIMON is trained an unsupervised fashion with the following objective adapted from the MONET loss (Eq. (5)) for videos. To encourage the model to learn about object motion, we include prediction objective in the form second decoder likelihood on the next-step prediction pe px(x<+1 (.+1, and an additional mask loss term, which encourages the predicted VAE mask distribution po 11 (2(++1,))) to be close to the attention mask distribution go (m,+ (x+1) of the next time step for each slot k:
L(o; 0xx;;))
Σ t=1
DeglL + BTppioo + +yy 7Tmask
K
Inegi.. -(lg
(n..ppo(Xe(2ee.p + log
m 1 &pp (x. ( (2441,
K
Lprior = DKL g (Z m)x))l(
k=
K
Lmaak
DRL(ge (ma (po (2,,))) + DKL. (gs (me+1, (lppl
C.3 TRACKING ANIMATION
Tracking by Animation (TBA) (He al. 2019) is spatial transformer-based attention model which uses simple 2D rendering pipeline as the decoder. Objects are assigned tracking templates and pose parameters by tracker array, such that they can be reconstructed in parallel using a renderer based on affine spatial transformation (Fig. C.2). In contrast to VIMON, TBA uses explicit parameters encode the position, size, aspect ratio and occlusion properties for each slot. Importantly, TBA
is designed for scenes with static backgrounds, and preprocesses sequences using background subtraction (Bloisi Tocchi 2012) before they are input to the tracker array.
17

---

Under review as a conference paper at ICLR 2021
Tracker Array: TBA uses tracker array to output latent repre- sentation 2, E RiX at time using a feature extractor f(x: 1) and recurrent 'state update', where C2 R xxMNXXC is convolutional feature representation. The convolutional feature and latent repre- sentation have far fewer elements than X2, acting bottleneck:
K slots parallel
C₁ ((x,5));
(10)
Spatial Transformer
h. = (h C,
(11)
2, g((;))) r).
(12)
(x. VLA
Though the state update could be implemented as any generic recur- rent neural network block, such LSTM (Hochreiter & Schmidd huber 1997) or GRU (Cho et al. 2014), TBA introduces a Reprior- itized Attentive Tracking (RAT) block that uses attention to achieve explicit association of slots with similar features over time. Firstly, the previous tracker state 1 is used generate key variables ki,, and 3r.:: KL, ] = The11k
MLP
Tracker Array
C
Feature Extractor
(13)
Br,k 1 1m(1 exp (Br,)),,)),
(14)
where T is a learned linear transformation, k E RS is the ad- dressing key, and Br E R is an un-normalized version of key strength variable Br (1, +ox). This key strength acts like tem- perature parameter to modulate the feature re- re-weighting, which is described in the following. Each feature vector et. denoted by Ct.m,n E RSS where m E {1,2..... } and n E {1, .1,2.....N N are the convolutional feature dimensions, is first used to get attention weights:
Figure C.2: TBA, Feature ex- tractor CNN f and tracker ar- ray g to get latent Zr, MLP h outputs mid-level represen- tation Yt,k, and Spatial Trans- former renders reconstruction.
exp(&r,xSim(kkkk Ct,m,a)
W Wi,km,
(15)
Em a exp(Br. Sim( Ct.m",n)) mn))
Here, Sim is the cosine similarity defined as Sim q) ppq (Iplllla)))) and W.m.m is an element of the attention weight W., E [0, 1| satisfying Em,,. Wr, Wr,kmm =1 1. Next, a read operation is defined as a weighted combination of all feature vectors of c,
F2,6 Σ W Wr,kmm, Cr,men
(16)
where rr, RRS the read vector, representing the associated input feature for slot k. Intuitively, for slots in which objects are present in the previous frame, the model can suppress the features in Fr. that are not similar to the features of that object, helping achieve better object-slot consistency. On the other hand, if there are slots which far do not contain any object, the key strength parameter allows F to remain similar to e, facilitating the discovery of new objects.
The tracker state h of the RAT block is updated with RNN parameterized by т,, taking Fe, instead of c, as its input feature:
b RNN TT
(17)
The RAT block additionally allows for sequential prioritization trackers, which in turn allows only subset of trackers to update their state given time step, improving efficiency. For full details on the reprioritization and adaptive computation time elements of the RAT block, please refer to the original paper (He et al. 2019).
18

---

Under review as a conference paper at ICLR 2021
Mid-Level Representation: The key feature of TBA is that each latent vector Zt,k is further decoded into a mid-level representation Yt,k (Vi Vi, Vi E Yi,k) corresponding to interpretable, explicit object properties, via fully-connected neural network ai 0) as follows:
Yt,k h(zi,);;0) 0).
(18)
hg is shared by all slots, improving parameter efficiency. The different components of the mid-level representation are:
Confidence vf,u E {0, 1]: Probability of existence of an object in that slot.
Layer ył,, 100 One-hot encoding of the discretized pseudo-depth of the object relative other objects in the frame. Each image is considered to be composed of O object layers, where higher layer objects occlude lower layer objects and the background zeroth (lowest) layer. E.g., when O 4, y [ö, 0, 1,0) denotes the third layer. For simplicity and without loss of generality, we can also denote the same layer with its integer representation v/.,
the
Pose y², = [7 8 Pr
L 2.
Normalized object pose for calculating the scale
[s],,, s = 17 ST 1 S and the translation [t" & = 2 t,k' 2 2,,k where
W
for
H
rf" n > O are constants.
Shape Y E {0, 1JUXV and Appearance Y E [0, 1| Object template, with hyperpa- rameters and V typically set much smaller than the image dimensions H and W. Note that the shape is discrete (for details, see below) whereas the appearance is continuous.
In the output layer of ha, uf,x and Y i,k are generated by the sigmoid function, y2,, is generated by the tanh function, and yi, as well as Yi,, are sampled from the Categorical and Bernoulli distributions, respectively. As sampling is non-differentiable, the Straight-Through Gumbel-Softmax estimator (Jang et al. (017) is used to reparameterize both distributions so that backpropagation can still be applied.
Renderer: To obtain a frame reconstruction, the renderer scales and shifts Y and Y according to yf via a Spatial Transformer Network (STN) (Jaderberg et al. 2015):
m k STNNNNN (Y vL.); _ = (Y
(19)
(20)
where m, E (0, 1) and & E [0, 12 are the spatially transformed shape and appearance respectively. To obtain the final object masks m an occlusion check is performed by initializing m yf, mr then removing the elements of me, for which there exists an object in higher
layer. That is, for k= 1,2, 1.2..... K and Vj + & where of.s > Vi.s-
m (1 mc. ② mi
(21)
In practice, the occlusion check is sped up by creating intermediate 'layer masks', partially paral- lelizing the operation. Please see the original paper for more details (He et al. (019). The final reconstruction is obtained by summing over the K slots, &, = Σ m C &x
Loss: Learning is driven by pixel-level reconstruction objective, defined as:
T
(o m: 0:x)
MSE(2,, x.) + +
8
(22)
where MSE refers the mean squared enror and the second term penalizes large scales [s]]ee in order to make object bounding boxes more compact.
sy,
19

---

Under review as a conference paper at ICLR 2021
C.4 IODINE
The Iterative Object Decomposition Inference NEtwork (IODINE) (Greff et al., 2019), sim- ilar to MONET (Burgess al.] 2019), learns to decompose a static scene into multi-slot representation, in which each slot represents object in the scene and the slots share the under- lying format the independent representations. In contrast to MONET, it does not recurrently segment the image using spatial attention, rather it starts from an initial guess of the segmenta- tion of the whole image and iteratively refines it. Thus, the inference component of both mod- els differ, while the generative component the same.
M refinements sequential
K slots parallel
VAE Dec.
ZrA
(Mowaay (wnit aeg
Dyn. Net
h
LSTM
h
Iterative Inference. As with MONET, IO- DINE models the latent posterior (zx(xx) per slot & as Gaussian parameterized by (pmm.k:: m, IR_22, To obtain latent rep- resentations for independent regions of the input image, IODINE starts from initial learned pos- terior parameters (H1, 11.k) and iteratively re- fines them using the refinement network fo for fixed number of refinement steps M. fo con- sists of convolutional neural network (CNN) in combination with an LSTM cell (Hochreiter Schmidhuber 1997) parameterized by Φ. In each processing step, fo receives input the image [0, xx,13x sample from the current posterior estimate k R' and various auxiliary inputs ag, which are listed in the original pa- per (Greff et al. 2019). The posterior parameters are concatenated with the output of the convolutional part of the refinement network and together form the input to the refinement LSTM. The posterior parameters are additively updated in each step m parallel for all K slots:
Refine Net
Figure C.3: OP3. Refinement network followed by LSTM and dynamics network d compute la- tent Zt,k-
(km+1,ks mm+1k) (Hmmk m, (Zm,ks X, at))
(23)
Decoder. In each refinement step m, the image is represented by K latent representations Zm,k Similar to MONET, each Zm, is independently decoded into a reconstruction of the image &m, [0, 1| x3 and mask logits mm,k, which are subsequently normalized by applying the softmax across slots to obtain the masks m,,kk [0, 1| The reconstruction of the whole image each refinement step m is composed by summing over the K masked reconstructions of the decoder:
X Σ 1=1 Mm,kk ① Km,k.
Training. IODINE is trained by minimizing the following loss function that consists of the the Evidence Lower BOund (ELBO) (Kingma & Welling 2014) unrolled through N iterations:
M
K
L(B, Φ.
=
Σ
m
log
m m. Pmmx + DKL.
II k=
go (zm, (x)
M
1=1
(24)
where po(x|zm,k) is the decoder log-likelihood weighted by the mask my and DKL is the Kullback- Leibler divergence between the unit Gaussian prior p(z) = =N I) and the latent posterior distribu- tion g(2 ( k xx) factorized across slots.
20

---

Under review as a conference paper at ICLR 2021
C.5 OBJECT-CENTREN PERCEPTION, PREDICTION, AND PLANNING (OP3)
Object-centric Perception, Prediction, and Planning (OP3) /Veerapaneni et al. 2019) extends 10- DINE to work videos and in reinforcement learning (RL) setting. It uses the above described IODINE as an observation model to decompose visual observations into objects and represent them independently. These representations are subsequently processed by a dynamics model that models the individual dynamics of the objects, the pairwise interaction between the objects, as well as the action's 8 effect on the object's dynamics, predicting the next frame latent space (Fig. C.3 By modeling the action's 's influence on individual objects, OP3 can be applied to RL tasks.
OP3 performs M refinement steps after each dynamics step.
Refinement network. The refinement steps proceed as in the description for IODINE in Section C.4 The input image x, E [o, 1] || x3, which is the frame from video at time i, processed by the refinement network fa conditioned on sample from the current posterior estimate At.m.k E R The refinement network outputs an update the posterior parameters (Hr,,, (t,m,k) (see Eq. 23 The posterior parameters (KI,IIA; I1,1,) are randomly initialized.
Dynamics model. After refinement, samples from the current posterior estimate Mr,M& for each slot l are used as input to the dynamies network. The dynamics model dy consists of series of linear layers and nonlincarities parameterized by vb. It models the individual dynamics of the objects per slot k, the pairwise interaction between all combinations of objects, aggregating them into prediction of the posterior parameters for the next time step t++ for each object k. The full dynamics model additionally contains an action component that models the influence of a given action on each object, which we do not use in tracking setting. The predicted posterior parameters then used in the next time step as initial parameters for the refinement network.
(14,1 (.11 = (2t-,,, ((11))
(25)
Training. OP3 is trained end-to-end with the ELBO used every refinement and dynamics step, with the loss L(O, Φ; x) given by:
min(m, M) (- log Σ
K
P (xx, + DEL II go m, (x)mmmm/)))))((,.....
M
k=1
(26)
where for time step ((LL1A(x)) (),
C.6 SCALABLE OBJECT-ORIENTER REPRESENTATION (SCALOR)
SCALable Object-oriented Representation (SCALOR) (Jiag et al. 2020) is a spatial transformer- based model that extends SQAIR (Kosiorek et al. 2018) to scale to cluttered scenes. Similar to TBA is factors the latent representations in pose, depth and appearance per object and uses spatial transformers (Jaderberg et al. 2015) to render objects in parallel. In contrast to TBA, it can handle dynamic backgrounds by integrating background RNN that models background transitions.
Proposal- Proposal-Rejectionn Module:: SCALOR uses a proposal-rejection module g to discover new objects. All frames up to the current time step I first encoded using convolutional LSTM The resulting features are then aggregated with an encoding of propagated object masks and divided into H W grid cells.
c cmg [xx b))
(27)
Mask Encoder (M,)) ct mask
(28)
egg Concat([c).)) img [ c.haak]]
(29)
Per grid cell latent variable Zt,,, is proposed. Proposal generation is done in parallel. Each Zt,h, consists of existence, pose, depth and appearance parameters (z pose depth. wwoat
21

---

Under review as a conference paper at ICLR 2021
2 t,
pres Bern (ern(-lgi(c
(30)
depth
(1)
(31)
Z t.h.,w
pose
(32)
where 91 92 and 33 are convolutional layers.
The appearance parameters 2t, specified by z tth,w via a Spatial Transformer Network (STN) (Jaderberg et al. 2015) and subsequently extracting features from it via convolutional neural network:
whhat are obtained by first taking glimpse from frame Ia of the area
atl ST (Ie, t,
(33)
Ct,,w
what L,h,
( Glimpssm Enole
(34)
Ot.h,w
Mr
STN- (el whot pose h,
(35)
where Or is the object RGB glimpse and mt,h, is the object mask glimpse.
In the rejection phase, objects that overlap more than a threshold r in pixel space with a prop- agated object from the previous time step are rejected.
Propagation Module:: During propagation, for each object l: from the previous time step - feature attention map ar from the encoded frame features C₁ img is extracted centered on the position of the object in the previous time step and used to update the hidden state h of the tracker RNN for object k.
K slots parallel
L/g
Spatial Transformer
bg
BG-
Network
a (Ct ing 2
(36)
Reject- ion
Zrk
(x,
h GRU 11,1, 1(-1,8)
(37)
Zr, wupdatt(aaa (ar, he,k, 1,
(38)
MLP
where STN is spatial transformer module pres
(Jaderberg et al. 2015). If z 1 the la- tent representation 2 of the respective object will be propagated to the next time step.
Proposal
h
Tracker RNNs
Background:: The background of each frame 2, is encoded using convolutional neural net- work conditioned on the masks M, of the ob- jects present time step and decoded using convolutional neural network.
Feature Extractor
(1 N0) = EnEnccder(Ea (1 Mc))
1:1
(39)
2
N o
09)
(40)
Figure C.4: SCALOR Feature extractor CNN followed by tracker RNNs or proposal-rejection module compute latent 2 Spatial Transformer in addition to background module renders recon- struction.
bg BgDecoder(z)) ho)
(41)
Rendering: To obtain frame reconstructions 8, foreground object appearances and masks
22

---

Under review as a conference paper at ICLR 2021
scaled and shifted using via Spatial Trans- former Network (STN):
f0 STN (OL 2..
(42)
Ye,k = SYTN-IMA,
26 0 1, -
prea denth))
(43)
8/9 Σ E
(44)
Subsequently, foreground objects and background reconstruction are combined as follows to obtain the final reconstruction:
x = x + (1 M2) O
bb
(45)
Training:: SCALOR is trained on frame reconstruction using the evidence lower bound (ELBO):
logPo(X((zz) +
(46)
D MODEL IMPLEMENTATION DETAILS
D.1 VIDEO MONET
VAE: Following (Burgess al. 2019). the VAE encoder is CNN with 3x3 kernels, stride and
ReI.U activations (Table D.Ip receives the input image and mask from the attention network as input and outputs (p, log or) of 16-dimensional Gaussian latent posterior. The GRU has 128 latent dimensions and one hidden state per slot followed by linear layer with 32 output dimensions.
The VAE decoder is Broadcast decoder as published by (Watters al. 2019b) with no padding. 3x3 kernels, stride and ReLU activations (Table D.2). The output distribution is an independent pixel-wise Gaussian with fixed scale of 0.09 for the background slot and 0.11 for the foreground slots.
The attention network is a U-Net (Ronneherger et al., 2015) and follows the architecture proposed by (Burgess et al.) (019).. The down and up-sampling components consist each of five blocks with 3x3 kernels, 32 channels, instance normalisation, ReLU activations and down- or up-sampling by factor of two. The convolutional layers are bias-free and use stride I and padding 1. three-layer MLP with hidden layers of size 128 connect the down- and the up-sampling part of the U-Net.
Attention Network:
Training: MONET and VIMON are implemented in PyTorch (Paszke et al. 2019) and trained with the Adam optimizer (Kingma & Ba 2015) with batch size of 64 for MON ET and 32 for VIMON, using an initial learning rate of 0.0001. Reconstruction performance is evaluated after each epoch on the validation set and the learning rate decreased by a factor of 3 after the validation loss hasn't 't improved in 25 consecutive epochs for MONET and 100 epochs for VIMON, respectively. MONET and VIMON are trained for 600 and 1000 epochs, respectively. The checkpoint with the lowest reconstruction eтror is selected for the final MOT evaluation. MONET is trained with 0.5 and 7 and VIMON trained with
and y 2. K 5 for SpMOt, K 6 for VMDS and K = 8 for VOR. Due to the increased slot number for VOR, batch size for VIMON had to be decreased to to into the GPU memory. Respectively, the initial learning rate set to 0.000075 for VIMON on VOR. We initialize the attention network and the VAE in VIMON with the pre-trained weights from MONET to facilitate learning and speed up the training. Note that for all evaluations, the reconstructed masks in from the VAE were used.
Sprites-MOT Initialization: When training MONET and Video MONET on Sprites-MON from scratch, MONET struggles to learn the extreme color values of the objects that Sprites-MOT features. Instead completely focuses on learning the shapes. To circumvent that, we initialized the weights of the models with MONET weights that were trained for 100 epochs Multi-dSprites.
23

---

Under review as a conference paper at ICLR 2021
Table D.I: Architecture of VIMON VAE Encoder.
Type
Size/Ch.
Act. Func.
Comment
Input
RGB Mask
Conv 3x3
32
ReLU
Conv 3x3
32
ReLU
Conv 3x3
64
ReLU
Conv 3x3
ReLU
MLP
256
ReLU
MLP
32
Linear
Table D.2: Architecture of VIMON VAE Decoder.
Type
Size/Ch.
Act. Func.
Comment
Input
Broadcast
18
+ coordinates
Conv 3x3
32
ReLU
Conv 3x3
32
ReLU
Conv 3x3
32
ReLU
Conv 3x3
32
ReLU
Conv 1x1
Linear
RGB + Mask
D.2 TRACKING BY ANIMATION
Preprocessing: TBA expects its input frames to contain only foreground objects. In (He et al. 2019). the authors use Independent Multimodal Background Subtraction (IMBS) (Bloisi & locchi 2012 to remove the background from datasets consisting of natural videos with static backgrounds. Background subtraction algorithms maintain spatio-temporal window around each pixel in the sequence, and remove the dominant mode based on histogram of color values. Since the default implementation IMBS has several hand-tuned thresholds corresponding to natural videos (e.g., for shadow suppression), it cannot be directly applied to synthetic datasets like VMDS without significant nyper-parameter tuning. We instead re- generate all of the VMDS datasets with identical objects and motion but a black background for experiments with TBA, to mimic well-tuned background subtraction algorithm.
For SpMOT, follow the same architecture in (He al. 2019) while increase the number slots from K 4 to K = and number of layers from = to for VMDS. Since TBA does not model the background, this makes the number of foreground slots equal the other models in our study.
Architecture:
Further, increase the size prior parameters U x V used for the shape and appearance templates from 21 x 21 which is used for SpMOT, to 64 x 64 for VMDS, which we empirically found gave the best validation loss among 48 48, 56 56, 64 64 and 72 x 72. All other architectural choices are kept fixed for both datasets, and follow (He et al. 2019). Note that due to this, we trained the TBA models its default resolution of 128x 128 unlike the 64% x resolution used by MONET and OP3.
Training and Evaluation: We train for 1000 epochs using the same training schedule in (He) al. 2019). The checkpoint with the lowest validation loss selected for the final MOT evaluation. Further, we observed that the discrete nature of the shape code used in TBA's mid-level representation leads to salt-and-pepper noise in the reconstructed masks. We therefore use 2 minimum pooling operation the final output masks to remove isolated, single pixel foreground predictions and generate 64 X 64 resolution outputs, similar to MONET and OP3 before evaluation.
Deviation of SpMOT results compared to original publication: Our results were generated with 100k training frames, while the original TBA paper (He et al., 2019) uses 2M training frames for the simple SpMOT task. Further, report the mean of three training runs, while the original paper reports one run (presumably the best). Our best run achieves MOTA of 90.5 (Table Third, we evaluate using intersection over union (IoU) of segmentation masks instead of bounding boxes.
24

---

Under review as a conference paper at ICLR 2021
D.3 OP3
Training: The OP3 loss is a weighted sum over all refinement and dynamics steps (Eq. (26) For our evaluation multi-object tracking, we weight all time steps equally. In contrast to the original training loss, in which the weight value is linearly increased indiscriminately. thus weighting later predictions more highly, we perform the linear increase only for the refinement steps between dynamics steps, thus weighting all predictions equally.
OP3, published by (Veerapaneni et al. 2019), uses curriculum learning. For the first 100 epochs, M refinement steps are taken, followed by a single dynamics step, with a final refinement step afterwards. Starting after 100 epochs, the number of dynamics steps incremented by every 10 epochs, until five dynamics steps are reached. Thus, only 5 frames of the sequence are used during training maximum.
We chose to use an alternating schedule for training, where after each dynamics step, M = 2 refinement steps are taken, and this continued for the entire sequence. Thus, the entire available sequence is used, and error is not propagated needlessly, since the model is enabled to refine previous predictions on the reconstruction before predicting again. Note that this is the schedule OP3 uses by default at test-time, when it is used for model predictive control. Note that we still use 4 refinement steps on the initial observation to update the randomly initialized posterior parameters, as in the released implementation. We split all 10-step sequences into 5-step sequences to avoid premature divergence.
We train OP3 with a batch size of 16 for 300 epochs using learning rate of 0.0003 for VMDS and VOR and 0.0001 for SpMOT. K = 5 for SpMOT, K = 6 for VMDS and K = 8 for VOR are used. Larger learning rates for SpMOT led to premature divergence. Note OP3 by default uses batch size of 80 with the default learning rate of 0.0003, this led to suboptimal performance in our experiments. Finally, training OP3 is very unstable, leading to eventual divergence almost all experiments that have been performed for this study.
The checkpoint prior to divergence with the lowest KL loss is selected for the final MOT evaluation, as the KL loss enforces consistency in the latents over the sequence. Interestingly. the checkpoint almost always corresponded to the epochs right before divergence.
D.4 SCALOR
Architecture: We follow the same architecture in (Jiang et al. 2020). We use grid of 4 for object discovery with maximum number of objects of 10. The standard deviation of the image distribution is set to 0.1. Size anchor and variance are set to 0.2 and 0.1, respectively.
For SpMOT, background modeling is disabled and the dimensionality of the latent object appearance is set to 8.
For VMDS, the dimensionality of background is set to 3 and the dimensionality of the latent object appearance set to 16. For object discovery, grid of 3 x cells with a maximum number objects of 8 is used.
For VOR, the dimensionality of background set to 8 and the dimensionality of the latent object appearance is set to 16.
Hyperparameter tuning: For VMDS, we run hyperparameter search over number of grid cells (3 x 3, 4 x 4), background dimension [1, 3. 5), maximum number of objects (5, 8, 10) (dependent on number of grid cells), size anchor (0.2, 0.25, 0.3, 0.4), z what dimenisonality [8, 16, 24) and end value of tau (0.3, 0.5).
For SpMOT, we run hyperparameter search over maximum number of objects (4, 10), size anchor 10.1, 0.2, 0.3), what dimensionality (8, 16) and whether to model background (with background dimensionality 1) or not.
For VOR, we hyperparameter search over size anchor (0.2, 0.3) and background dimensionality (8, 12).
We picked best hyper parameters according the validation loss.
25

---

Under review as a conference paper at ICLR 2021
Fraction of failure cases [%6]
ID Switches
FPs
Misses
80
60
VIMON
40
OP3
SCALOR
20
2 3
4
1
2
4
1
2
3
4
Number objects video
Figure E.1: Distribution of failure cases dependent on number of objects in VMDS videos split by failure class. Mean of three training runs. Error bars: SD.
Training: We train SCALOR with batch size of 16 for 300 epochs using learning rate of 0.0001 for SpMOT and VOR and for 400 epochs for VMDS. For the final MOT evaluation, the checkpoint with the lowest loss on the validation set is chosen.
E ADDITIONAL RESULTS
Table E. lists the individual results for the three training runs with different random seeds per model and dataset. The results of VIMON and SCALOR are coherent between the three runs with different random seed, while TBA has one run SpMOT with significantly lower performance than the other two and shows variation the three training runs on VMDS. OP3 exhibits one training run on SpMOT with lower performance than the other two.
Fig. E. shows the fraction of failure cases dependent on the number of objects present in the video for the three different failure cases separately: ID switches, FPs and misses. For VIMON, TBA and SCALOR, the number of failures increase with the number of objects present regardless of the type of failure. In contrast, OP3 shows this pattern for ID switches and misses, while it accumulates higher number of false positives (FPs) in videos with fewer (only one or two) objects.
Fig. E.2 shows comparison between MONET and VIMON VMDS. MONET correctly finds and segments objects, but does not assign them to consistent slots over time, while VIMON maintains a consistent slot assignment throughout the video.
ViMON
MONet
Ground Truth
Reconstruction
Segmentation
t=3
t=5
t=7
t=9
t=3
t=5
t=9
Figure E.2: Comparison of MONET and VIMON on VMDS. Example sequence of dataset shown with corresponding outputs of the model. Reconstruction shows sum of components from all slots, weighted by the attention masks. Color-coded segmentation maps in third row signify slot-assignment. Note how the object-slot assignment changes for consecutive frames (3rd row) for MONET, while VIMON maintains a consistent slot assignment throughout the video.
Fig. E.4 4hows failures cases of OP3 on VOR.
26

---

Under review as a conference paper at ICLR 2021
Table E.2 and Table E.3 list the results for the four models, VIMON, TBA, OP3 and SCALOR, the VMDS challenge sets and out-of-distribution (o.o.d.) sets respectively. Results are shown as the mean and standard deviation of three training runs with different random seed per model.
Table E.1: Analysis of SOTA object-centric representation learning models for MOT. Results for three runs with different random training seeds.
Model
Run
MOTA
1OTP
MD f
MT
Match t Miss
IDS.
FPs
MSE +
SpMOT
70.0
90.6
92.8
49.4
74.7
4.1
21.2
10.4
MONET
90.0
21.6
13.4
71.3
88.1
91.6
53.8
77.1
4.9
18.0
5.8
15.2
92.7
92.0
87.5
87.0
94.9
4.9
0.2
2.2
10.5
VIMON
92.8
92.0
86.3
5.0
0.2
11.8
93.2
91.6
88.3
95.2
4.6
0.2
10.9
90.5
71.4
90.2
89.8
94.4
5.3
0.3
3,9
10.3
TBA
58.4
70.7
69.6
60.8
75.0
18.1
6.9
16.6
14.6
90.1
71.5
90.3
89.4
5.5
0.5
10.9
92.4
80.0
94.5
93.7
97.3
2.4
0.4
4.8
4.3
OP3
81.9
74.9
86.9
86.5
92.8
6.8
0.3
10.9
30.1
92.9
80.1
95.9
95.2
97.6
2.0
0.4
4.7
5.6
94.4
80.1
96.5
92.3
95.4
2.4
2.2
1.0
3.3
SCALOR
94.7
80.2
96.4
93.1
95.8
2.4
1.8
1.1
3.4
95.5
80.2
96.3
94.0
96.4
2.4
1.2
0.9
3.6
VOR
28.0
81.3
73.8
26.7
57.4
18.0
24.6
29.4
14.1
MONET
44.5
82.4
78.2
45.4
68.7
15.0
16.3
24.2
11.8
38.5
81.6
78.7
39.8
67.0
18.5
28.5
10.8
89.0
88.9
90.2
89.8
92.9
6.8
0.3
3.9
7.1
VIMON
89.0
89.8
89.9
89.6
93.0
6.8
0.2
4.0
6.2
89.0
89.9
91.0
90.6
93.8
6.0
0.2
4.8
5.9
64.8
89.5
87.2
85.1
90.3
8.8
0.9
25.5
3.1
OP3
66.2
88.1
88.6
85.1
90.7
7.9
1.4
24.5
2.9
65.3
89.3
88.2
86.1
91.1
8.0
0.9
25.8
3.0
74.1
85.8
75.6
75.5
77.4
22.6
0.0
3.3
6.4
SCALOR
74.6
86.0
75.9
75.9
78.1
21.9
0.1
3.5
6.4
75.1
86.1
76.5
76.4
78.2
21.7
0.0
3.1
6.3
VMDS
51.7
79.6
75.1
36.7
67.6
12.9
19.5
15.9
20.8
MONET
44.3
76.1
71.8
34,8
65.9
15.0
19.1
21.5
25.3
52.2
80.2
75.6
35.5
66.5
13.0
20.5
14.2
20.4
87.0
86.8
86.7
85.4
92.4
6.8
0.7
5.5
10.6
VIMON
87.1
86.8
86.1
85.1
92.3
7.1
0.6
5.3
10.8
86.5
86.7
86.0
84.6
92.1
7.2
0.7
5.6
10.6
68.5
76.1
69.3
65.3
80.7
16.5
2.8
12.2
26.0
TBA
38.9
73.8
55.1
50.5
70.2
26.6
3.2
31.3
30.8
56.0
75.0
64.3
59,2
76.7
19.8
3.5
20.8
27.5
94.2
1.9
0.2
OP3
92.7
93.4
96.9
96.3
97.8
2.0
0.2
5.1
4.3
89.4
93.3
96.2
95.8
97.6
2.2
0.2
8.3
4.6
75.7
88.1
69.4
68.3
79.8
19.4
0.8
4.0
13.9
SCALOR
72.7
87.2
65.6
0.8
14.2
73.7
87.6
67.5
66.2
77.9
21.2
0.9
4.2
14.0
27

---

Under review as a conference paper at ICLR 2021
Table E.2: Performance on VMDS challenge sets, Results shown as mean + standard deviation for three runs with different random training seeds. Examples sequences for each challenge set shown below.
Occtasion
Same Color
Small Objects
Large Objects
Model
MOTP
VIMON
0.04
2:1
70.4 1:01
70.7
TBA
37.5 124
72.8 a.0
38.3 1:16
47.2
94
73.0 0.67
45.2 1:38
74.3 6:27
71.9 0:84
66.3 1:4
25.6 150
73.4 1.9
44.7 26
OP3
85.3 1:10
91.6 0:04
89.6
8:00
51.5
1.
86.3
66.3 0:13
93.3 14
93.0 0:84
900
83.8
92.2 9222
93.5
SCALOR
58.8
10
86662.0
46.8
53.7
83.4
46.2
74.4
200
86.1
2:84
67
66.1
86.6 2.6
62.4
Doc usieo
Color
Objects
Lange Ohjerts
Table E.3: Performance VMDS OOD test sets Results shown mean + standard deviation for three runs with different random training seeds. Examples sequences for each o.o.d. set shown below.
VIMON
6:.4::25
78.0 1:03
11.3 #:1
06.8
57.4 3:4-
86.2 1:11
86.4 =.1
-10.4 ::40::400
20.5 2:44
39.5 ::14
29.8 10
VIMON
80.3..0
82.1
82.5 ::4
79.8 1:05
84.5 1.66
84.6
83.4 215
$1.8 2:3
78.7 1/6
82.0 2:2
79.2 2:14
76.4
TBA
52.3
73.3 1:11
59.8 448
5188
56.1..112
75.1
63.7
5:4
59.0 52
52.4 0.00
73.6 1:1
59.3
6:63
49.8
TBA*
1.3
68.4
30.6
248
-16.5
69.6
29.1
25.4
-7.5 =m
69.4
26.6
20.6
S70.119
93.5
87.1
95.9 8.6
83.6
95.5 65
SCALOR
68.1 2:12
84.9
1:44
63.3
60.0
89.9
14
65.7
1.
46.5 1
82.1 2:13
41.9 2:13
SCALORR
1:
85.2
1:6
61.2
57.1
1:07
73.3
89.8
648
11
63.0
0M
61.6
2:11
83.5
$3.4
1:15
50.2 21
Models trained on dalasel that leatured volor, size and orientation changes of abjects during the sequence.
Size
Color
Rotation
6+2
E.1 OUT-OF-DISTRIBLUS TEST SETS
To test whether the models can in principle learn additional object transformations as featured the VMDS o.o.d. sets, additionally train the models changes as well as rotation of objects. VIMON, OP3 and SCALOR are able to learn additional property changes of the objects when they are part of the training data while TBA fails learn tracking on this more challenging dataset (Fig. E.3 for absolute values Table E.3
training that includes size and color
E.2 STABILITY OF TRAINING AND RUNTIME
To assess runtime in fair way despite the mod- els being trained on different hardware, we re- port the training progress of all models after one hour of training addition, we quantify inference time on the full VMDS test set using batch size of one.
MOTA
single GPU (Table E.4). In
Performance
Test get
VMONN
0
SCALOR*
GP37
E.3 VIMON ABLATIONS
Size
Color
Rota-
Size
Color
Rota-
tion
tion
Removing the GRU or the mask conditioning of the attention network reduces tracking perfor- mance (MOTA on VMDS from 86.8% to 70.6% and 81 81.4%, respectively; Table E.5
Figure E.3: Performance out-of-distribution sets relative VMDS test set (100%). * indicates that models were trained on dataset that included color, size and orientation changes of objects.
F SUPPLEMENTARY FIGURES
See figures F. F.8 for additional, randomly picked examples of reconstruction and segmentation for VIMON, TBA, OP3 and SCALOR on the three datasets (VMDS, SpMOT and VOR).
28

---

Under review as a conference paper at ICLR 2021
Table E.4: Runtime analysis (using single RTX 2080 Ti GPU). Training: models trained on VMDS for one hour. Inference: models evaluated on VMDS test set with batch size=1 (10 frames).
Training
Inference
Model
Resolution
No. Param.
Batch Size
Memory [MiB]
No. lerr
Epochs
Memory [MiB]
rumtime batch
Total runtine
714,900
TBA
128s
3 3,8846444
4421
28.29
972
shi
4min 05s
OP3
64x64
876,305
10
10,874
2204
2.20
4092
0.54 sãt
9min Ols
SCALOR
64x64
2,763,526
10,942
2547
12.23
930
0.29 sht
4min 48t
The TBA parameter count scalles with the feature resolution, which kept using adaptive pooling. This makes the parameter count
independent of input resolution.
Table E.5: Ablation experiments for VIMON on VMDS.
Model
MOTA
MOTP 1
MT 1
Match f
Miss
ID S.
FPs.
MSE
VIMON
87.8
75.7
66.0
81.4
13.4
10.8
VIMON w/o GRU
81.4
86.9
79.8
77.3
88.2
10.3
1.4
6.8
18.9
t=1
t=3
t=
t=7
t= 9
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Segm. Recon.
Figure E.4: Failure cases of OP3 VOR. Exam- ple sequences of VOR test set shown with corre- sponding outputs of the model after final refine- ment step. Binarized colour-coded segmentation maps in third row signify slot-assigmment.
29

---

Under review as a conference paper at ICLR 2021
t=1
t=2
t=3
t=4
t=8
t=9
=0
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Segm. Recon. Truth
Figure F.1: Results of VIMON on VMDS. Random example sequences VMDS test set shown with corresponding outputs of the model. Reconstruction shows sum of components from all slots, weighted by the reconstructed masks from the VAE. Binarized colour-coded segmentation maps in third row signify slot-assignment.
30

---

Under review as a conference paper at ICLR 2021
t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
A
Segm.
Ground Truth
Segm. Recon.
Figure F.2: Results VIMON on SpMOT. Random example sequences of SpMOT test set shown with corresponding outputs of the model. Reconstruction shows sum of components from all slots, weighted by the reconstructed masks from the VAE. Binarized colour-coded segmentation maps in third row signify slot-assignment.
31

---

Under review as a conference paper at ICLR 2021
t=1
t=2
t=3
t=4
t=5
t=6
t=7
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Segm. Recon.
Figure F.:: Results VIMON on VOR, Random example sequences of VOR test set shown with corresponding outputs of the model. Reconstruction shows sum of components from all slots, weighted by the reconstructed masks from the VAE. Binarized colour-coded segmentation maps in third row signify slot-assignment.
32

---

Under review as a conference paper at ICLR 2021
t=1
t=2
t=3
t=4
5
t=6
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Segm. Recon.
Figure F:4: Results TBA on VMDS. Random example sequences of VMDS test set shown with corresponding outputs of the model. Binarized colour-coded segmentation maps in third row signify slot-assignment. Note that background subtraction is performed in the preprocessing of TBA, hence the black background in the reconstructions.
33

---

Under review as a conference paper at ICLR 2021
t=2
[=4
[=6
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
*
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
9
Ground
Truth
Recon.
Segm.
Figure F.5; Results of TBA on SpMOT. Random example sequences of SpMOT test set shown with corresponding outputs of the model. Binarized colour-coded segmentation maps in third row signify slot-assignment
34

---

Under review as a conference paper at ICLR 2021
[=4
t=6
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Segm. Recon.
Figure F.6: Results of OP3 VMDS. Random example sequences of VMDS test shown with corresponding outputs of the model after final refinement step. Binarized colour-coded segmentation maps third row signify assignetinm
35

---

Under review as a conference paper at ICLR 2021
t=2
[=4
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Segm. Recon.
Figure F.7: Results of OP3 on SpMOT. Random example sequences of SpMOT test set shown with corresponding outputs of the model after final refinement step. Binarized colour-coded segmentation maps third row signify alst-assignment.
36

---

Under review as a conference paper at ICLR 2021
t=3
[=4
t=5
t=6
t=7
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Segm. Recon.
Figure F.8: Results of OP3 VOR. Random example sequences of VOR test set shown with corresponding outputs of the model after final refinement step. Binarized colour-coded segmentation maps third row signify alst-assignment
37

---

Under review as a conference paper at ICLR 2021
[=4
t=6
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Recon.
Segm.
Figure F:9; Results of SCALOR on VMDS. Random example sequences of VMDS shown with corresponding outputs of the model. Binarized colour-coded segmentation maps in third row signify assinnssinmmnt.
38

---

Under review as a conference paper at ICLR 2021
t=2
[=4
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Segm. Recon.
Figure F.10: Results of SCALOR on SpMOT, Random example sequences of SpMOT test set shown with corresponding outputs of the model. Binarized colour-coded segmentation maps in third row signify alstnnesigmm
39

---

Under review as a conference paper at ICLR 2021
t=3
[=4
t=5
t=6
t=7
t=8
t=9
t=10
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground Truth
Recon.
Segm.
Ground
Truth
Recon.
Segm.
Figure F.11: Results of SCALOR on VOR. Random example sequences of VOR shown with corresponding outputs of the model. Binarized colour-coded segmentation maps in third row signify slot-assignment.
40


Reviews:
[{"review_id": "v_57zmIUwnj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a benchmark for the evaluation of unsupervised learning of object-centric representation. The benchmark consists of three datasets, multi-object tracking metrics and of the evaluation of four methods. The proposed dataset consists of three sets of video sequences, procedurally generated, which are either generated from slight variations of existing works (Sprites-MOT) or on the basis of existing datasets (dSpirites, Video Object Room). For evaluation, authors propose to use a slight variation of the protocol of the MOT challenge for evaluation (with the addition of a Mostly Detected measure which does not penalize ID switches). As part of the  paper, they also evaluate and discuss the performances of four object-centric representation models, one of them (Video MONet) being an extension of an existing approach, proposed as part of this paper, and the remaining being state of the art approaches for the task.\n\n**Paper Strengths**\n- Although I am not familiar with unsupervised learning of object-centric representation, I like the idea of proposing a common protocol for evaluation. Potentially, this can be useful for the community and can help to create a common ground for evaluation.\n- The benchmark is comprehensive, in that it contains both data and evaluation measures. The adoption of MOT metrics also appears to be a reasonable choice.\n- The comparison between different models is interesting, and authors have also added a custom designed novel method (Video MONet). They investigate a set of challenging scenarios and carry out out-of-distribution tests.\n- The paper is a pleasure to read, and authors have also made a good effort in writing a clear and comprehensive supplemental.\n\n**Paper Weaknesses**\nMy main concern about the paper is the lack of novelty. While I realize that the objective of the paper is to create a common evaluation background, I fail to see a sufficient level of quality and of novelty. In particular:\n- the dataset associated with the benchmark are mostly based on existing works - they might be appropriate for evaluating this task, but the level of contribution is a bit limited;\n- on the metrics, the only contribution is to suggest using MOT metrics, which are again pre-existing;\n- the experimental and the insights it gives, again, can foster the community towards better model, but it's not a sufficient contribution for ICLR in my view.\n\nOverall, I think the paper could be a nice contribution to the literature on unsupervised learning of object-centric representation, but it lacks sufficient contribution for ICLR, in my view. I would therefore suggest to reject the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "tkbFYWlIjG", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an empirical evaluation of a number of recent models for unsupervised object-based video modelling. Five different models are evaluated on three (partially novel) benchmarks, providing a unifying perspective on the relative performance of these models. Several common issues are identified and highlighted using challenge datasets: The reliance on color as a cue for object segmentation, occlusion, object size, and change in object appearance. The paper concludes with several ideas for alleviating these issues.\n\nStrengths:\n 1. The paper represents a much needed comparison of several related models which have previously not been evaluated on common benchmarks. Given the rapidly increasing number of competing models in this space, I believe analysis papers like this one serve an important role.\n 2. The paper highlights important weaknesses of unsupervised object models, such as the overreliance on color or the difficulties with handling occlusion. While I believe that some of these weaknesses were already known to the people working with these methods, they have not always been formally documented in the respective publications. This paper rectifies this, and also provides guidance as to the relative vulnerability of the different methods.\n 3. The methodology is convincing and thorough. Architecture and hyperparameter choices are clearly documented in the appendix, and the datasets have been published. \n\nWeaknesses:\n1. As an analysis paper, this publication does not provide specific new technical contributions to the issues it is evaluating. \n2. The results are not entirely conclusive, in that there is no clear best model, and their relative quality varies with datasets and metrics. That said, some things can be very clearly observed, for instance the importance of object size for the performance of TBA.\n\nOverall, the paper serves an important role in consolidating the ecosystem of unsupervised object representations. Given the increasing need for such analysis papers, and the competent execution, I recommend acceptance.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sJ386HAzlUb", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is actually very well written and tries to answer the question of how various unsupervised learning of object-centric representations to on controlled tasks (synthetic datasets). The positives:\n1. Designs a benchmark of three datasets of varying complexity\n2. Compare a single image model and four video models (total five) \n3. Defined clear metrics that are around precision, detection, segmentation, tracking \n\nThe cons to the paper:\n1. All of the datasets are synthetic and it would have been good to at least pick a real world dataset and confirm the conclusions stand\n2. Since, the core goal of the paper is not novelty but a better understanding of various models I would have liked for the discussion to have some clear conclusions and better structure (use X in scenario Y, Model Z needs to be extended for scenarios Y etc.). I feel this section was short and mostly verbose without a clear conclusion\n3. Instead of focusing on such a comprehensive set of things - 3 datasets, five models and multiple metrics it would have been better if authors did some minor extensions of the models and showcase novel directions. But, instead the paper is mostly an understand only work and i worry that the conclusions don't necessarily give clear future directions for other researchers to build on.\n\nGiven the cons and specifically on not a clear actionable suggestion on how to improve models and no analysis beyond synthetic datasets I am leaning towards a rating of below acceptance threshold.", "extra_scores": {}, "extra_text": {}}, {"review_id": "aUAlcH6LG4H", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, this paper is interesting in setting up a benchmark for unsupervised object representations which is a very important problem in computer vision, reinforcement learning, etc. But the paper contains a very small amount of information to make use of the benchmark. Following are my comments:\n\n1. The paper needs to clearly set the contributions. The video datasets are from self-made simulations or taken from other sources because the appendix cites many references for each part of the dataset. \n\n2. The appendices are linked in the paper but given in the supplementary. It is good to combine both in one paper.  The appendix doesn't require information of existing methods and the current version contains of appendix contain mostly these explanations. \n\n3. Apart from these, the paper is well written and useful in the community. But the empirical evaluation is not convincing. There could be many baseline approaches for this, for example, paper [1] and the methods it is comparing with. This is my major concern on the paper and if stated well, the recommendation can change. \n\n[1] SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition, ICLR 2020\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 reviews with mixed initial ratings: 7, 5, 4, 5. The main concerns of R1, R2 and R4, who gave unfavorable scores, included: lack of methodological novelty (analysis-only paper), absence of experiments on real data (3 synthetic-only benchmarks), missing baselines and an overall inconclusive discussion. At the same time R5 notes that the offered fair comparison between SOTA methods was indeed "much needed", and the paper can "serve an important role" in guiding future developments in the community. In response to that, the authors submitted a new revision and provided detailed answers to each of the reviews separately. R1, R2 and R4 did not participate in the discussion, and R5 stayed with the positive rating.
AC agrees with R5 that the provided analysis is insightful, and the effort put into organizing the research community around a single set of benchmarks and metrics is indeed valuable. However, given a simplistic nature of the proposed datasets and lack of other methodological contributions, the submission is not meeting the acceptance bar for ICLR. After discussion with PCs, the final recommendation is to reject.
Decision: Reject