Journal of Machine Learning Research 22 (2021) 1-61

Submitted 2/21; Revised 6/21; Published 6/21

Benchmarking Unsupervised Object Representations forVideo Sequences
Marissa A. Weis1,4

marissa.weis@uni-goettingen.de

Kashyap Chitta3,6

kashyap.chitta@tue.mpg.de

Yash Sharma4

yash.sharma@uni-tuebingen.de

Wieland Brendel4,5

wieland.brendel@uni-tuebingen.de

Matthias Bethge4,5

matthias.bethge@uni-tuebingen.deAndreas Geiger3,6

a.geiger@uni-tuebingen.deAlexander S. Ecker1,2,7

ecker@cs.uni-goettingen.deInstitute of Computer Science, University of Göttingen, Germany
2
Campus Institute Data Science, Göttingen, Germany
3
Department of Computer Science, University of Tübingen, Germany
4
Institute for Theoretical Physics, University of Tübingen, Germany
5Bernstein Center for Computational Neuroscience, Tübingen, Germany
6
Max Planck Institute for Intelligent Systems, Tübingen, Germany
7
Max Planck Institute for Dynamics and Self-Organization, Göttingen, Germany
1

Editor: Christoph Lampert

Abstract
Perceiving the world in terms of objects and tracking them through time is a crucial
prerequisite for reasoning and scene understanding. Recently, several methods have been
proposed for unsupervised learning of object-centric representations. However, since thesemodels were evaluated on different downstream tasks, it remains unclear how they compare
in terms of basic perceptual abilities such as detection, figure-ground segmentation andtracking of objects. To close this gap, we design a benchmark with four data sets of varying
complexity and seven additional test sets featuring challenging tracking scenarios relevant
for natural videos. Using this benchmark, we compare the perceptual abilities of four
object-centric approaches: ViMON, a video-extension of MONet, based on recurrent
spatial attention, OP3, which exploits clustering via spatial mixture models, as well as
TBA and SCALOR, which use explicit factorization via spatial transformers. Our results
suggest that the architectures with unconstrained latent representations learn more powerfulrepresentations in terms of object detection, segmentation and tracking than the spatialtransformer based architectures. We also observe that none of the methods are able togracefully handle the most challenging tracking scenarios despite their synthetic nature,suggesting that our benchmark may provide fruitful guidance towards learning more robust
object-centric video representations.
Keywords: Unsupervised learning, object-centric representations, benchmark, tracking,segmentation©2021 Marissa A. Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger andAlexander S. Ecker.
License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided
at http://jmlr.org/papers/v22/21-0199.html.

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

1. Introduction
Humans understand the world in terms of objects (Marr, 1982; Spelke and Kinzler, 2007;
Johnson, 2018). Being able to decompose our environment into independent objects that caninteract with each other is an important prerequisite for reasoning and scene understanding(Spelke, 1988). Similarly, an artificial intelligence system would benefit from the ability torepresent visual scenes in a structured way (Ullman et al., 2017; Lake et al., 2017; Greff et al.,2020) by both extracting objects and their interactions from video streams, and keeping
track of them over time.
Recently, there has been an increased interest in unsupervised learning of object-centric
representations. The key insight of these methods is that the compositionality of visual scenes
can be used to both discover (Eslami et al., 2016; Greff et al., 2019; Burgess et al., 2019)
and track objects in videos (Greff et al., 2017; van Steenkiste et al., 2018; Veerapaneni et al.,2020) without supervision. However, it is currently not well understood how the learned
visual representations of different models compare to each other quantitatively, since themodels have been developed with different downstream tasks in mind and have not been
evaluated using a common protocol. Hence, in this work, we propose a benchmark based onprocedurally generated video sequences to test basic perceptual abilities of object-centric
video models under various challenging tracking scenarios.
An unsupervised object-based video representation should (1) effectively identify objects
as they enter a scene, (2) accurately segment objects, as well as (3) maintain a consistent
representation for each individual object in a scene over time. These perceptual abilities canbe evaluated quantitatively in the established multi-object tracking framework (Bernardin
and Stiefelhagen, 2008; Milan et al., 2016). We propose to utilize this protocol for analyzing
the strengths and weaknesses of different object-centric representation learning methods,
independent of any specific downstream task, in order to uncover the different inductive
biases hidden in their choice of architecture and loss formulation. We therefore compiled
a benchmark consisting of four procedurally generated video data sets of varying levels ofvisual complexity and two generalization tests. Using this benchmark, we quantitatively
compared three classes of object-centric models, leading to the following insights:• All of the models have shortcomings handling occlusion, albeit to different extents.
• OP3 (Veerapaneni et al., 2020) performs strongest in terms of quantitative metrics, but
exhibits a surprisingly strong dependency on color to separate objects and accumulates
false positives when fewer objects than slots are present.
• Spatial transformer models, TBA (He et al., 2019) and SCALOR (Jiang et al., 2020),
train most efficiently and feature explicit depth reasoning in combination with amodal
masks, but are nevertheless outperformed by the simpler model, ViMON, lacking adepth or interaction model, suggesting that the proposed mechanisms may not yet
work as intended.
Our code, data, as well as a public leaderboard of results is available at https://eckerlab.org/code/weis2021/.
2

Benchmarking Unsupervised Object Representations

2. Related work
Several recent lines of work propose to learn object-centric representations from visual inputs
for static and dynamic scenes without explicit supervision. Though their results are promising,methods are currently restricted to handling synthetic data sets and as of yet are unable
to scale to complex natural scenes. Furthermore, a systematic quantitative comparison of
methods is lacking.
Selecting and processing parts of an image via spatial attention has been one prominent
approach for this task (Mnih et al., 2014; Eslami et al., 2016; Kosiorek et al., 2018; Burgess
et al., 2019; Yuan et al., 2019; Crawford and Pineau, 2019). As an alternative, spatial
mixture models decompose scenes by performing image-space clustering of pixels that belong
to individual objects (Greff et al., 2016, 2017, 2019; van Steenkiste et al., 2018; Locatello
et al., 2020). Xu et al. (2019) use optic flow to learn to segment objects or object partsin dynamic scenes. While some approaches aim at learning a suitable representation fordownstream tasks (Watters et al., 2019a; Veerapaneni et al., 2020), others target scene
generation (Engelcke et al., 2020; Kügelgen et al., 2020; Ehrhardt et al., 2020; Jiang and
Ahn, 2020). We analyze three classes of models for processing videos, covering three models
based on spatial attention and one based on spatial mixture modeling.
Spatial attention models with unconstrained latent representations use perobject variational autoencoders, as introduced by Burgess et al. (2019). Kügelgen et al.
(2020) adapt this approach for scene generation. So far, such methods have been designed
for static images, but not for videos. We therefore extend MONet (Burgess et al., 2019) tobe able to accumulate evidence over time for tracking, enabling us to include this class ofapproaches in our evaluation. Recent concurrent work on AlignNet (Creswell et al., 2020)
applies MONet frame-by-frame and tracks objects by subsequently ordering the extracted
objects consistently.
Spatial attention models with factored latents use an explicit factorization of thelatent representation into properties such as position, scale and appearance (Eslami et al., 2016;
Crawford and Pineau, 2019). These methods use spatial transformer networks (Jaderberg
et al., 2015) to render per-object reconstructions from the factored latents (Kosiorek et al.,2018; He et al., 2019; Jiang et al., 2020). SQAIR (Kosiorek et al., 2018) does not performsegmentation, identifying objects only at the bounding-box level. Henderson and Lampert(2020) explicitly reason about the underlying 3D scene structure. We select Tracking-byAnimation (TBA) (He et al., 2019) and SCALOR (Jiang et al., 2020) for analyzing spatialtransformer methods in our experiments, which explicitly disentangle object shape andappearance, providing access to object masks.
Spatial mixture models cluster pixels using a deep neural network trained with expectation maximization (Greff et al., 2017; van Steenkiste et al., 2018). IODINE (Greffet al., 2019) extends these methods with an iterative amortised variational inference procedure (Marino et al., 2018), improving segmentation quality. SPACE (Lin et al., 2020)combines mixture models with spatial attention to improve scalability. To work with video
sequences, OP3 (Veerapaneni et al., 2020) extends IODINE by modeling individual objects’dynamics as well as pairwise interactions. We therefore include OP3 in our analysis as arepresentative spatial mixture model.
3

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

Objects

Background

Data Set

Shape

MotionCount Over Sequence

Size Variation

Orientation

Color

Motion

Color

SpMOT
VOR
VMDS
texVMDS

4 Templates (2D)
2 Templates (3D)
3 Templates (2D)
3 Templates (2D)LinearStatic
Non-Linear
Non-Linear

Varies (0-3)
Varies (0-4)
Fixed (1-4)
Fixed (1-4)

MinimalModerate
Moderate
Moderate

Fixed
RandomRandomRandom6 Colors
6 Colors
2563 Colors
ImageNet textures

None
Moving Camera
None
LinearBlack
Random2563 Colors
ImageNet crop

SpMOT

t=1

VOR

t=3

t=5

t=7

t=9

t=1

VMDS

t=3

t=5

t=7

t=9

t=1

t=3

t=5

t=7

t=9

Table 1: Summary of data sets and example video sequences. See Appendix B for details.

3. Object-Centric Representation Benchmark
To compare the different object-centric representation learning models on their basic perceptual abilities, we use the well-established multi-object tracking (MOT) protocol (Bernardin
and Stiefelhagen, 2008). In this section, we describe the data sets and metrics considered inour benchmark, followed by a brief description of the models evaluated.
3.1 Data Sets
Current object-centric models are not capable of modeling complex natural scenes (Burgess
et al., 2019; Greff et al., 2019; Lin et al., 2020). Hence, we focus on synthetic data sets that
resemble those which state-of-the-art models were designed for. Specifically, we evaluate
on four synthetic data sets1 (see Table 1), which cover multiple levels of visual and motion
complexity. Synthetic stimuli enable us to precisely generate challenging scenarios in acontrollable manner in order to disentangle sources of difficulty and glean insights on whatmodels specifically struggle with. We design different scenarios that test complexities thatwould occur in natural videos such as partial or complete occlusion as well as similar object
appearances and complex textures.
Sprites-MOT (SpMOT, Table 1 left), as proposed by He et al. (2019), features simple
2D sprites moving linearly on a black background with objects moving in and out of frame
during the sequence. Video-Multi-dSprites (VMDS, Table 1 right) is a video data set we
generated based on a colored, multi-object version of the dSprites data set (Matthey et al.,2017). Each video contains one to four sprites that move non-linearly and independentlyof each other with the possibility of partial or full occlusion. Besides the i.i.d. sampledtraining, validation and test sets of VMDS, we generate seven additional challenge sets that
we use to study specific test situations we observed to be challenging, such as guaranteed
occlusion, specific object properties, or out-of-distribution appearance variations. Video
Objects Room (VOR, Table 1 middle) is a video data set we generated based on the static
Objects Room data set (Burgess et al., 2019), which features static objects in a 3D roomwith a moving camera. Textured Video-Multi-dSprites (texVMDS, Table 3) is basedon the generative model of VMDS regarding shapes, number and movement dynamics ofthe objects, but instead of being uniformly colored the objects feature random texture crops
taken from ImageNet (Deng et al., 2009). Additionally, the background is a linearly moving
ImageNet crop. For full details on the data sets and their generation, see Appendix B.
1

Data sets are available at this URL.

4

Benchmarking Unsupervised Object Representations

3.2 Metrics
Our evaluation protocol follows the multi-object tracking (MOT) challenge, a standard
and widely-used benchmark for supervised object tracking (Milan et al., 2016). The MOT
challenge uses the CLEAR MOT metrics (Bernardin and Stiefelhagen, 2008), which quantitatively evaluate different performance aspects of object detection, tracking and segmentation.
To compute these metrics, predictions have to be matched to ground truth. Unlike Bernardin
and Stiefelhagen (2008) and Milan et al. (2016), we use binary segmentation masks forthis matching instead of bounding boxes, which helps us better understand the models’segmentation capabilities. We consider an Intersection over Union (IoU) greater than 0.5
as a match (Voigtlaender et al., 2019). We include an ablation experiment to test how thisthreshold affects the results. The error metrics used are the fraction of Misses (Miss),
ID switches (ID S.) and False Positives (FPs) relative to the number of ground truth
masks. In addition, we report the Mean Squared Error (MSE) of the reconstructed
image outputs summed over image channels and pixels.
To quantify the overall number of failures, we use the MOT Accuracy (MOTA), whichmeasures the fraction of all failure cases compared to the total number of objects present inall frames. A model with 100% MOTA perfectly tracks all objects without any misses, ID
switches or false positives. To quantify the segmentation quality, we define MOT Precision(MOTP) as the average IoU of segmentation masks of all matches. A model with 100%
MOTP perfectly segments all tracked objects, but does not necessarily track all objects.
Further, to quantify detection and tracking performance independent of false positives, wemeasure the Mostly Detected (MD) and Mostly Tracked (MT) metrics, the fractionof ground truth objects that have been detected and tracked for at least 80% of their lifespan,
respectively. If an ID switch occurs, an object is considered detected but not tracked. For
full details regarding the matching process and the evaluation metrics, refer to Appendix A.
3.3 Models
We consider one color-segmentation baseline and three classes of unsupervised object-centric
representation learning models: (1) a spatial attention model with unconstrained latents, ViMON, which is our video extension of MONet (Burgess et al., 2019); (2) spatial transformerbased attention models, TBA (He et al., 2019) and SCALOR (Jiang et al., 2020); (3) ascene mixture model, OP3 (Veerapaneni et al., 2020). At a high-level, these methods share
a common structure which is illustrated in Fig. 1a. They decompose an image into a fixed
number of slots (Burgess et al., 2019), each of which contains an embedding zt,k and amask mt,k of (ideally) a single object. These slots are then combined in a decoding step
to reconstruct the image. Below, we briefly describe each method. Appendix C provides adetailed explanation of the methods in a unified mathematical framework.
As a simple baseline, we evaluate the performance of k-Means clustering on the RGB
pixel values on a per frame basis and tracking performance by frame-to-frame matchingbased on the IoU of the segmentation masks using Hungarian matching (Kuhn, 1955). For
full details refer to Appendix D.1.
Video MONet (ViMON) is our video extension of MONet (Burgess et al., 2019).
MONet recurrently decomposes a static scene into slots, using an attention network tosequentially extract attention masks mk ∈ [0, 1]H×W of individual objects k. A Variational
5

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

K slots parallel

K slots sequential

Reconstruction

mt+1,K

DecoderSlot 1

Masks mt,1

mt,K

Latents zt,1zt,KSlot 1

Slot 2 Slot K

Slot 2 Slot K

zt+1,K

GRU

...

VAE
Encoder

e
Tim
t +1

Encoder

VAE
Dec.

K slots parallel

SpatialTransformer(x,y)...

M refinements
sequential

SpatialTransformer
BGNetwork

K slots parallel

VAE
Dec.

(x,y)...

MLP+
Tracker
Array

Attn.
Net

FeatureExtractor

Dyn.
Net

LSTM
Reﬁne
Net

Rejection

MLP
Proposal
Tracker
RNNs
FeatureExtractor

ImageVideo frame t(a) Common Principle(b) ViMON(c) TBA(d) OP3(e) SCALOR

Figure 1: Common principles of all models: Decomposition of an image into a fixed numberof slots, each of which contains an embedding zt,k and a mask mt,k of (ideally) a single object.
Dotted lines: temporal connections. Solid lines: information flow within one frame.
Autoencoder (VAE) (Kingma and Welling, 2014) encodes each slot into a latent representation
zk ∈ RL of the corresponding object. We use MONet as a simple frame-by-frame baseline
for detection and segmentation that does not employ temporal information. ViMON
accumulates evidence about the objects over time to maintain a consistent object-slot
assignment throughout the video. This is achieved by (1) seeding the attention network theb t,k ∈ [0, 1]H×W from the previous time step and (2) introducing a gatedpredicted mask mrecurrent unit (GRU) (Cho et al., 2014), which aggregates information over time for eachslot separately, enabling it to encode motion information. For full details on MONet and
ViMON, as well as information regarding hyperparameter tuning and ablations to provide
context for the design decisions, refer to Appendix C.1, C.2 and E.3.
Tracking-by-Animation (TBA) (He et al., 2019) is a spatial transformer-based attention model. Frames are encoded by a convolutional feature extractor f before being passed
to a recurrent block g called Reprioritized Attentive Tracking (RAT). RAT re-weights slot
input features based on their cosine similarity with the slots from the previous time stepand outputs latent representations for all K slots in parallel. Each slot latent is further
decoded into a mid-level representation yt,k consisting of pose and depth parameters, as well
as object appearance and shape templates (see Fig. 1c). For rendering, a Spatial Transformer
Network (STN) (Jaderberg et al., 2015) is used with an additional occlusion check basedon the depth estimate. TBA is trained on frame reconstruction with an additional penalty
for large object sizes to encourage compact bounding boxes. TBA can only process scenes
with static backgrounds, as it preprocesses sequences using background subtraction (Bloisi
and Iocchi, 2012). For full details on TBA as well as information regarding hyperparametertuning, refer to Appendix C.3.
Object-centric Perception, Prediction, and Planning (OP3) (Veerapaneni et al.,2020) extends IODINE (Greff et al., 2019) to operate on videos. IODINE decomposes animage into objects and represents them independently by starting from an initial guess of thesegmentation of the entire frame, and subsequently iteratively refines it using the refinement
network f (Marino et al., 2018). In each refinement step m, the image is represented by K
slots with latent representations zm,k . OP3 applies IODINE to each frame xt to extract
latent representations zt,m,k , which are subsequently processed by a dynamics network d (see6

Benchmarking Unsupervised Object Representations

Model

MOTA ↑

MOTP ↑MD ↑MT ↑Match ↑Miss ↓

ID S. ↓

FPs ↓

MSE ↓

19.3 ± 0.0
4.4 ± 0.4
4.8 ± 0.2
9.6 ± 6.0
3.7 ± 2.2
2.4 ± 0.0

0.2 ± 0.0
20.3 ± 1.6
0.2 ± 0.0
2.6 ± 3.0
0.4 ± 0.0
1.7 ± 0.4

44.3 ± 0.0
5.1 ± 0.5
2.1 ± 0.1
8.1 ± 6.0
6.8 ± 2.9
1.0 ± 0.1

13.0 ± 2.0
11.1 ± 0.6
11.9 ± 1.9
13.3 ± 11.9
3.4 ± 0.1

22.1 ± 0.0
15.8 ± 1.6
6.5 ± 0.4
8.2 ± 0.4
22.1 ± 0.4

5.2 ± 0.0
19.8 ± 3.5
0.3 ± 0.0
1.1 ± 0.2
0.0 ± 0.0

110.7 ± 0.1
27.4 ± 2.3
4.2 ± 0.4
25.3 ± 0.6
3.3 ± 0.2

12.2 ± 1.4
6.4 ± 0.6
3.0 ± 0.1
6.4 ± 0.1

1.0 ± 0.0
13.6 ± 1.0
7.0 ± 0.2
21.0 ± 4.2
2.0 ± 0.1
20.7 ± 1.0

2.6 ± 0.0
19.7 ± 0.6
0.7 ± 0.0
3.2 ± 0.3
0.2 ± 0.0
0.8 ± 0.0

99.7 ± 0.0
17.2 ± 3.1
5.5 ± 0.1
21.4 ± 7.8
6.1 ± 1.5
4.4 ± 0.4

22.2 ± 2.2
10.7 ± 0.1
28.1 ± 2.0
4.3 ± 0.2
14.0 ± 0.1

SpMOT
k-Means
MONet
ViMON
TBA
OP3
SCALOR

36.2 ± 0.0
70.2 ± 0.8
92.9 ± 0.2
79.7 ± 15.0
89.1 ± 5.1
94.9 ± 0.5

77.6 ± 0.0
89.6 ± 1.0
91.8 ± 0.2
71.2 ± 0.3
78.4 ± 2.4
80.2 ± 0.1

76.8 ± 0.0
92.4 ± 0.6
87.7 ± 0.8
83.4 ± 9.7
92.4 ± 4.0
96.4 ± 0.1

76.3 ± 0.0
50.4 ± 2.4
87.2 ± 0.8
80.0 ± 13.6
91.8 ± 3.8
93.2 ± 0.7

80.5 ± 0.0
75.3 ± 1.3
95.0 ± 0.2
87.8 ± 9.0
95.9 ± 2.2
95.9 ± 0.4
VOR

k-Means
MONet
ViMON
OP3
SCALOR-38.0 ± 0.1
37.0 ± 6.8
89.0 ± 0.0
65.4 ± 0.6
74.6 ± 0.4

76.5 ± 0.0
81.7 ± 0.5
89.5 ± 0.5
89.0 ± 0.6
86.0 ± 0.2

69.9 ± 0.1
76.9 ± 2.2
90.4 ± 0.5
88.0 ± 0.6
76.0 ± 0.4

62.9 ± 0.1
37.3 ± 7.8
90.0 ± 0.4
85.4 ± 0.5
75.9 ± 0.4

k-Means
MONet
ViMON
TBA
OP3
SCALOR-3.3 ± 0.0
49.4 ± 3.6
86.8 ± 0.3
54.5 ± 12.1
91.7 ± 1.7
74.1 ± 1.2

89.8 ± 0.0
78.6 ± 1.8
86.8 ± 0.0
75.0 ± 0.9
93.6 ± 0.4
87.6 ± 0.4

98.3 ± 0.0
74.2 ± 1.7
86.2 ± 0.3
62.9 ± 5.9
96.8 ± 0.5
67.9 ± 1.1

93.3 ± 0.1
35.7 ± 0.8
85.0 ± 0.3
58.3 ± 6.1
96.3 ± 0.4
66.7 ± 1.1

72.7 ± 0.1
64.4 ± 5.0
93.2 ± 0.4
90.7 ± 0.3
77.9 ± 0.4
VMDS96.4 ± 0.0
66.7 ± 0.7
92.3 ± 0.2
75.9 ± 4.3
97.8 ± 0.1
78.4 ± 1.0

Table 2: Analysis of SOTA object-centric representation learning models for MOT. Results
shown as mean ± standard deviation of three runs with different random training seeds.

Fig. 1d), which models both the individual dynamics of each slot k as well as the pairwiseinteraction between all combinations of slots, aggregating them into a prediction of theposterior parameters for the next time step t + 1 for each slot k. For full details on IODINE
and OP3, refer to Appendix C.4 and C.5, respectively.
SCALable Object-oriented Representation (SCALOR) (Jiang et al., 2020) is aspatial transformer-based model that factors scenes into background and multiple foregroundobjects, which are tracked throughout the sequence. Frames are encoded using a convolutional
LSTM f . In the proposal-rejection phase, the current frame t is divided into H × W grid cells.
For each grid cell a object latent variable zt,h,w is proposed, that is factored into existence,pose, depth and appearance parameters. Subsequently, proposed objects that significantly
overlap with a propagated object are rejected. In the propagation phase, per object GRUs
are updated for all objects present in the scene. Additionally, SCALOR has a backgroundmodule to encode the background and its dynamics. Frame reconstructions are renderedusing a background decoder and foreground STNs for object masks and appearance. For
full details on SCALOR as well as information regarding hyperparameter tuning, refer toAppendix C.6.

4. Results
We start with a summary of our overall results across the three data sets and four models(Table 2) before analyzing more specific challenging scenarios using variants of the VMDS
data set.
7

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

A simple color segmentation algorithm is not sufficient to solve tracking even in simplistic
data sets (Table 2), in which color is a good predictor for object identity. While k-Means
does well at detecting objects, on VMDS even outperforming the other models, its overall
tracking performance is significantly worse than the other models, mainly due to a highnumber of false positives and in SpMOT and VOR also due to a high number of misses,
which occur in these two data sets due to multiple objects often having the same color, whicha color segmentation algorithm is by definition not able to separate. In the following, we will
therefore focus on analyzing the object-centric methods.
We first ask whether tracking could emerge automatically in an image-based model like
MONet, which may produce consistent slot assignments through its learned object-slot
assignment. This is not the case: MONet exhibits poor tracking performance (Table 2).
While MONet correctly finds and segments objects, it does not assign them to consistentslots over time (Fig. E.4). In the following, we will thus focus on the video models: ViMON,
TBA, OP3 and SCALOR.
SpMOT. All models perform tracking well on SpMOT with the exception of one training
run of TBA with poor results leading to high standard deviation (cp. best TBA model:
89.8% MT; Table E.1). SCALOR outperforms the other models on the detection andtracking metrics MD and MT, while ViMON exhibits the highest MOTP, highlighting its
better segmentation performance on SpMOT.
VOR. TBA is not applicable to VOR due to the dynamic background which cannot be
resolved using background subtraction. ViMON and OP3 show similarly good performanceon detection (MD) and segmentation (MOTP), while ViMON outperforms OP3 on the
tracking metrics MOTA and MT. OP3 accumulates a high number of false positives leading
to a low MOTA due to the splitting of objects into multiple masks as well as randomly
segmenting small parts of the background (Fig. E.5). In contrast, SCALOR has almost no
false positives or ID switches, but accumulates a high number of misses leading to a poor
MOTA. It often segments two objects as one that are occluding each other in the first frame,
which is common in VOR due to the geometry of its scenes (Fig. F.17, last row).
VMDS. OP3 outperforms the other models on VMDS, on which TBA performs poorly,
followed by SCALOR, which again accumulates a high number of misses. We will analyze
the models on VMDS qualitatively and quantitatively in more detail in the following.
4.1 Impact of IoU Threshold for Matching
To examine whether the IoU threshold used for matching object predictions to ground truth
impacts the ordering of the models, we compute all metrics with regard to the IoU thresholds[0.1 .. 0.9] in steps of 0.1 on the VMDS test set. The ordering of the models with regard totheir MOTA is consistent over all matching thresholds except for a threshold of 0.9, but theperformance difference between the models increases with higher thresholds (Fig. 2a). Thus,
we will perform all following experiments with an IoU threshold of 0.5. For all metrics, seeFig. E.6 in Appendix.
4.2 Accumulation of Evidence over Time
Recognition and tracking of objects should improve if models can exploit prior knowledge
about the objects in the scene from previous video frames. To test whether the models
8

0
50
100

0.1

ViMON
TBA
OP3
SCALOR0.3 0.5 0.7
IoU Threshold

MOTP

MOTA

50

0.9

95
90
85
80
75
70

0.1

0.3 0.5 0.7
IoU Threshold

MOTA

2
0

ViMON
TBA
OP3
SCALOR
Warm start=22
4

0 1 2 3 4 5 6 7 8 9 10
Number of frames for warm start

0.9

(a)

(b)

Fraction of failure cases [%]100

Performance difference [%]Benchmarking Unsupervised Object Representations

80
60

ViMON
TBA
OP3
SCALOR

40
20
0

1
2
3
4
Number of objects in video(c)

Figure 2: a) MOTA and MOTP performance of models shown for different IoU thresholdused for matching model predictions to ground truth objects. Note that MOTA can become
negative, since the number of FPs is unbounded. b) MOTA on frames 11–20 of the VMDS
test set with warm starts of 1–10 frames (0 = no warm start). Difference to performance ofwarm start = 2 shown. c) Distribution of failure cases dependent on number of objects in
VMDS videos. Mean of three training runs. Error bars: standard deviation.
exploit such knowledge, we evaluate their MOTA performance on VMDS after warm startingwith up to 10 frames which are not included in evaluation (Fig. 2b). Note that the models
were trained on sequences of length 10, but are run for 20 frames in the case of a warm
start of 10 frames. The performance of ViMON improves with longer warm starts, showingthat the GRU accumulates evidence over time. TBA, in contrast, does not use temporal
information beyond 2–3 frames, while SCALOR’s performance slightly drops after 3 frames.
OP3 appears to most strongly rely on past information and is able to integrate information
over longer time scales: its performance does not even saturate with a warm start of 10
frames. However, the effect for all models is rather small.
4.3 Challenging Scenarios for Different Models
The number of objects in the sequence matters for ViMON, TBA and SCALOR: moreobjects increase the number of failure cases (Fig. 2c). In contrast, OP3 does not exhibit thispattern: it accumulates a higher number of false positives (FPs) in videos with fewer (onlyone or two) objects (Fig. E.1), as it tends to split objects into multiple slots if fewer objects
than slots are present.
Occlusion leads to failure cases for all models (Fig. 3a–b). Partial occlusion can lead tosplitting of objects into multiple slots (Fig. 3a). Objects that reappear after full occlusion
are often missed when only a small part of them is visible (Fig. 3a). In particular, SCALOR
tends to segment two objects as one when they overlap while entering the scene, leading to ahigh number of misses.
Color of the object is important. TBA often misses dark objects (Fig. 3b). In contrast,
ViMON, OP3 and SCALOR struggle with scenes that feature objects of similar colors aswell as objects that have similar colors to the background (Fig. 3c,e).
False positives are more prevalent for OP3 and TBA than for ViMON and SCALOR(Table 2). FPs of OP3 are due to objects split in multiple masks (Fig. 3a) and random
9

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=3

t=9

t=5

t=7

ViMON
Recon.

Ground
Truth

t=9

TBA
Recon.

OP3
Segm.

SCALOR
Recon.

GroundTruth

SCALOR
Segm.

TBA
Recon.

SCALOR
Recon.

t=7

OP3
Segm.

GroundTruth
OP3
Recon.

OP3
Segm.
TBA
Recon.
TBA
Segm.

(c) OP3 misses object with similarcolor to background.

(e) Models struggle with similar
object color.

TBA
Segm.

SCALOR
Segm.

t=5

ViMON
Segm.

TBA
Recon.
TBA
Segm.

GroundTruth
ViMON
Recon.
ViMON
Segm.
OP3
Recon.

(b) TBA struggles with dark objects.

(a) Models struggle withocclusion.

t=3

t=9

OP3
Recon.

t=7

TBA
Segm.

t=5

Ground
Truth

t=3(d)TBA splits large objects into
multiple masks.

Figure 3: Example failure cases for all models on VMDS. Segmentation masks are binarized
and color-coded to signify slot assignment.
small parts of the background being individually segmented (Fig. 3e), while TBA tends tocompose larger objects using multiple smaller, simpler components (Fig. 3d).
4.4 Performance on Challenge Sets
Based on the challenging scenarios identified above, we design multiple ‘challenge sets’: videos
featuring (1) heavy occlusion, (2) objects with same colors, (3) only small objects and (4)only large objects (Fig. 4a, top). For details, see Appendix B.1.1.
Occlusion reduces performance of all models compared with the i.i.d. sampled VMDS
test set, albeit to different degrees (Fig. 4a; for absolute performance see Table E.3). OP3 ismore robust to occlusion than the other models.
Tracking objects with the same color is challenging for all models (Fig. 4a). In particular,
OP3 appears to rely on object color as a way to separate objects.
OP3, ViMON and SCALOR are not sensitive to object size (Fig. 4a). They exhibit
only slightly decreased performance on the large objects test set, presumably because large
objects cause more occlusion (Fig. 4a). TBA shows increased performance on small objects
but performs poorly on the large objects set.
10

Benchmarking Unsupervised Object Representations

Occlusion

Same Color

Size

Color

Small Objects

Large Objects

Rotation

t=1

t=4

150

t=7

t=10

MOTA

t=1

t=4

t=7

t=10

t=1

t=4

t=7

100
50
Occlu- Same Small Large Occlu- Same Small Large
sioncolor objects objects sioncolor objects objects

t=7

MT

100
Test set

Test set
ViMON
TBA
OP3
SCALOR

ViMON
50
TBA
OP3
0
SCALOR
Size

(a)

t=10

t=10

MOTA

MT

Performance %Performance %

t=1

t=4

Color

Rotation

Size

Color

Rotation(b)

Figure 4: a) Performance on challenge sets relative to performance on VMDS test set (100%)and example video sequences. b) Performance on out-of-distribution sets relative to VMDS
test set (100%) and example video sequences. Note that MOTA can become negative, since
the number of FPs is unbounded.

4.5 Performance on Out-of-distribution Test Sets
Next, we assess generalization to out-of-distribution (o.o.d.) changes in object appearance
that are not encountered during training. In the training set of VMDS, object color, sizeand orientation are constant throughout a video. To test o.o.d. generalization, we evaluate
models trained on VMDS on three data sets that feature unseen object transformations(Fig. 4b and Table E.4): continuous changes in object color or size as well as continuous
rotation around the object’s centroid while moving. For details, see Appendix B.1.2.
Continuous changes in object size do not pose a serious problem to TBA, OP3 and
SCALOR, while ViMON’s performance drops (Fig. 4b). Surprisingly, continuous color
changes of objects do not impact the performance of any model. Tracking performance of
ViMON drops significantly for rotated objects, while OP3 and SCALOR are affected less.
TBA’s tracking performance is not as strongly influenced by object rotation (for absolutevalues, see Table E.4).
4.6 Stability of Training and Runtime
TBA and SCALOR train faster and require less memory than OP3 and ViMON (seeTable E.5 for details). However, two training runs converged to suboptimal minima for
TBA (note the high variance in Table 2 and see Table E.1 and Table E.2 for individualruns). Training OP3 is sensitive to the learning rate and unstable, eventually diverging in all
experiments. Interestingly, it often reached its best performance directly prior to divergence.
ViMON and TBA are less sensitive to hyperparameter settings in our experiments. For a
more detailed analysis of the runtime, see Appendix E.2.
11

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

Model

MOTA ↑

MOTP ↑MD ↑MT ↑Match ↑Miss ↓

ID S. ↓

FPs ↓

MSE ↓

69.2 ± 0.0
73.1 ± 5.1
65.0 ± 1.7
76.9 ± 6.7
87.5 ± 0.4

0.5 ± 0.0
2.2 ± 0.8
0.3 ± 0.0
0.2 ± 0.1
0.2 ± 0.0

129.8 ± 0.0
98.0 ± 1.7
120.2 ± 2.5
133.4 ± 2.9
111.5 ± 11.4

200.5 ± 5.7
171.4 ± 3.3
132.8 ± 16.2
133.7 ± 11.1

texVMDS
k-Means
MONet
ViMON
OP3
SCALOR

t=1-99.5 ± 0.0-73.3 ± 5.5-85.5 ± 2.8-110.4 ± 4.3-99.2 ± 11.7

t=3

t=5

76.4 ± 0.0
67.7 ± 1.1
69.0 ± 0.6
70.6 ± 0.6
74.0 ± 0.5

t=725.3 ± 0.0
16.0 ± 3.4
24.2 ± 1.3
16.5 ± 5.1
6.5 ± 0.6

t=9

t=1

24.3 ± 0.1
12.3 ± 3.1
23.8 ± 1.4
16.2 ± 5.0
6.3 ± 0.6

t=3

30.3 ± 0.0
24.7 ± 4.7
34.7 ± 1.7
22.9 ± 6.6
12.3 ± 0.4

t=5

t=7

t=9

t=1

t=3

t=5

t=7

t=9

Table 3: MOT performance on texVMDS and example video sequences. Results shown asmean ± standard deviation of three runs with different random training seeds. Note that
MOTA can become negative, since the number of FPs is unbounded.
4.7 Towards natural data
Object-centric methods have mostly been shown to work on synthetic data sets and seem torely to a certain extent on color segmentation. To quantify how these methods fare withmore realistic images, we created a fourth data set that keeps the simple scene composition of
VMDS (1–4 object and few shapes), but has a higher level of visual complexity. To this end,
we generated moving object masks as in VMDS, but filled each object and the backgroundwith the content of a different image taken from ImageNet (Table 3, bottom). We refer tothis data set as texVMDS. Humans can easily detect objects in these video through motion
cues,2 while the evaluated models fail at detecting and tracking objects in this data set(Table 3).
Most of the statistics of the objects in texVMDS are identical to those of VMDS (shape,size, velocity, number of objects); the main difference is the visual complexity of the object
and background texture. We therefore tune the dimensionality of the latent space forMONet and SCALOR, to account for the higher visual complexity, while keeping allother hyperparameters identical to VMDS. OP3 already has a significantly larger latentdimensionality and uses that for synthetic as well as natural data in their publication.
Therefore we keep these hyperparameters. For full details see Appendix D.
TBA is not applicable to texVMDS due to the dynamic background which cannot be
resolved using background subtraction. All other models perform poorly. They fail to learn
the object boundaries despite the fact that there are only four different object shapes andobjects and background move independently, which makes them easy to tell apart for humans.
Moreover, they generate large amounts of false positives, often using multiple slots for the
background or an object, with the split between slots mostly following color changes (seeFig. F.8, Fig. F.14, Fig. F.18).

5. Discussion
Our experimental results provide insights into the inductive biases and failure cases ofobject-centric models that were not apparent from their original publications. Despite2

See https://eckerlab.org/code/weis2021/ for example videos.

12

Benchmarking Unsupervised Object Representations

the positive results shown in each of the papers for the evaluated methods, a controlled,
systematic analysis demonstrates that they do not convincingly succeed at tracking, which isfundamentally what object-centric video methods should enable.
TBA has a significantly lower MOTP than the other models on all datasets, suggestingthat the simple rendering-based decoder using a fixed template might be less suitable togenerate accurate segmentation masks (see also Fig. F.10 and Fig. F.9) compared to the
VAE-based approaches of ViMON, OP3 and SCALOR.
Handling occlusion of objects during the video is a key property object-centric representations should be capable of. Qualitatively and quantitatively, OP3 is more robust to
occlusion than the other models, suggesting that its dynamics network which models interactions between objects is currently most successful at modeling occluded objects. Surprisingly,
TBA and SCALOR, which explicitly encode depth, do not handle occlusion more gracefully
than ViMON, whose simpler architecture has no explicit way of dealing with depth. Moving
forward, occlusion handling is a key component that object-centric video models need tomaster. It can be addressed by either equipping the model with a potent interaction module,
that takes pairwise interaction between objects (including occlusion) into account, similar
to OP3’s dynamics model, or ensuring that the depth reasoning of the models works asintended, which may be preferable, as explained below.
All models struggle with detecting objects that have similar color as the background (for
TBA: dark objects, since background is removed and set to black in a pre-processing step).
Color is a reliable cue to identify objects in these datasets. However, the auto-encoding
objective incurs little extra loss for missing objects with similar color as the background and,thus, the models appear to not to learn to properly reconstruct them. In order to scale todata with more visual complexity, one might want to replace the pixel-wise reconstruction
with for instance a loss based in feature space in order to focus more on reconstructing
semantic content rather than high-frequency texture, as is done when using perceptual loss
functions (Gatys et al., 2015; Hou et al., 2017) or by using contrastive learning (Kipf et al.,2020). Furthermore, the models—particularly so OP3—struggle with separating objectsof similar colors from each other. This result hints at a mismatch between the intuitions
motivating these models and what the models actually learn: it should be more efficient interms of the complexity of the latent representation to decompose two objects—even of similarcolors—into two masks with simple shapes, rather than encoding the more complicatedshape of two objects simultaneously in one slot. However, since none of the models handle
occlusion with amodal segmentation masks (i. e. including the occluded portion of the object)successfully, they learn to encode overly complex (modal) mask shapes. As a consequence,
they tend to merge similarly colored objects into one slot. This result suggests that resolving
the issues surrounding the depth reasoning in combination with amodal segmentation masks
would enable much more compact latents and could also resolve the merging of similarlycolored objects.
A major difference between models is the spatial transformer based model formulationof TBA and SCALOR, compared to ViMON and OP3, which operate on image-sized masks.
The parallel processing of objects and the processing of smaller bounding boxes renders
training TBA and SCALOR to be significantly faster and more memory efficient, enablingthem to scale to a larger number of objects. On the downside, the spatial transformer
introduces its own complications. TBA depends strongly on its prior on object size and13

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

performs well only when this prior fits the data well as well as when the data contains little
variation in object sizes, as in SpMOT (Table 2). However, it is not able to handle VMDS
and its larger variation in object sizes and shapes. SCALOR performs tracking well in scenes
where objects are clearly separated, but struggles to separate objects that partially occlude
each other when entering the scene. This difficulty is caused by its discovery mechanism
which can propose at most one bounding box per grid cell, leading to a high number ofmisses on data sets which feature significant occlusion (VOR and VMDS). Unfortunately,
simply increasing the number of proposals does not provide a simple solution, as SCALOR’s
performance is sensitive to properly tweaking the number of proposals.
Choosing a class of models is therefore dependent on the data set one wants to apply
it to as well as the computational resources at one’s disposal. Data sets that feature ahigh number of objects (>10) that are well separated from each other make a method like
SCALOR, which can process objects in parallel, advisable. On data sets with a lower numberof objects per scene which feature heavy occlusion, methods like OP3 and ViMON will likely
achieve better results, but require a high computational budget for training.
In conclusion, our analysis shows that none of the models solve the basic challenges oftracking even for relatively simple synthetic data sets. Future work should focus on developing
robust mechanisms for reliably handling depth and occlusion, additionally combining thetransformer-based efficiency of TBA and SCALOR with the stable training of ViMON
and the interaction model of OP3. The key open challenges for scaling these models tonatural videos include their computational inefficiency, complex training dynamics, as well
as over-dependence on simple appearance cues like color.

Acknowledgments
This work was supported by the German Federal Ministry of Education and Research throughthe Tübingen AI Center (FKZ: 01IS18039A, 01IS18039B); Deutsche Forschungsgemeinschaft(DFG, German Research Foundation) via grant EC 479/1-1 to A.S.E; and Collaborative
Research Center (Projektnummer 276693517 – SFB 1233: Robust Vision). We thank theInternational Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting
Marissa A. Weis, Kashyap Chitta, and Yash Sharma. The authors would also like to thankEshed Ohn-Bar for helpful discussions, as well as Rishi Veerapaneni, John D. Co-Reyes, andMichael Chang particularly with regards to applying OP3 to our experimental setting.

14

Benchmarking Unsupervised Object Representations

AppendicesIn the appendix, we first discuss the metrics used (Section A) and describe the data
generation process (Section B). We then describe the methods MONet, ViMON, TBA,
IODINE, OP3 and SCALOR (Section C). Section D contains information regarding theimplementation details and training protocols. Finally, we provide additional qualitative andquantitative experimental results in Section E.

Appendix A. Evaluation Protocol Details
We quantitatively evaluate all models on three data sets using the standard CLEAR MOT
metrics (Bernardin and Stiefelhagen, 2008). Our evaluation protocol is adapted from themulti-object tracking (MOT) challenge (Milan et al., 2016), a standard computer vision
benchmark for supervised object tracking. In particular, we focus on the metrics provided
by the py-motmetrics package3 .
A.1 Mapping
In each frame, object predictions of each model in the form of binary segmentation masks
are mapped to the ground truth object segmentation masks. We require that each pixel isuniquely assigned to at most one object in the ground truth and the predictions, respectively.
Matching is based on the Intersection over Union (IoU) between the predictions and theground truth masks (Voigtlaender et al., 2019). A valid correspondence between prediction
and object has to exceed a threshold in IoU of 0.5. Predictions that are not mapped to any
ground truth mask are classified as false positives (FPs). Ground truth objects that are notmatched to any prediction are classified as misses. Following (Bernardin and Stiefelhagen,2008), ground truth objects that are mapped to two different hypothesis IDs in subsequent
frames are classified as ID switches for that frame.
A.2 MOT Metrics
MOT Accuracy (MOTA) measures the fraction of all failure cases, i.e. false positives(FPs), misses and ID switches compared to total number of objects present in all frames.
MOT Precision (MOTP) measures the total accuracy in position for matched object
hypothesis pairs, relative to total number of matches made. We use percentage Intersection
over Union (IoU) of segmentation masks as the accuracy in position for each match. Mostly
Tracked (MT) is the ratio of ground truth objects that have been tracked for at least 80%of their lifespan.(i.e. 80% of the frames in which they are visible). MT as implemented by
py-motmetrics counts trajectories of objects as correctly tracked even if ID switches occur.
We use a strictly more difficult definition of MT that counts trajectories with ID switches ascorrectly detected but not correctly tracked. Consequently, we add the Mostly Detected(MD) measure which does not penalize ID switches. Match, Miss, ID Switches (ID S.)

3https://pypi.org/project/motmetrics/

15

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

and FPs are reported as the fraction of the number of occurrences divided by the total
number of object occurrences.
PT
MOTA = 1 −t=1 Mt + FPt + IDStPT
t=1 Ot(1)

where Mt , FPt ,and IDSt are the number of misses, false positives and ID switches, respectively,
for time t, and Ot is the number of objects present in frame t. Note that MOTA can become
negative, since the number of FPs is unbounded.
PT
MOTP =

t=1PIi
i=1 dtPT

t=1 ct(2)where dit is the total accuracy in position for the ith matched object-hypothesis pair measured
in IoU between the respective segmentation masks and ct is the number of matches made inframe t.
Note that we exclude the background masks for ViMON and OP3 before evaluating
tracking based on IoU. The Video Object Room (VOR) data set can contain up to threebackground segments, namely the floor and up to two wall segments. In order to exclude
all background slots regardless of whether the model segments the background as one oras multiple masks, we remove all masks before the tracking evaluation that have an IoU of
more than 0.2 with one of the ground truth background masks; we empirically tested that
this heuristic is successful in removing background masks regardless of whether the models
segments it as one or as three separate ones.

Appendix B. Data Set Generation Details
B.1 Video Multi-dSprites (VMDS)The Multi-DSprites Video data set consists of 10-frame video sequences of 64×64 RGB images
with multiple moving sprites per video. In order to test temporal aggregation properties ofthe models, the test set contains 20 frame-long sequences. Each video contains one to four
sprites following the data set proposed in Burgess et al. (2019) that move independentlyof each other and might partially or fully occlude one another. The sprites are sampled
uniformly from the dSprites data set (Matthey et al., 2017) and colored with a random RGB
color. The background is uniformly colored with a random RGB color. Random trajectories
are sampled per object by drawing x and y coordinates from a Gaussian process with squared
exponential covariance kernel cov[xs , xt ] = exp[−(xs − xt )2 /(2τ 2 )] and time constant τ = 10
frames, and then shifted by an initial (x, y)-position of the sprite centroid, which is uniformly
sampled from [10, 54] to ensure that the object is within the image boundaries. Trajectories
that leave these boundaries are rejected. In occlusion scenarios, larger objects are always infront of smaller objects to disambiguate prediction of occlusion. The training set consists of
10,000 examples whereas the validation set as well as the test set contain 1,000 examples
each. Additionally, we generated four challenge sets and three out-of-distribution test sets for
VMDS that contain specifically challenging scenarios. Each test set consists of 1,000 videosof length 10 frames, which we describe in the following.
16

Benchmarking Unsupervised Object Representations

B.1.1 VMDS Challenge Sets
Occlusion test set. In each video, one or more objects are heavily occluded and thus often
are not visible at all for a few frames. This is ensured by sampling object trajectories thatcross path, i.e. at least in one video frame, two objects are centered on the same pixel. The
time step and spatial position of occlusion is sampled randomly. Object trajectories are
sampled independently as described above and then shifted such that they are at the sampled
position of occlusion at time t. Videos contain two to four sprites (Fig. E.3), since at least
two objects are necessary for occlusion.
Small Objects. Videos contain one to four sprites with all sprites being of the smallest
size present in the original dSprites (Matthey et al., 2017) data set (Fig. E.3). Other thanthat, it follows the generation process of the regular training and test set.
Large Objects. Videos contain one to four sprites with all sprites being of the largest
size present in the original dSprites (Matthey et al., 2017) data set (Fig. E.3). Other thanthat, it follows the generation process of the regular training and test set.
Same Color. Videos contain two to four sprites which are identically colored with arandomly chosen color. Other than that, it follows the generation process of the regulartraining and test set (Fig. E.3).
B.1.2 VMDS Out-of-Distribution Test Sets
Rotation test set. Sprites rotate around their centroid while moving. The amount ofrotation between two video frames is uniformly sampled between 5 and 40 degrees, and isconstant for each object over the course of the video. Direction of rotation is chosen randomly.
Rotation is not included as a transformation in the training set (Fig. E.4).
Color change test set. Sprites change their color gradually during the course of thevideo. The initial hue of the color is chosen randomly as well as the direction and amountof change between two frames, which stays the same for each object over the course of thevideo. Saturation and value of the color are kept constant. Color changes are not part of the
training set (Fig. E.4).
Size change test set. Sprites change their size gradually during the course of the video.
The original dSprites data set (Matthey et al., 2017) contains six different sizes per object.
For each object, its size is sampled as either the smallest or largest in the first frame as well
as a random point in time, at which it starts changing its size. At this point in time, it will
either become larger or smaller, respectively, increasing or decreasing each frame to the next
larger or smaller size present in the original dSprites data set, until the largest or smallest
size is reached. Size changes are not part of the training set (Fig. E.4).
B.1.3 texVMDS
The Textured Multi-DSprites Video data set consists of 10-frame video sequences of 64×64
RGB images with 1–4 moving sprites per video. The background is a linearly moving cropfrom a uniformly sampled image (Deng et al., 2009) from the ImageNet Large Scale Visual
Recognition Challenge dataset (ILSVRC2012) and the texture of each sprite is similarly
cropped from a uniformly sampled ILSVRC2012 image (Deng et al., 2009). The directionof the moving background is uniformly sampled from one of 8 directions (up, down, left,
17

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

right, upper-right, upper-left, lower-right, lower-left). Other than that, texVMDS follows thegeneration process of the regular VDMS.
B.2 Sprites-MOT (SpMOT)
Sprites-MOT, originally introduced by He et al. (2019), consists of video sequences of length
20 frames. Each frame is a 128×128 RGB image. It features multiple sprites moving linearly
on a black background. The sprite can have one of four shapes and one of six colors. For
more information, refer to the original paper (He et al., 2019). We generate a training set
consisting of 9600 examples, validation set of 384 samples and test set of 1,000 examples
using the author-provided public codebase4 . However, instead of using the default setting of
20 frames per sequence, we instead generate sequences of length 10, in order to facilitatecomparison to the other data sets in our study which have only 10 frames per sequence.
Frames are downsampled to a resolution of 64×64 for training ViMON, OP3 and
SCALOR.
B.3 Video Objects Room (VOR)
We generate a video data set based on the static 3D Objects Room data set (Burgess et al.,2019), with sequences of length 10 frames each at a resolution of 128×128. This data set
is rendered with OpenGL using the gym-miniworld5 reinforcement learning environment.
It features a 3D room with up to four static objects placed in one quadrant of the room,
and a camera initialized at the diagonally opposite quadrant. The objects are either static
cubes or spheres, assigned one of 6 colors and a random orientation on the ground planeof the room. The camera then follows one of five trajectories moving towards the objects,
consisting of a small fixed distance translation and optional small fixed angle of rotation eachtime step. The wall colors and room lighting are randomized, but held constant throughout
a sequence. The training set consists of 10,000 sequences whereas the validation set and thetest set contain 1,000 sequences each.
Frames are downsampled to a resolution of 64×64 for training ViMON, OP3 and
SCALOR.

Appendix C. Methods
In this section we describe the various methods in a common mathematical framework. For
details about implementation and training, please refer to Section D.
C.1 MONet
Multi-Object-Network (MONet) (Burgess et al., 2019) is an object-centric representation
model designed for static images. It consists of a recurrent attention network that sequentiallyextracts attention masks of individual objects and a variational autoencoder (VAE) (Kingma
and Welling, 2014) that reconstructs the image region given by the attention mask in eachprocessing step.
45

https://github.com/zhen-he/tracking-by-animation
https://github.com/maximecb/gym-miniworld18

Benchmarking Unsupervised Object Representations

Attention Network: The attention network is a U-Net (Ronneberger et al., 2015) parameterized by ψ. At each processing step k, the attention network receives the full imagex ∈ [0, 1]H×W ×3 as input together with the scope variable sk ∈ [0, 1]H×W . The scope sk keeps
track of the regions of the image that haven’t been attended to in the previous processing
steps and thus remain to be explained. The attention network outputs a soft attention mask
mk ∈ [0, 1]H×W and the updated scope with the current mask subtracted:
mk = sk−1 αψ (x, sk−1 ),(3)

sk+1 = sk (1 − αψ (x, sk )),(4)

where αψ (x, sk ) ∈ [0, 1]H×W is the output of the U-net and s0 = 1. The attention maskforPKthe last slot is given by mK = sK−1 to ensure that the image is fully explained, i.e.
k=1 mk = 1.
VAE: The VAE consists of an encoder g : [0, 1]H×W ×3 × [0, 1]H×W → RL×2 and a decoderh : RL → [0, 1]H×W ×3 × [0, 1]H×W which are two neural networks parameterized by φ and θ,
respectively. The VAE encoder receives as input the full image x and the attention mask mk
and computes (µk , log σ k ), which parameterize the Gaussian latent posterior distribution
qφ (zk |x, mk ) = N (µk , σ k I). Using the reparametrization trick (Kingma and Welling, 2014),
zk ∈ RL is sampled from the latent posterior distribution. zk is decoded by the VAE decoderbk ∈ [0, 1]H×W ×3 and mask logits, which areinto a reconstruction of the image component x
b k ∈ [0, 1]H×W via a pixelwise softmax
used to compute the reconstruction of the mask m
across slots. The reconstruction of the whole
PK image is composed by summing over the Kb = k=1 mbk x
bk .
masked reconstructions of the VAE: x
Loss: MONet is trained end-to-end with the following loss function:
L(φ; θ; ψ; x) = − logKX

mk pθ (x|zk ) + βDKL (

k=1

K
Y

qφ (zk |x, mk )kp(z))

k=1

+γ

K
X

(5)

DKL (qψ (mk |x)kpθ (mk |zk )),k=1

where pθ (x|zk ) is the Gaussian likelihood of the VAE decoder and zk ∈ RL is the latent
representation of slot k.
The first two loss terms are derived from the standard VAE objective, the Evidence Lower
BOund (ELBO) (Kingma and Welling, 2014), i.e. the negative log-likelihood of the decoder
and the Kullback–Leibler divergence between the unit Gaussian prior p(z) = N (0, I) andthe latent posterior distribution qφ (zk |x, mk ) factorized across slots. Notably, the decoder
log-likelihood term pθ (x|zk ) constrains only the reconstruction within the mask, since it isweighted by the mask mk . Additionally, as a third term, the Kullback–Leibler divergenceb k |zk ) isof the attention mask distribution qψ (mk |x) with the VAE mask distribution pθ (mminimized, to encourage the VAE to learn a good reconstruction of the masks.
C.2 Video MONet
We propose an extension of MONet (Burgess et al., 2019), called Video MONet (ViMON),
which accumulates evidence over time about the objects in the scene (Fig. C.1).
19

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

ViMON processes a video recurrently by reconstructing one frame at a time and predicting
the next frame of the video. The processing of each frame follows a logic similar to MONetwith some notable differences. In the following, we use t to indicate the time step in thevideo and k to indicate the processing step within one video frame.
Attention Network: The attention network of ViMON outputs
an attention mask mt,k ∈ [0, 1]H×W in each step k conditionedon the full frame xt ∈ [0, 1]H×W ×3 , the scope st,k ∈ [0, 1]H×W
b t,k ∈ [0, 1]H×W that was predicted
and additionally the mask mby the VAE in the previous time step, in order to provide it withinformation about which object it should attend to in this specificslot k:(6)

b t,k ).
mt,k = st,k−1 αψ (xt , st,k−1 , m

xt

xt+1

^

^

K slots sequential

VAE
Dec.

zt,k
ht-1,k

GRU

ht,k

VAE
Encodermt,k

VAE: The VAE of ViMON consists of an encoder g(xt , mt,k ; φ)and a decoder h(zt,k ; θ). In contrast to MONet, the encoder in ^
Attn.
mt,k
Net
ViMON is followed by a gated recurrent unit (GRU) (Cho et al.,2014) with a separate hidden state ht,k per slot k. Thus, the GRU
aggregates information over time for each object separately. The
GRU outputs (µt,k , log σ t,k ) which parameterize the Gaussian latentxt
posterior distribution qφ (zt,k |xt , mt,k ) where zt,k ∈ RL is the latent
Figure C.1: ViMON.
representation for slot k at time t:
Attention network followed by VAE encoder
and GRU computes laz0t,k = g(xt , mt,k ; φ),(7) tent zt,k .
(µt,k , log σ t,k ), ht,k = f (GRU (z0t,k , ht−1,k ))),
(8)
qφ (zt,k |xt , mt,k ) = N (µt,k , σ t,k I)∀t, k,(9)

where g is the VAE encoder and f is a linear layer. The latent representation zt,k issampled from the latent posterior distribution using the reparametrization trick (Kingma
and Welling, 2014). Subsequently, zt,k is linearly transformed into b
zt+1,k via a learnedtransformation A ∈ RL×L : b
zt+1,k = Azt,k with b
zt+1,k being the predicted latent code forthe next time step t + 1. Both zt,k and b
zt+1,k are decoded by the shared VAE decoder
bt,k ∈ [0, 1]H×W ×3 and a reconstruction of the maskhθ into a reconstruction of the image x
H×W
b t,k ∈ [0, 1]
bt+1,k and mb t+1,k , respectively.
m
as well as x
Loss: ViMON is trained in an unsupervised fashion with the following objective adaptedfrom the MONet loss (Eq. (5)) for videos. To encourage the model to learn about object
motion, we include a prediction objective in the form of a second decoder likelihood on thenext-step prediction pθ (xt+1 |b
zt+1,k ) and an additional mask loss term, which encourages
b t+1,k |b
the predicted VAE mask distribution pθ (mzt+1,k ) to be close to the attention mask
distribution qψ (mt+1,k |xt+1 ) of the next time step for each slot k:
20

Benchmarking Unsupervised Object Representations

L(φ; θ; ψ; x) =T
X

LnegLL + βLprior + γLmask

t=1

LnegLL = −(logLprior = DKL (KX

mt,k pθ (xt |zt,k ) + log

k=1
K
Y

K
X

mt+1,k pθ (xt+1 |b
zt+1,k ))k=1

qφ (zt,k |xt , mt,k )kp(z))

k=1

Lmask =KX

DKL (qψ (mt,k |xt )kpθ (mt,k |zt,k )) + DKL (qψ (mt+1,k |xt+1 )kpθ (mt+1,k |b
zt+1,k ))k=1

C.3 Tracking-by-Animation
Tracking-by-Animation (TBA) (He et al., 2019) is a spatialtransformer-based attention model which uses a simple 2D rendering pipeline as the decoder. Objects are assigned tracking
templates and pose parameters by a tracker array, such thatthey can be reconstructed in parallel using a renderer based onaffine spatial transformation (Fig. C.2). In contrast to ViMON,
TBA uses explicit parameters to encode the position, size, aspect ratio and occlusion properties for each slot. Importantly,
TBA is designed for scenes with static backgrounds, and preprocesses sequences using background subtraction (Bloisi andIocchi, 2012) before they are input to the tracker array.
Tracker Array: TBA uses a tracker array to output a latent
representation zt ∈ RL×K at time t using a feature extractor
f (xt ; ψ) and a recurrent ’state update’, where ct ∈ RM ×N ×C
is a convolutional feature representation. The convolutional ht-1,kfeature and latent representation have far fewer elements thanxt , acting as a bottleneck:xt^

K slots parallel

SpatialTransformer(x,y)... yt,k
MLP

zt,k
Tracker
Array

ht,k

ct
FeatureExtractor(10)

ct = f (xt ; ψ),(11)ht,k = RAT (ht−1,k , ct ; π),(12)zt = g(ht ; φ).

Though the state update could be implemented as any
generic recurrent neural network block, such as an LSTM
(Hochreiter and Schmidhuber, 1997) or GRU (Cho et al., 2014),
TBA introduces a Reprioritized Attentive Tracking (RAT)
block that uses attention to achieve explicit association of slots
with similar features over time. Firstly, the previous tracker
state ht−1,k is used to generate key variables kt,k and βt,k :21

xt
Figure C.2: TBA. Feature extractor CNN f andtracker array g to get latent zt,k .
MLP h outputs mid-level representation yt,k , and Spatial Transformer renders reconstruction.

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker{kt,k , βbt,k } = Tht−1,k ,βt,k = 1 + ln(1 + exp(βbt,k )),

(13)
(14)

where T is a learned linear transformation, kt,k ∈ RS is the addressing key, and βbt,k ∈ R
is an un-normalized version of a key strength variable βt,k ∈ (1, +∞). This key strength acts
like a temperature parameter to modulate the feature re-weighting, which is described in
the following. Each feature vector in ct , denoted by ct,m,n ∈ RS , where m ∈ {1, 2, . . . , M }and n ∈ {1, 2, . . . , N } are the convolutional feature dimensions, is first used to get attention
weights:

Wt,k,m,n = P

exp(βt,k Sim(kt,k , ct,m,n )).
m0 ,n0 exp(βt,k Sim(kt,k , ct,m0 ,n0 ))(15)Here, Sim is the cosine similarity defined as Sim(p, q) = pqTP/(kpkkqk), and Wt,k,m,n isM×N
an element of the attention weight Wt,k ∈ [0, 1], satisfying m,n Wt,k,m,n = 1. Next, aread operation is defined as a weighted combination of all feature vectors of ct :X
Wt,k,m,n ct,m,n ,(16)
rt,k =
m,n

where rt,k ∈ RS is the read vector, representing the associated input feature for slot k.
Intuitively, for slots in which objects are present in the previous frame, the model can
suppress the features in rt,k that are not similar to the features of that object, helping achieve
better object-slot consistency. On the other hand, if there are slots which so far do notcontain any object, the key strength parameter allows rt,k to remain similar to ct , facilitating
the discovery of new objects.
The tracker state ht,k of the RAT block is updated with an RNN parameterized by π,taking rt,k instead of ct as its input feature:ht,k = RN N (ht−1,k , rt,k ; π).

(17)

The RAT block additionally allows for sequential prioritization of trackers, which inturn allows only a subset of trackers to update their state at a given time step, improving
efficiency. For full details on the reprioritization and adaptive computation time elements of
the RAT block, please refer to the original paper (He et al., 2019).
Mid-Level Representation: The key feature of TBA is that each latent vector zt,k is furc , yl , yp , Y s , Y a } corresponding
ther decoded into a mid-level representation yt,k = {yt,k
t,k
t,k
t,k
t,k
to interpretable, explicit object properties, via a fully-connected neural network h(zt,k ; θ) as
follows:
yt,k = h(zt,k ; θ).

(18)hθ is shared by all slots, improving parameter efficiency. The different components of the
mid-level representation are:22

Benchmarking Unsupervised Object Representationsc ∈ [0, 1]: Probability of existence of an object in that slot.
• Confidence yt,kl ∈ {0, 1}O : One-hot encoding of the discretized pseudo-depth of the object• Layer yt,krelative to other objects in the frame. Each image is considered to be composed of O
object layers, where higher layer objects occlude lower layer objects and the backgroundl = [0, 0, 1, 0] denotes the third layer.
is the zeroth (lowest) layer. E.g., when O = 4, yt,kFor simplicity and without loss of generality, we can also denote the same layer withl = 3.
its integer representation yt,kp
= [bsxt,k , sbyt,k , b
txt,k , btyt,k ] ∈ [−1, 1]4 : Normalized object pose for calculating the scale• Pose yt,kbx H by[sxt,k , syt,k ] = [1 + η x sbxt,k , 1 + η y sbyt,k ] and the translation [txt,k , tyt,k ] = [ W
2 tt,k , 2 tt,k ], whereη x , η y > 0 are constants.
s ∈ {0, 1}U ×V and Appearance Y a ∈ [0, 1]U ×V ×3 : Object template, with• Shape Yt,kt,khyperparameters U and V typically set much smaller than the image dimensions H
and W . Note that the shape is discrete (for details, see below) whereas the appearance
is continuous.
c and Y a are generated by the sigmoid function, yp is
In the output layer of hθ , yt,k
t,k
t,k
l
s are sampled from the Categorical andgenerated by the tanh function, and yt,k as well as Yt,kBernoulli distributions, respectively. As sampling is non-differentiable, the Straight-ThroughGumbel-Softmax estimator (Jang et al., 2017) is used to reparameterize both distributions
so that backpropagation can still be applied.
s and Y a
Renderer: To obtain a frame reconstruction, the renderer scales and shifts Yt,kt,kpaccording to yt,k via a Spatial Transformer Network (STN) (Jaderberg et al., 2015):ps
mt,k = ST N (Yt,k, yt,k
),

(19)

p
a
bt,k = ST N (Yt,kx, yt,k).

(20)

bt,k ∈ [0, 1]D×3 are the spatially transformed shape and appearance
where mt,k ∈ {0, 1}D and x
b t,k , an occlusion check is performed byrespectively. To obtain the final object masks mc m , then removing the elements of mb t,k = yt,kb t,k for which there exists an
initializing mt,kl > yl :object in a higher layer. That is, for k = 1, 2, . . . , K and ∀j 6= k where yt,jt,kb t,k = (1 − mt,j )m(21)

b t,k .
m

In practice, the occlusion check is sped up by creating intermediate ‘layer masks’, partiallyparallelizing the operation. Please see the original paper for more details
et al., 2019).
P(HeK
bt = k=1 m
b t,k xbt,k .
The final reconstruction is obtained by summing over the K slots, x
Loss: Learning is driven by a pixel-level reconstruction objective, defined as:

L(φ; ψ; π; θ; x) =

T
X
t=1

K

1 X x yM SE(bxt , xt ) + λ ·st,k st,k
K

!
,

(22)

k=1

where M SE refers to the mean squared error and the second term penalizes large scales[sxt,k , syt,k ] in order to make object bounding boxes more compact.
23

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

C.4 IODINE
The Iterative Object Decomposition Inference NEtwork (IODINE) (Greff et al., 2019),
similar to MONet (Burgess et al., 2019),
learns to decompose a static scene into a
multi-slot representation, in which each slot
represents an object in the scene and theslots share the underlying format of the independent representations. In contrast to
MONet, it does not recurrently segment
the image using spatial attention, rather it
starts from an initial guess of the segmentation of the whole image and iteratively
refines it. Thus, the inference componentof both models differ, while the generative
component is the same.

xt^

M refinements
sequential
K slots parallel

VAE
Dec.

zt,k
(μt,m-1,k , σt,m-1,k)

ht-1,k

+

Dyn.
Net

LSTM

ht,k

(μt+1,1,k , σt+1,1,k)

Reﬁne
Net

Iterative Inference: As with MONet,at,kIODINE models the latent posterior q(zk |x)
per slot k as a Gaussian parameterized byxt(µm,k , σ m,k ) ∈ RL×2 . To obtain latent representations for independent regions of the Figure C.3: OP3. Refinement network f folinput image, IODINE starts from initial lowed by LSTM and dynamics network d comlearned posterior parameters (µ1,k , σ 1,k ) and pute latent zt,k .
iteratively refines them using the refinement
network fφ for a fixed number of refinement steps M . fφ consists of a convolutional neuralnetwork (CNN) in combination with an LSTM cell (Hochreiter and Schmidhuber, 1997)
parameterized by φ. In each processing step, fφ receives as input the image x ∈ [0, 1]H×W ×3 ,a sample from the current posterior estimate zm,k ∈ RL and various auxiliary inputs ak , which
are listed in the original paper (Greff et al., 2019). The posterior parameters are concatenatedwith the output of the convolutional part of the refinement network and together form the
input to the refinement LSTM. The posterior parameters are additively updated in eachstep m in parallel for all K slots:(µm+1,k , σ m+1,k ) = (µm,k , σ m,k ) + fφ (zm,k , x, ak ).

(23)

Decoder: In each refinement step m, the image is represented by K latent representations
zm,k . Similar to MONet, each zm,k is independently decoded into a reconstruction of thebm,k ∈ [0, 1]H×W ×3 and mask logits m
e m,k , which are subsequently normalized byimage xapplying the softmax across slots to obtain the masks mm,k ∈ [0, 1]H×W . The reconstructionof the whole image at each refinement
Pstep m is composed by summing over the K masked
b= K
bm,k .
reconstructions of the decoder: xx
k=1 mm,k24

Benchmarking Unsupervised Object Representations

Training: IODINE is trained by minimizing the following loss function that consistsof the Evidence Lower BOund (ELBO) (Kingma and Welling, 2014) unrolled through N
iterations:L(θ, φ, (µ1,k , σ 1,k ); x) =

M
X
m
m=1

M

"
− log

K
X

mm,k pθ (x|zm,k ) + DKL

k=1

K
Y

!#
qφ (zm,k |x)kp(z)

k=1

,

(24)

where pθ (x|zm,k ) is the decoder log-likelihood weighted by the mask mk and DKL is the
Kullback-Leibler divergence between the unit Gaussian prior p(z) = N (0, I) and the latentposterior distribution q(zm,k |x) factorized across slots.
C.5 Object-centric Perception, Prediction, and Planning (OP3)
Object-centric Perception, Prediction, and Planning (OP3) (Veerapaneni et al., 2020) extends
IODINE to work on videos and in a reinforcement learning (RL) setting. It uses the above
described IODINE as an observation model to decompose visual observations into objects
and represent them independently. These representations are subsequently processed by adynamics model that models the individual dynamics of the objects, the pairwise interaction
between the objects, as well as the action’s effect on the object’s dynamics, predicting thenext frame in latent space (Fig. C.3). By modeling the action’s influence on individual
objects, OP3 can be applied to RL tasks.
OP3 performs M refinement steps after each dynamics step.
Refinement network: The refinement steps proceed as in the description for IODINE
in Section C.4. The input image xt ∈ [0, 1]H×W ×3 , which is the frame from a video
at time t, is processed by the refinement network fφ conditioned on a sample from thecurrent posterior estimate zt,m,k ∈ RL . The refinement network outputs an update of theposterior parameters (µt,m,k , σ t,m,k ) (see Eq. (23)). The posterior parameters (µ1,1,k , σ1,1,k )are randomly initialized.
Dynamics model: After refinement, samples from the current posterior estimate zt,M,k
for each slot k are used as input to the dynamics network. The dynamics model dψ consistsof a series of linear layers and nonlinearities parameterized by ψ. It models the individual
dynamics of the objects per slot k, the pairwise interaction between all combinations ofobjects, aggregating them into a prediction of the posterior parameters for the next time stept + 1 for each object k. The full dynamics model additionally contains an action component
that models the influence of a given action on each object, which we do not use in our
tracking setting. The predicted posterior parameters are then used in the next time step as
initial parameters for the refinement network.

(µt,1,k , σ t,1,k ) = dψ (zt−1,M,k , zt−1,M,[6=k] ).
25

(25)

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

Training. OP3 is trained end-to-end with the ELBO used at every refinement anddynamics step, with the loss L(θ, φ; x) given by:T
M +1X
1 X min(m, M )
t=1

T

m=1

M

− log

K
X

mt,m,k pθ (xt |zt,m,k ) + DKL (

k=1

K
Y

!
qφ (zt,m,k |xt )kq(zt,1,k |xt )) ,k=1

where for time step 1, q(z1,1,k |x1 ) = N (0, I).

(26)

C.6 SCALable Object-oriented Representation (SCALOR)SCALable Object-oriented Representation (SCALOR) (Jiang et al., 2020) is a spatialtransformer-based model that extends SQAIR (Kosiorek et al., 2018) to scale to clutteredscenes. Similar to TBA, it factors the latent representations in pose, depth and appearance
per object and uses spatial transformers (Jaderberg et al., 2015) to render objects in parallel.
In contrast to TBA, it can handle dynamic backgrounds by integrating a background RNN
that models background transitions.
Proposal-Rejection Module: SCALOR uses a proposal-rejection module g to discover
new objects. All frames up to the current time step x1:t are first encoded using a convolutional
LSTM f . The resulting features are then aggregated with an encoding of propagated objectmasks and divided into H × W grid cells.
cimg= f (x1:t ; ψ),
t

(27)

cmask
= M askEncoder(MtP ),tcagg= Concat([cimg, cmask
].
t
t
t

(28)
(29)

Per grid cell a latent variable zt,h,w is proposed. Proposal generation is done in parallel.
pose
depth what
Each zt,h,w consists of existence, pose, depth and appearance parameters (zpres
t,h,w , zt,h,w , zt,h,w , zt,h,w ).
agg
zpres
t,h,w ∼ Bern(·|g1 (ct )),

(30)

agg
zdeptht,h,w ∼ N (·|g2 (ct )),

(31)

agg
zpose
t,h,w ∼ N (·|g3 (ct )),(32)

where g1 , g2 and g3 are convolutional layers.
The appearance parameters zwhat
t,h,w are obtained by first taking a glimpse from frame xt
poseof the area specified by zt,h,w via a Spatial Transformer Network (STN) (Jaderberg et al.,2015) and subsequently extracting features from it via a convolutional neural network:
pose
catt
t,h,w = ST N (xt , zt,h,w ),
att
zwhat
t,h,w ∼ N (·|GlimpseEnc(ct,h,w )),pose
ot,h,w , mt,h,w = ST N −1 (GlimpseDec(zwhat
t,h,w ), zt,h,w ),where ot,h,w is the object RGB glimpse and mt,h,w is the object mask glimpse.
26

(33)
(34)
(35)

Benchmarking Unsupervised Object Representations

In the rejection phase, objects that overlap morethan a threshold τ in pixel space with a propagatedobject from the previous time step are rejected.
Propagation Module: During propagation, foreach object k from the previous time step t − 1 afeature attention map at,k from the encoded framefeatures cimg
is extracted centered on the positiontof the object in the previous time step and used to
update the hidden state ht,k of the tracker RNN forobject k.
at,k = att(ST N (cimg, zposett−1,k ),

(36)

ht,k = GRU ([at,k , zt−1,k ], ht−1,k ),

(37)

xt

^

K slots parallel ^xfgt,kxt

BGNetworkzt,k(x,y)...
MLP

Rejection

zt,h,w
Proposal(38) ht-1,kzt,k = update(at,k , ht,k , zt−1,k ),^bg

SpatialTransformer

Tracker
RNNs

where ST N is a spatial transformer module (JaderFeature
Extractor
berg et al., 2015). If zpres
t,k = 1 the latent representation zt,k of the respective object k will be propagatedto the next time step.
Background: The background of each frame xt
x1:t
is encoded using a convolutional neural network conditioned on the masks Mt of the objects present at Figure C.4: SCALOR Feature extime step t and decoded using a convolutional neural tractor CNN f followed by tracker
network.
RNNs or proposal-rejection module
to compute latent zt,k . Spatial Transformer in addition to background(µbg , σ bg ) = BgEncoder(xt , (1 − Mt )),(39)
module renders reconstruction.
bg
bg
zbg
(40)
t ∼ N (µ , σ ),bg
bbg
xt = BgDecoder(zt ).

(41)

bt foreground object appearances andRendering: To obtain frame reconstructions x
masks are scaled and shifted using via a Spatial Transformer Network (STN):
bft,kg = ST N −1 (ot,k , zposext,k ),

(42)

depth
pose
γ t,k = ST N −1 (mt,k · zprest,k σ(−zt,k ), zt,k ),X fg
bft g =
bt,k γ t,k .
x
x

(43)
(44)

K

Subsequently, foreground objects and background reconstruction are combined as follows
to obtain the final reconstruction:
bt = xbft g + (1 − Mt )
x
27

bbg
x
t .

(45)

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

Training: SCALOR is trained on frame reconstruction using the evidence lower bound(ELBO):T
X− log pθ (xt |zt ) + DKL (qφ (zt |z<t , x≤t )kq(zt |z<t )).

(46)

t=1

Appendix D. Model Implementation Details
D.1 Baseline: k-Means
As a baseline, we implement a simple color segmentation algorithm to evaluate if this issufficient to solve the benchmark data sets. To this end, we cluster the videos using the
k-Means algorithm as implemented by sklearn (Pedregosa et al., 2011). The number ofclusters are chosen equal to the number of slots in the other models, i.e. K = 5 for SpMOT,
K = 6 for VMDS and K = 8 for VOR. We run the clustering three times per video withdifferent random seeds. Each pixel is treated as one sample with 3 features (RGB values).
k-Means clustering is performed for each frame in the video separately. To obtain the tracking
performance, we match objects in consecutive frames using IoU of the extracted segmentation
masks with the Hungarian matching algorithm (Kuhn, 1955).
D.2 MONet and Video MONet
VAE: Following Burgess et al. (2019), the VAE encoder is a CNN with 3x3 kernels, stride 2,
and ReLU activations (Table D.1). It receives the input image and mask from the attention
network as input and outputs (µ, log σ) of a 16-dimensional Gaussian latent posterior. The
GRU has 128 latent dimensions and one hidden state per slot followed by a linear layer with
32 output dimensions. The VAE decoder is a Broadcast decoder as published by Watters
et al. (2019b) with no padding, 3x3 kernels, stride 1 and ReLU activations (Table D.2). The
output distribution is an independent pixel-wise Gaussian with a fixed scale of σ = 0.09 forthe background slot and σ = 0.11 for the foreground slots.
Attention Network: The attention network is a U-Net (Ronneberger et al., 2015) andfollows the architecture proposed by Burgess et al. (2019). The down- and up-sampling
components consist each of five blocks with 3x3 kernels, 32 channels, instance normalization,
ReLU activations and down- or up-sampling by a factor of two. The convolutional layers arebias-free and use stride 1 and padding 1. A three-layer MLP with hidden layers of size 128
connect the down- and the up-sampling part of the U-Net.
Training: MONet and ViMON are implemented in PyTorch (Paszke et al., 2019) andtrained with the Adam optimizer (Kingma and Ba, 2015) with a batch size of 64 for MONetand 32 for ViMON, using an initial learning rate of 0.0001. Reconstruction performanceis evaluated after each epoch on the validation set and the learning rate is decreased by afactor of 3 after the validation loss has not improved in 25 consecutive epochs for MONetand 100 epochs for ViMON, respectively. MONet and ViMON are trained for 600 and1000 epochs, respectively. The checkpoint with the lowest reconstruction error is selected
for the final MOT evaluation. MONet is trained with β = 0.5 and γ = 1 and ViMON istrained with β = 1 and γ = 2. K = 5 for SpMOT, K = 6 for VMDS and K = 8 for VOR.
Due to the increased slot number for VOR, batch size for ViMON had to be decreased to28

Benchmarking Unsupervised Object Representations

Type

Size/Ch.

Input
Conv 3x3
Conv 3x3
Conv 3x3
Conv 3x3
MLP
MLP

4
32
32
64
64
256
32

Act. Func.
ReLU
ReLU
ReLU
ReLU
ReLU
LinearComment
RGB + Mask

Table D.1: Architecture of ViMON VAE Encoder.
Type

Size/Ch.

Input
Broadcast
Conv 3x3
Conv 3x3
Conv 3x3
Conv 3x3
Conv 1x1

16
18
32
32
32
32
4

Act. Func.

ReLU
ReLU
ReLU
ReLU
LinearComment+ coordinates

RGB + Mask

Table D.2: Architecture of ViMON VAE Decoder.
24 to fit into the GPU memory. Respectively, the initial learning rate is set to 0.000075 for
ViMON on VOR. We initialize the attention network and the VAE in ViMON with the
pre-trained weights from MONet to facilitate learning and speed up the training. Note thatb from the VAE were used.
for all evaluations, the reconstructed masks mSprites-MOT Initialization: When training MONet and Video MONet on SpritesMOT from scratch, MONet struggles to learn the extreme color values of the objects that
Sprites-MOT features. Instead it completely focuses on learning the shapes. To circumvent
that, we initialized the weights of the models with MONet weights that were trained for
100 epochs on Multi-dSprites.
Hyperparameter tuning: Hyperparameters for MONet and ViMON were selected using
the best MSE on the validation set of VMDS. Hyperparameter tuning for MONet included
β from {0.5, 1.0} and γ from {0.5, 1.0, 2.0} and initial learning rate from {1e-05, 1e-04}.
Hyperparameter tuning for ViMON included β from {0.5, 1.0} and γ from {1.0, 2.0, 5.0,
10.0, 50.0} and initial learning rate from {1e-05, 1e-04, 1e-03}.
Hyperparameters for texVMDS were kept fixed to the best selected hyperparametersfor VMDS, except for the VAE latent dimension, which is tuned to account for the higher
visual complexity. The VAE latent dimension 16 is selected from {16, 32, 64, 128} as the one
with the best MOTA for MONet on texVMDS validation set — analogous to the SCALOR
hyperparameter tuning paradigm — and subsequently also used for training of ViMON on
texVMDS.
29

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

D.3 Tracking by Animation
Preprocessing: TBA expects its input frames to contain only foreground objects. In He
et al. (2019), the authors use Independent Multimodal Background Subtraction (IMBS)(Bloisi and Iocchi, 2012) to remove the background from data sets consisting of natural videos
with static backgrounds. Background subtraction algorithms maintain a spatio-temporalwindow around each pixel in the sequence, and remove the dominant mode based on ahistogram of color values. Since the default implementation of IMBS has several hand-tuned
thresholds corresponding to natural videos (e.g., for shadow suppression), it cannot be directly
applied to synthetic data sets like VMDS without significant hyperparameter tuning. We
instead re-generate all of the VMDS data sets with identical objects and motion but a black
background for our experiments with TBA, to mimic a well-tuned background subtraction
algorithm.
Architecture: For SpMOT, we follow the same architecture as in (He et al., 2019),
while we increase the number of slots from K = 4 to K = 5 and number of layers from O = 3
to O = 4 for VMDS. Since TBA does not model the background, this makes the number offoreground slots equal to the other models in our study.
Hyperparameter tuning: Further, we increase the size prior parameters U × V used
for the shape and appearance templates from 21 × 21 which is used for SpMOT, to 64 × 64
for VMDS, which we empirically found gave the best validation loss among 48 × 48, 56 × 56,
64 × 64 and 72 × 72. All other architectural choices are kept fixed for both data sets, andfollow He et al. (2019). Note that due to this, we trained the TBA models at its default
resolution of 128×128 unlike the 64×64 resolution used by MONet and OP3.
Training and Evaluation: We train for 1000 epochs using the same training schedule
as He et al. (2019). The checkpoint with the lowest validation loss is selected for the final
MOT evaluation. Further, we observed that the discrete nature of the shape code used in
TBA’s mid-level representation leads to salt-and-pepper noise in the reconstructed masks.
We therefore use a 2 × 2 minimum pooling operation on the final output masks to remove
isolated, single pixel foreground predictions and generate 64 × 64 resolution outputs, similar
to MONet and OP3 before evaluation.
Deviation of SpMOT results compared to original publication: Our results weregenerated with 100k training frames, while the original TBA paper (He et al., 2019) uses
2M training frames for the simple SpMOT task. Further, we report the mean of three
training runs, while the original paper reports one run (presumably the best). Our best run
achieves MOTA of 90.5 (Table E.1). Third, we evaluate using intersection over union (IoU)
of segmentation masks instead of bounding boxes.
D.4 OP3
Training: The OP3 loss is a weighted sum over all refinement and dynamics steps (Eq. (26)).
For our evaluation on multi-object tracking, we weight all time steps equally. In contrast
to the original training loss, in which the weight value is linearly increased indiscriminately,
thus weighting later predictions more highly, we perform the linear increase only for therefinement steps between dynamics steps, thus weighting all predictions equally.
OP3, as published by Veerapaneni et al. (2020), uses curriculum learning. For the first
100 epochs, M refinement steps are taken, followed by a single dynamics step, with a final
30

Benchmarking Unsupervised Object Representations

refinement step afterwards. Starting after 100 epochs, the number of dynamics steps isincremented by 1 every 10 epochs, until five dynamics steps are reached. Thus, only 5 framesof the sequence are used during training at maximum.
We chose to use an alternating schedule for training, where after each dynamics step,M = 2 refinement steps are taken, and this is continued for the entire sequence. Thus, theentire available sequence is used, and error is not propagated needlessly, since the model isenabled to refine previous predictions on the reconstruction before predicting again. Notethat this is the schedule OP3 uses by default at test-time, when it is used for model predictive
control. Note that we still use 4 refinement steps on the initial observation to update therandomly initialized posterior parameters, as in the released implementation. We split all
10-step sequences into 5-step sequences to avoid premature divergence.
Hyperparameter tuning: Note that OP3 by default uses a batch size of 80 with thedefault learning rate of 0.0003. As we found training OP3 to be very unstable, leading
to eventual divergence in almost all experiments that have been performed for this study,
we tested batch sizes [64, 32, 16, 8] and found reducing the batch size significantly, to 16,
improved performance. We found the default learning rate to be too large for SpMOT in
particular, as the model diverged significantly earlier than on VMDS and VOR. We tested
learning rates [0.0002, 0.0001, 0.00006, 0.00003], and found decreasing the learning rate to
0.0001 rectified the observed premature divergence.
The checkpoint prior to divergence with the lowest KL loss is selected for the final MOT
evaluation, as the KL loss enforces consistency in the latents over the sequence. Interestingly,
the checkpoint almost always corresponded to the epochs right before divergence.
D.5 SCALOR
Architecture: We follow the same architecture as in Jiang et al. (2020). We use a grid of
4 × 4 for object discovery with a maximum number of objects of 10. The standard deviationof the image distribution is set to 0.1. Size anchor and variance are set to 0.2 and 0.1,
respectively.
For SpMOT, background modeling is disabled and the dimensionality of the latent object
appearance is set to 8.
For VMDS, the dimensionality of background is set to 3 and the dimensionality of thelatent object appearance is set to 16. For object discovery, a grid of 3 × 3 cells with amaximum number of objects of 8 is used.
For VOR, the dimensionality of background is set to 8 and the dimensionality of thelatent object appearance is set to 16.
For texVMDS, the dimensionality of background is set to 32 and the dimensionality of
the latent object appearance is set to 16. For object discovery, a grid of 3 × 3 cells with amaximum number of objects of 8 is used.
Best model checkpoint according to the validation loss was chosen for MOT evaluation.
Hyperparameter tuning: For VMDS, we run hyperparameter search over number ofgrid cells {3 × 3, 4 × 4}, background dimension {1, 3, 5}, maximum number of objects {5, 8,
10} (dependent on number of grid cells), size anchor {0.2, 0.25, 0.3, 0.4}, zwhat dimensionality{8, 16, 24} and end value of τ {0.3, 0.5}.
31

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

For SpMOT, we run hyperparameter search over maximum number of objects {4, 10},
size anchor {0.1, 0.2, 0.3}, zwhat dimensionality {8, 16} and whether to model background(with background dimensionality 1) or not.
For VOR, we run hyperparameter search over size anchor {0.2, 0.3} and background
dimensionality {8, 12}.
For texVMDS, we keep hyperparameters fixed as for VMDS and only tune background
dimension {3, 8, 16, 32, 64, 128} and zwhat dimensionality {16, 32, 64, 128} to account for
the higher visual complexity of the data set.
We picked best hyperparameters according to the validation loss with the exception of thedata set texVMDS, for which the best validation loss was given by the model using a latentdimensionality of 128 for both background and object. This led to the model using only
the background slot to reconstruct the whole frame without using the object slots (object
matches ˜3%). As a sanity check, we picked the model which had the best MOTA on thevalidation set (shown in Table 3), but it also did not perform well on the more complex
texVMDS data set.
We additionally tested if down-weighing the log-likelihood during training by a factor of
10, while keeping the high latent dimensionality (128 for both zwhat and background) would
help, by decreasing the focus on reconstructing the high frequency textures, but it still reliedon the background to reconstruct the image without using object slots.
Training: We train SCALOR with a batch size of 16 for 300 epochs using a learning
rate of 0.0001 for SpMOT and VOR and for 400 epochs for VMDS. For the final MOT
evaluation, the checkpoint with the lowest loss on the validation set is chosen.

Appendix E. Additional Results
Table E.1 lists the individual results for the three training runs with different random seeds
per model and data set. The results of ViMON and SCALOR are coherent between thethree runs with different random seed, while TBA has one run on SpMOT with significantly
lower performance than the other two and shows variation in the three training runs on
VMDS. OP3 exhibits one training run on SpMOT with lower performance than the othertwo.
Fig. E.2 shows the fraction of failure cases dependent on the number of objects over all
types of failures (misses, ID. switches and FPs). Fig. E.1 shows the fraction of failure cases
dependent on the number of objects present in the video for the three different failure cases
separately; ID switches, FPs and misses. For ViMON, TBA and SCALOR, the numberof failures increase with the number of objects present regardless of the type of failure. In
contrast, OP3 shows this pattern for ID switches and misses, while it accumulates a higher
number of false positives (FPs) in videos with fewer (only one or two) objects.
Fig. E.4 shows a comparison between MONet and ViMON on VMDS. MONet correctly
finds and segments objects, but it does not assign them to consistent slots over time, while
ViMON maintains a consistent slot assignment throughout the video.
Fig. E.5 shows failures cases of OP3 on VOR.
Table E.3 and Table E.4 list the results for the four models, ViMON, TBA, OP3 and
SCALOR, on the VMDS challenge sets and out-of-distribution (o.o.d.) sets respectively.
32

Benchmarking Unsupervised Object Representations

Model

Run

MOTA ↑

MOTP ↑MD ↑MT ↑Match ↑Miss ↓

ID S. ↓

FPs ↓

MSE ↓

SpMOT
k-Means

1
2
3

36.2
36.2
36.3

77.6
77.6
77.6

76.8
76.8
76.8

76.3
76.3
76.2

80.5
80.5
80.5

19.2
19.2
19.3

0.2
0.2
0.2

44.3
44.3
44.2

-

MONet

1
2
3

70.0
69.4
71.3

90.6
90.0
88.1

92.8
92.7
91.6

49.4
48.1
53.8

74.7
74.2
77.1

4.1
4.1
4.9

21.2
21.6
18.0

4.7
4.8
5.8

10.4
13.4
15.2

ViMON

1
2
3

92.7
92.8
93.2

92.0
92.0
91.6

87.5
86.9
88.8

87.0
86.3
88.3

94.9
94.8
95.2

4.9
5.0
4.6

0.2
0.2
0.2

2.2
2.0
2.0

10.5
11.8
10.9

TBA

1
2
3

90.5
58.4
90.1

71.4
70.7
71.5

90.2
69.6
90.3

89.8
60.8
89.4

94.4
75.0
94.0

5.3
18.1
5.5

0.3
6.9
0.5

3.9
16.6
3.9

10.3
14.6
10.9

OP3

1
2
3

92.4
81.9
92.9

80.0
74.9
80.1

94.5
86.9
95.9

93.7
86.5
95.2

97.3
92.8
97.6

2.4
6.8
2.0

0.4
0.3
0.4

4.8
10.9
4.7

4.3
30.1
5.6

SCALOR

1
2
3

94.4
94.7
95.5

80.1
80.2
80.2

96.5
96.4
96.3

92.3
93.1
94.0

95.4
95.8
96.4

2.4
2.4
2.4

2.2
1.8
1.2

1.0
1.1
0.9

3.3
3.4
3.6

VOR
k-Means

1
2
3

-38.0
-38.1
-37.9

76.5
76.5
76.5

69.8
69.8
70.0

62.8
63.0
62.9

72.6
72.7
72.8

22.1
22.1
22.0

5.3
5.2
5.2

110.7
110.8
110.7

-

MONet

1
2
3

28.0
44.5
38.5

81.3
82.4
81.6

73.8
78.2
78.7

26.7
45.4
39.8

57.4
68.7
67.0

18.0
15.0
14.4

24.6
16.3
18.5

29.4
24.2
28.5

14.1
11.8
10.8

ViMON

1
2
3

89.0
89.0
89.0

88.9
89.8
89.9

90.2
89.9
91.0

89.8
89.6
90.6

92.9
93.0
93.8

6.8
6.8
6.0

0.3
0.2
0.2

3.9
4.0
4.8

7.1
6.2
5.9

OP3

1
2
3

64.8
66.2
65.3

89.5
88.1
89.3

87.2
88.6
88.2

85.1
85.1
86.1

90.3
90.7
91.1

8.8
7.9
8.0

0.9
1.4
0.9

25.5
24.5
25.8

3.1
2.9
3.0

SCALOR

1
2
3

74.1
74.6
75.1

85.8
86.0
86.1

75.6
75.9
76.5

75.5
75.9
76.4

77.4
78.1
78.2

22.6
21.9
21.7

0.0
0.1
0.0

3.3
3.5
3.1

6.4
6.4
6.3

Table E.1: Analysis of SOTA object-centric representation learning models for MOT. Results
for three runs with different random training seeds.

33

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

Model

Run

MOTA ↑

MOTP ↑MD ↑MT ↑Match ↑Miss ↓

ID S. ↓

FPs ↓

MSE ↓

VMDS
k-Means

1
2
3

-3.3
-3.3
-3.2

89.8
89.8
89.8

98.3
98.2
98.2

93.4
93.2
93.4

96.4
96.4
96.4

1.0
1.1
1.0

2.6
2.6
2.6

99.7
99.7
99.6

-

MONet

1
2
3

51.7
44.3
52.2

79.6
76.1
80.2

75.1
71.8
75.6

36.7
34.8
35.5

67.6
65.9
66.5

12.9
15.0
13.0

19.5
19.1
20.5

15.9
21.5
14.2

20.8
25.3
20.4

ViMON

1
2
3

87.0
87.1
86.5

86.8
86.8
86.7

86.7
86.1
86.0

85.4
85.1
84.6

92.4
92.3
92.1

6.8
7.1
7.2

0.7
0.6
0.7

5.5
5.3
5.6

10.6
10.8
10.6

TBA

1
2
3

68.5
38.9
56.0

76.1
73.8
75.0

69.3
55.1
64.3

65.3
50.5
59.2

80.7
70.2
76.7

16.5
26.6
19.8

2.8
3.2
3.5

12.2
31.3
20.8

26.0
30.8
27.5

OP3

1
2
3

93.1
92.7
89.4

94.2
93.4
93.3

97.2
96.9
96.2

96.7
96.3
95.8

98.0
97.8
97.6

1.9
2.0
2.2

0.2
0.2
0.2

4.9
5.1
8.3

4.0
4.3
4.6

SCALOR

1
2
3

75.7
72.7
73.7

88.1
87.2
87.6

69.4
66.7
67.5

68.3
65.6
66.2

79.8
77.6
77.9

19.4
21.6
21.2

0.8
0.8
0.9

4.0
4.9
4.2

13.9
14.2
14.0

texVMDS
k-Means

1
2
3

-99.5
-99.5
-99.5

76.4
76.4
76.4

25.4
25.3
25.3

24.3
24.4
24.3

30.3
30.4
30.3

69.2
69.2
69.2

0.5
0.5
0.5

129.8
129.9
129.8

-

MONet

1
2
3

-70.8
-68.1
-80.9

67.1
69.2
66.7

15.5
20.4
12.1

13.4
15.4
8.1

24.9
30.4
18.9

73.9
66.5
78.9

1.2
3.1
2.2

95.7
98.6
99.8

203.0
192.5
205.9

ViMON

1
2
3

-88.2
-86.6
-81.6

68.3
69.0
69.8

24.2
22.6
25.8

23.8
22.0
25.4

35.5
32.4
36.3

64.2
67.4
63.4

0.3
0.3
0.3

123.7
119.0
117.9

174.6
172.7
166.9

OP3

1
2
3

-105.6-116.0-109.7

71.5
70.1
70.3

20.6
9.4
19.5

20.4
9.2
19.0

28.3
13.6
26.9

71.6
86.3
72.8

0.2
0.1
0.3

133.9
129.6
136.6

113.6
153.2
131.8

SCALOR

1
2
3

-84.8
-99.5
-113.4

73.3
74.4
74.4

6.4
7.3
5.8

6.3
7.0
5.6

12.5
12.7
11.8

87.3
87.1
88.0

0.2
0.3
0.3

97.3
112.1
125.2

149.5
125.0
126.7

Table E.2: Analysis of SOTA object-centric representation learning models for MOT. Results
for three runs with different random training seeds.

34

Fraction of failure cases [%]Benchmarking Unsupervised Object Representations

ID Switches

80

FPs

Misses

60

ViMON
TBA
OP3
SCALOR

40
20
0

1

2

3

4

1 2 3 4
Number of objects in video

1

2

3

4

Figure E.1: Distribution of failure cases dependent on number of objects in VMDS videos
split by failure class. Mean of three training runs. Error bars: SD.
Fraction of failure cases [%]for number of objects in video
3

2

3

2
1
4

1

ViMON

TBA

2

4

3

1

2
1

3

4

4
SCALOR

OP3

Figure E.2: Distribution of failure cases dependent on number of objects in VMDS videos
for each model. Mean of three training runs.
Results are shown as the mean and standard deviation of three training runs with different
random seed per model.
E.1 Out-of-distribution test sets
To test whether the models can in principle learn additional object transformations asfeatured in the VMDS o.o.d. sets, we additionally train the models on a new training set
that includes size and color changes as well as rotation of objects.
ViMON, OP3 and SCALOR are able to learn additional property changes of the objects
when they are part of the training data while TBA fails to learn tracking on this morechallenging data set (Fig. E.3; for absolute values Table E.4).
E.2 Stability of training and runtime
To assess runtime in a fair way despite the models being trained on different hardware,
we report the training progress of all models after one hour of training on a single GPU
35

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

MOTA

Performance %100

MT
Test set
ViMON*TBA*
OP3*
SCALOR*

50
0
-50

Size

Color

Rota- Size
tion

Color

Rotation

Figure E.3: Performance on out-of-distribution sets relative to VMDS test set (100%). *indicates that models were trained on a data set that included color, size and orientation
changes of objects.

ViMON

MONet

Ground Truth
Reconstruction
Segmentation
t=3

t=5

t=7

t=9

t=3

t=5

t=7

t=9

Figure E.4: Comparison of MONet and ViMON on VMDS. Example sequence of data set
shown with corresponding outputs of the model. Reconstruction shows sum of components
from all slots, weighted by the attention masks. Color-coded segmentation maps in third row
signify slot-assignment. Note how the object-slot assignment changes for consecutive frames(3rd row) for MONet, while ViMON maintains a consistent slot assignment throughout thevideo.
(Table E.5). In addition, we quantify inference time on the full VMDS test set using a batch
size of one.
E.3 ViMON Ablations
Removing the GRU or the mask conditioning of the attention network reduces tracking
performance (MOTA on VMDS from 86.8% to 70.6% and 81.4%, respectively; Table E.6)
E.4 Impact of IoU Threshold for Matching
To examine whether the IoU threshold used for matching object predictions to ground truth
impacts the ordering of the models, we compute all metrics with regard to the thresholds[0.1 .. 0.9] in steps of 0.1 on the VMDS test set. The ordering of the models with regard36Benchmarking Unsupervised Object Representations

Occlusion

Same Color

Small Objects

Large Objects

Model

MOTA

MOTP

MT

MOTA

MOTP

MT

MOTA

MOTP

MT

MOTA

MOTP

MT

ViMON
TBA
OP3
SCALOR

67.1 ± 0.4
37.5 ± 10.4
85.3 ± 1.0
58.8 ± 1.0

82.5 ± 0.0
72.8 ± 0.8
91.6 ± 0.4
86.6 ± 0.4

63.0 ± 0.1
38.3 ± 4.6
89.6 ± 0.9
46.8 ± 1.2

72.2 ± 0.1
47.2 ± 9.4
51.5 ± 1.3
53.7 ± 1.1

83.6 ± 0.1
73.0 ± 0.7
86.5 ± 0.3
83.4 ± 0.3

70.4 ± 0.3
45.2 ± 3.9
66.3 ± 1.3
46.2 ± 1.1

86.3 ± 0.2
74.3 ± 0.7
93.3 ± 1.6
74.4 ± 0.7

83.3 ± 0.2
71.9 ± 0.4
93.0 ± 0.4
86.1 ± 0.4

83.4 ± 0.4
65.3 ± 1.6
97.0 ± 0.2
67.6 ± 1.3

70.7 ± 0.5
25.6 ± 15.0
83.8 ± 2.0
66.1 ± 1.9

85.1 ± 0.1
73.4 ± 0.9
92.2 ± 0.4
86.6 ± 0.5

76.1 ± 0.7
44.7 ± 6.7
93.5 ± 0.4
62.4 ± 1.4

Occlusion

t=1

t=4

Same Color

t=7

t=10

t=1

t=4

Small Objects

t=7

t=10

t=1

t=4

Large Objects

t=7

t=10

t=1

t=4

t=7

t=10

Table E.3: Performance on VMDS challenge sets. Results shown as mean ± standard
deviation for three runs with different random training seeds. Examples sequences for eachchallenge set shown below.
Size
Model
ViMON
ViMON*
TBA
TBA*
OP3
OP3*
SCALORSCALOR*

Color

Rotation

MOTA

MOTP

MD

MT

MOTA

MOTP

MD

MT

MOTA

MOTP

MD

MT

61.4 ± 2.5
80.3 ± 0.9
52.3 ± 8.7
1.3 ± 7.8
87.0 ± 1.9
84.0 ± 2.8
68.1 ± 1.7
67.5 ± 1.2

78.0 ± 0.3
82.1 ± 0.5
73.3 ± 0.7
68.4 ± 1.9
90.8 ± 0.4
91.2 ± 1.0
84.9 ± 0.4
85.2 ± 0.6

71.3 ± 2.1
82.5 ± 0.4
59.8 ± 4.9
30.6 ± 4.5
96.4 ± 0.1
95.9 ± 0.8
63.3 ± 1.7
61.2 ± 1.2

66.8 ± 1.9
79.8 ± 0.5
51.8 ± 4.9
24.8 ± 3.4
95.3 ± 0.1
94.5 ± 1.2
60.0 ± 2.0
57.1 ± 0.7

87.4 ± 0.4
84.5 ± 0.6
56.1 ± 11.4
-16.5 ± 8.1
90.8 ± 1.2
83.6 ± 3.7
75.5 ± 1.1
73.3 ± 0.7

86.2 ± 0.2
84.6 ± 0.5
75.1 ± 0.9
69.6 ± 1.5
93.5 ± 0.5
91.6 ± 1.3
89.9 ± 0.5
89.8 ± 0.5

86.4 ± 0.1
83.4 ± 0.5
63.7 ± 5.4
29.1 ± 3.8
97.3 ± 0.1
95.5 ± 0.5
67.0 ± 1.4
64.8 ± 1.1

85.0 ± 0.2
81.8 ± 0.3
59.0 ± 5.2
25.4 ± 3.3
95.8 ± 0.1
92.9 ± 1.6
65.7 ± 1.6
63.0 ± 0.9-10.4 ± 4.0
78.7 ± 1.6
52.4 ± 9.9-7.5 ± 7.9
54.7 ± 5.7
74.5 ± 2.2
46.5 ± 1.8
61.6 ± 1.4

70.5 ± 0.4
82.0 ± 0.6
73.6 ± 0.8
69.4 ± 1.4
84.2 ± 0.7
89.8 ± 0.7
82.1 ± 0.5
83.5 ± 0.4

39.5 ± 2.6
79.2 ± 0.4
59.3 ± 6.2
26.6 ± 4.0
87.1 ± 1.7
94.8 ± 0.6
41.9 ± 1.7
53.4 ± 1.5

29.8 ± 1.0
76.4 ± 0.6
49.8 ± 5.5
20.6 ± 3.4
80.5 ± 2.5
93.3 ± 0.8
37.1 ± 1.3
50.2 ± 1.1* Models trained on a data set that featured color, size and orientation changes of objects during the sequence.
Size
Color
Rotation

t=1

t=3

t=5

t=7

t=9

t=1

t=3

t=5

t=7

t=9

t=1

t=3

t=5

t=7

t=9

Table E.4: Performance on VMDS OOD test sets. Results shown as mean ± standard
deviation for three runs with different random training seeds. Examples sequences for eacho.o.d. set shown below.
to their MOTA is consistent over all matching thresholds, but the performance difference
between the models increases with higher thresholds (Fig. E.6). For low thresholds (0.1 –0.3) TBA outperforms SCALOR in MD, MT, Matches and Misses but has a significantly
lower segmentation performance as measured by MOTP.

37

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=3

t=5

t=7

t=9

Segm. Recon.

GroundTruth

Segm. Recon.

GroundTruth

Segm. Recon.

Ground
Truth

t=1

100

90

80

80

85

60

60

MD

80
75

40

0.3

0.5

0.7

IoU Threshold

0.9

0
0.1

0.3

0.5

0.7

IoU Threshold

0.9

40
20

20

70
0.1

MT

100

95

MOTP

100
75
50
25
0
25
50
75
100

0
0.1

0.3

0.5

0.7

IoU Threshold

0.9

0.1

6

100

80

5

80

60

60

4

60

40

0.1

0.3

0.5

0.7

IoU Threshold

0.9

0

3
2

20

20
0

40

FPs

100

80

ID S.

100

Miss

Match

MOTA

Figure E.5: Failure cases of OP3 on VOR. Example sequences of VOR test set shownwith corresponding outputs of the model after final refinement step. Binarized color-coded
segmentation maps in third row signify slot-assignment.

0
0.3

0.5

0.7

IoU Threshold

0.9

0.5

0.7

0.9

0.7

0.9

IoU Threshold

ViMON
TBA
OP3
SCALOR

40
20

1
0.1

0.3

0.1

0.3

0.5

0.7

IoU Threshold

0.9

0

0.1

0.3

0.5

IoU Threshold

Figure E.6: Results of ViMON, TBA, OP3 and SCALOR on the VMDS test set using
different IoU thresholds for matching ground truth objects to model predictions.

38Benchmarking Unsupervised Object Representations

Training

Inference

Model

Resolution

No. Param.

Batch Size

Memory [MiB]

No. Iters

EpochsMemory [MiB]

Avg. runtime / batch

Total runtime

ViMON
TBA
OP3
SCALOR

64×64
128×128
64×64
64×64

714,900
3,884,644*
876,305
2,763,526

18
64
10
48

10,86010,564
10,874
10,942

3687
4421
2204
2547

6.63
28.29
2.20
12.23

910
972
4092
930

0.28 s/it
0.24 s/it
0.54 s/it
0.29 s/it

4min 39s
4min 05s
9min 04s
4min 48s* The TBA parameter count scales with the feature resolution, which is kept fixed using adaptive pooling. This
makes the parameter count independent of input resolution.

Table E.5: Runtime analysis (using a single RTX 2080 Ti GPU). Training: models trainedon VMDS for one hour. Inference: models evaluated on VMDS test set with batch size=1(10 frames).
Model

MOTA ↑

MOTP ↑MD ↑MT ↑Match ↑Miss ↓

ID S. ↓

FPs ↓

MSE ↓

ViMON w/o Mask conditioning
ViMON w/o GRU

70.6
81.4

87.8
86.9

75.7
79.8

66.0
77.3

81.4
88.2

13.4
10.3

5.2
1.4

10.8
6.8

16.9
18.9

Table E.6: Ablation experiments for ViMON on VMDS.

Appendix F. Supplementary Figures
See figures F.5 – F.13 for additional, randomly picked examples of reconstruction andsegmentation for k-Means, ViMON, TBA, OP3 and SCALOR on the four data sets
(VMDS, SpMOT, VOR and texVMDS).

39

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.1: Results of k-Means on VMDS. Random example sequences of VMDS test
set shown with corresponding outputs of the model. Note that k-Means does not give a
reconstruction of the input.
40

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.2: Results of k-Means on SpMOT. Random example sequences of SpMOT test
set shown with corresponding outputs of the model. Note that k-Means does not give a
reconstruction of the input.
41

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.3: Results of k-Means on VOR. Random example sequences of VOR test set shownwith corresponding outputs of the model. Note that k-Means does not give a reconstructionof the input.
42

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.4: Results of k-Means on texVMDS. Random example sequences of texVMDS
test set shown with corresponding outputs of the model. Note that k-Means does not give a
reconstruction of the input.
43

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.5: Results of ViMON on VMDS. Random example sequences of VMDS test set
shown with corresponding outputs of the model. Reconstruction shows sum of components
from all slots, weighted by the reconstructed masks from the VAE. Binarized color-coded
segmentation maps in third row signify slot-assignment.
44

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.6: Results of ViMON on SpMOT. Random example sequences of SpMOT test set
shown with corresponding outputs of the model. Reconstruction shows sum of components
from all slots, weighted by the reconstructed masks from the VAE. Binarized color-coded
segmentation maps in third row signify slot-assignment.
45

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.7: Results of ViMON on VOR. Random example sequences of VOR test set
shown with corresponding outputs of the model. Reconstruction shows sum of components
from all slots, weighted by the reconstructed masks from the VAE. Binarized color-coded
segmentation maps in third row signify slot-assignment.
46

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.8: Results of ViMON on texVMDS. Random example sequences of texVMDS
test set shown with corresponding outputs of the model. Reconstruction shows sum ofcomponents from all slots, weighted by the reconstructed masks from the VAE. Binarizedcolor-coded segmentation maps in third row signify slot-assignment.
47

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.9: Results of TBA on VMDS. Random example sequences of VMDS test set
shown with corresponding outputs of the model. Binarized color-coded segmentation maps
in third row signify slot-assignment. Note that background subtraction is performed in the
preprocessing of TBA, hence the black background in the reconstructions.
48

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.10: Results of TBA on SpMOT. Random example sequences of SpMOT test set
shown with corresponding outputs of the model. Binarized color-coded segmentation maps
in third row signify slot-assignment.
49

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.11: Results of OP3 on VMDS. Random example sequences of VMDS test set shownwith corresponding outputs of the model after final refinement step. Binarized color-coded
segmentation maps in third row signify slot-assignment.
50

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.12: Results of OP3 on SpMOT. Random example sequences of SpMOT test
set shown with corresponding outputs of the model after final refinement step. Binarizedcolor-coded segmentation maps in third row signify slot-assignment.
51

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.13: Results of OP3 on VOR. Random example sequences of VOR test set shownwith corresponding outputs of the model after final refinement step. Binarized color-coded
segmentation maps in third row signify slot-assignment.
52

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.14: Results of OP3 on texVMDS. Random example sequences of texVMDS test
set shown with corresponding outputs of the model after final refinement step. Binarizedcolor-coded segmentation maps in third row signify slot-assignment.
53

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.15: Results of SCALOR on VMDS. Random example sequences of VMDS test set
shown with corresponding outputs of the model. Binarized color-coded segmentation maps
in third row signify slot-assignment.
54

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.16: Results of SCALOR on SpMOT. Random example sequences of SpMOT test
set shown with corresponding outputs of the model. Binarized color-coded segmentation
maps in third row signify slot-assignment.
55

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.17: Results of SCALOR on VOR. Random example sequences of VOR test set
shown with corresponding outputs of the model. Binarized color-coded segmentation maps
in third row signify slot-assignment.
56

Benchmarking Unsupervised Object Representations

t=2

t=3

t=4

t=5

t=6

t=7

t=8

t=9

t=10

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

GroundSegm. Recon. Truth

t=1

Figure F.18: Results of SCALOR on texVMDS. Random example sequences of texVMDS
test set shown with corresponding outputs of the model. Binarized color-coded segmentation
maps in third row signify slot-assignment.
57

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

References
Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance:
The clear mot metrics. EURASIP Journal on Image and Video Processing, 2008.
Domenico Daniele Bloisi and Luca Iocchi. Independent multimodal background subtraction.
In CompIMAGE, 2012.
Christopher P. Burgess, Loïc Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins,
Matthew Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition
and representation. arXiv.org, 1901.11390, 2019.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares,
Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on
Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734, 2014. doi:
10.3115/v1/D14-1179.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection withconvolutional neural networks. Proc. of the Conf. on Artificial Intelligence (AAAI), 2019.
Antonia Creswell, Kyriacos Nikiforou, Oriol Vinyals, Andre Saraiva, Rishabh Kabra, Loic
Matthey, Chris Burgess, Malcolm Reynolds, Richard Tanburn, Marta Garnelo, and Murray
Shanahan. Alignnet: Unsupervised entity alignment. arXiv.org, 2007.08973, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2009.
Sebastien Ehrhardt, Oliver Groth, Aron Monszpart, Martin Engelcke, Ingmar Posner, Niloy
Mitra, and Andrea Vedaldi. Relate: Physically plausible multi-object scene synthesis
using structured latent spaces. In Advances in Neural Information Processing Systems(NeurIPS), volume 33, 2020.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis:
Generative scene inference and sampling with object-centric latent representations. In
Proc. of the International Conf. on Learning Representations (ICLR), 2020.
S. M. Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Koray
Kavukcuoglu, and Geoffrey E Hinton. Attend, infer, repeat: Fast scene understanding
with generative models. In Advances in Neural Information Processing Systems (NeurIPS),
2016.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artisticstyle. arXiv.org, 1508.06576, 2015.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information
Processing Systems (NeurIPS). 2016.
58

Benchmarking Unsupervised Object Representations

Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems (NeurIPS). 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess,Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object
representation learning with iterative variational inference. In Proc. of the International
Conf. on Machine learning (ICML), 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in
artificial neural networks. arXiv.org, 2012.05208, 2020.
Zhen He, Jian Li, Daxue Liu, Hangen He, and David Barber. Tracking by animation:
Unsupervised learning of multi-object attentive trackers. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2019.
Paul Henderson and Christoph H. Lampert. Unsupervised object-centric video generation anddecomposition in 3D. In Advances in Neural Information Processing Systems (NeurIPS)
33, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8):1735–1780, 1997. ISSN 0899-7667. doi: 10.1162/neco.1997.9.8.1735.
X. Hou, L. Shen, K. Sun, and G. Qiu. Deep feature consistent variational autoencoder.
In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pages
1133–1141, 2017. doi: 10.1109/WACV.2017.131.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatialtransformer networks. In Advances in Neural Information Processing Systems (NeurIPS).
2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax.
In Proc. of the International Conf. on Learning Representations (ICLR), 2017.
Jindong Jiang and Sungjin Ahn. Generative neurosymbolic machines. In H. Larochelle,
M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information
Processing Systems (NeurIPS), 2020.
Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative
world models with scalable object representations. In Proc. of the International Conf. on
Learning Representations (ICLR), 2020.
Scott P Johnson. Object perception. In Oxford Research Encyclopedia of Psychology. Oxford
University Press, 2018.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc.
of the International Conf. on Learning Representations (ICLR), 2015.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In Proc. of theInternational Conf. on Learning Representations (ICLR), 2014.
59

Weis, Chitta, Sharma, Brendel, Bethge, Geiger and Ecker

Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured worldmodels. In Proc. of the International Conf. on Learning Representations (ICLR), 2020.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer,repeat: Generative modelling of moving objects. In Advances in Neural Information
Processing Systems (NeurIPS), 2018.
J. v. Kügelgen, I. Ustyuzhaninov, P. Gehler, M. Bethge, and B. Schölkopf. Towards causal
generative scene models via competition of experts. In ICLR 2020 Workshop "Causal
Learning for Decision Making", 2020.
H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics
Quarterly, 2(1-2):83–97, 1955.
Brenden M. Lake, Tomer D. Ullman, Joshua B. Tenenbaum, and Samuel J. Gershman.
Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017. doi: 10.1017/S0140525X16001837.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng,
Jindong Jiang, and Sungjin Ahn. Space: Unsupervised object-oriented scene representation
via spatial attention and decomposition. In Proc. of the International Conf. on Learning
Representations (ICLR), 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg
Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learningwith slot attention. In Advances in Neural Information Processing Systems (NeurIPS),
2020.
Joe Marino, Yisong Yue, and Stephan Mandt. Iterative amortized inference. In Proc. of theInternational Conf. on Machine learning (ICML), 2018.
David Marr. Vision: A computational investigation into the human representation andprocessing of visual information. W. H. Freeman, San Francisco, 1982.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017.
A. Milan, L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler. MOT16: A benchmark for
multi-object tracking. arXiv.org, 1603.00831, 2016.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models ofvisual attention. In Advances in Neural Information Processing Systems (NeurIPS), 2014.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan,
Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas
Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy,
Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style,high-performance deep learning library. In Advances in Neural Information Processing
Systems (NeurIPS), 2019.
60

Benchmarking Unsupervised Object Representations

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau,
M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python.
Journal of Machine Learning Research (JMLR), 12:2825–2830, 2011.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks forbiomedical image segmentation. In Medical Image Computing and Computer-Assisted
Intervention (MICCAI), 2015.
Elizabeth S. Spelke. Where perceiving ends and thinking begins: The apprehension of objects
in infancy. The Minnesota symposia on child psychology, 20:197–234, 1988.
Elizabeth S. Spelke and Katherine D. Kinzler. Core knowledge. Developmental Science, 10(1):89–96, 2007. doi: https://doi.org/10.1111/j.1467-7687.2007.00569.x.
Tomer D. Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B. Tenenbaum. Mind
games: Game engines as an architecture for intuitive physics. Trends in Cognitive Sciences,21(9):649–665, 2017. ISSN 1364-6613. doi: https://doi.org/10.1016/j.tics.2017.05.012.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational
neural expectation maximization: Unsupervised discovery of objects and their interactions.
In Proc. of the International Conf. on Learning Representations (ICLR), 2018.
Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun
Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-based
reinforcement learning. In Proc. Conf. on Robot Learning (CoRL), 2020.
Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana
Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-object tracking and segmentation.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
Nicholas Watters, Loic Matthey, Matko Bosnjak, Christopher P. Burgess, and Alexander
Lerchner. COBRA: Data-efficient model-based RL through unsupervised object discovery
and curiosity-driven exploration. arXiv.org, 1905.09275, 2019a.
Nicholas Watters, Loic Matthey, Christopher P. Burgess, and Alexander Lerchner. Spatial
broadcast decoder: A simple architecture for learning disentangled representations in
VAEs. arXiv.org, 1901.07017, 2019b.
Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy, William T. Freeman, Joshua B. Tenenbaum, and Jiajun Wu. Unsupervised discovery of parts, structure, and dynamics. In Proc.
of the International Conf. on Learning Representations (ICLR), 2019.
Jinyang Yuan, Bin Li, and Xiangyang Xue. Generative modeling of infinite occluded objects
for compositional scene representation. In Proc. of the International Conf. on Machinelearning (ICML), 2019.

61