Conditional Object-Centric Learning from Video
Authors: Thomas Kipf
Gamaleldin Fathy Elsayed
Aravindh Mahendran
Austin Stone
Sara Sabour
Georg Heigold
Rico Jonschkowski
Alexey Dosovitskiy
Klaus Greff
Venue: ICLR 2022 Poster (2022)

Abstract: Object-centric representations are a promising path toward more systematic generalization by providing flexible abstractions upon which compositional world models can be built. Recent work on simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision. However, such fully-unsupervised methods still fail to scale to diverse realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene. In this paper, we instead take a weakly-supervised approach and focus on how 1) using the temporal dynamics of video data in the form of optical flow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in significantly more realistic synthetic data. We introduce a sequential extension to Slot Attention which we train to predict optical flow for realistic looking synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the first frame, is sufficient to significantly improve instance segmentation. These benefits generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences. We also find that such initial-state-conditioning can be used during inference as a flexible interface to query the model for specific objects or parts of objects, which could pave the way for a range of weakly-supervised approaches and allow more effective interaction with trained models.

Published as a conference paper at ICLR 2022
CONDITIONAL OBJECT-CENTRIC LEARNING FROM VIDEO
Thomas Kipf*i, Gamaleldin F. Elsaayd'' Aravindh Mahendran', Austin Stone", Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy & Klaus Greff Google Research
ABSTRACT
Object-centric representations are promising path toward more systematic gen- eralization by providing flexible abstractions upon which compositional world models can be built. Recent work simple 2D and 3D datasets has shown that models with object-centric inductive biases can learn to segment and represent meaningful objects from the statistical structure of the data alone without the need for any supervision. However, such uully-unsueervieed methods still fail to scale to diverse realistic data, despite the use of increasingly complex inductive biases such as priors for the size of objects or the 3D geometry of the scene. In this paper, we instead take weakly- -upeevsd approach and focus on how 1) using the temporal dynamics of video data in the form of optical flow and 2) conditioning the model on simple object location cues can be used to enable segmenting and tracking objects in significantly more realistic synthetic data. We introduce a sequential extension to Slot Attention which we train to predict optical flow for realistic look- ing synthetic scenes and show that conditioning the initial state of this model on a small set of hints, such as center of mass of objects in the first frame, is sufficient significantly improve instance segmentation. These benefits generalize beyond the training distribution to novel objects, novel backgrounds, and to longer video sequences. We also find that such intitalstate-conndiiiinng can be used during inference as a flexible interface to query the model for specific objects or parts of objects, which could pave the way for range of supery-- approaches and allow more effective interaction with trained models.
Project page: http:::/sllotattennoooooooo
g withu.bii.
INTRODUCTION
Humans understand the world in terms of separate objects (Kahneman al., 1992; Spelke & Kinzler, 2007), which serve as compositional building blocks that can be processed independently and recombined. Such compositional model of the world forms the foundation for high-level cognitive abilities such as language, causal reasoning, mathematics, planning, etc. and is crucial for general- izing in predictable and systematic ways. Object-centric representations have the potential greatly improve sample efficiency, robustness, generalization to new tasks, and interpretability of machine learning algorithmss (Greff et al., 2020). In this work, we focus on the aspect of modeling motion of objects from video, because of its synergistic relationship with object-centric representations: On the one hand, objects support learning an efficient dynamics model by factorizing the scene into approx- imately independent parts with only sparse interactions. Conversely, motion provides powerful cue for which inputs should be grouped together, and is thus an important tool for learning about objects.
Unsupervised multi-object representation learning has recently made significant progress both on images (c.g. Burgess et al., 2019; Greff et al., 2019; Lin et al., 2020; Locatello et al., 2020) and on video (c.g. Veerapaneni et al., 2020; Weis et al., 2020; Jiang et al., 2020). By incorporating object- centric inductive biases, these methods Iearn to segment and represent objects from the statistical structure of the data alone without the need for supervision. Despite promising results these methods are currently limited by two important problems: Firstly, they are restricted to toy data like moving 2D sprites or very simple 3D scenes and generally fail at more realistic data with complex textures (Greff
"Equal contribution. 'Corresoondence to: k pfggooglelll e com

---

Published as a conference paper at ICLR 2022
Corrector
S
Predictor
SAVI Processor
A
Decccldl
Input Frame agg
Predicted Flow ga
Encode
Legend
Slots
Initializer
modille
Recurrent
SAVI Processor
Predetoos
Multi-Layer Perceptron
Niw
Goussian Noise
permutation equiverrent
Bounding Box Segmentation,
map activations
module
22
Predicted Flow
Figure 1: Slot Attention for Video (SAVi) architecture overview. The SAVi Processor recurrently applied to sequence of input video frames. SAVi maintains set S; [s] ........ 8 of K latent slot representations each time step f. Slots can be conditionally initialized based on cues such as center of mass coordinates of objects and subsequently learn to track and represent particular object. SAVi trained to predict optical flow or to reconstruct input frames.
al., 2019; Harley al., 2021; Karazija al., 2021). And secondly, it not entirely clear how to interface with these models both during training and inference. The notion of an object is ambiguous and desk-depen,nnt, and the segmentation learned by these models does not necessarily align with the tasks of interest. The model might, for example, over-segment desired object into distinct parts, or alternatively fail to segment into the desired parts. Ideally, we would like to be able to provide the model with hints to the desired level of granularity during training, and flexibly query the model during inference to request that the model detects and tracks particular object or a part thereof.
In this paper introduce sequential extension of Slot Attention (Locatello al., 2020) that call Slot Attention for Video (SAVi) to tackle the problem of unsupervised weakly-supervise multi- object segmentation and tracking video data. We demonstrate that 1) using optical flow prediction as a self-supervised objective and 2) providing a small set of abstract hints such as the center of mass position for objects conditional inputs in the first frame suffices direct the decomposition process in complex video scenes without otherwise requiring any priors on the size of objects or on the information content of their representations. We show successful segmentation and tracking for synthetic video data with significantly higher realism and visual complexity than the datasets used in prior work on unsupervised object representation learning. These results are robust with respect to noise on both the optical flow signal and the conditioning, and that they generalize almost perfectly to longer sequences, novel objects, and novel backgrounds.
SLOT ATTENTION FOR VIDEO (SAV1)
We introduce sequential extension of the Slot Attention (Locatello et al., 2020) architecture to video data, which call SAVi. Inspired by predictor-corret tial equations, SAVi performs two steps for each observed video frame: prediction and a correction step. The correction step uses Slot Attention to update (or correct) the of slot representations based on slot-normalized cross-attention with the inputs. The prediction step uses self-attention among the slots to allow for modeling of temporal dynamics and object interactions. The output of the predictor then used to initialize the corrector at the next time step, thus allowing the model to consistently track objects over time. Importantly, both of these steps are permutation equivariant and thus are able to preserve slot symmetry. See Figure 1 for schematic overview of the SAVi architecture.
methods for integration ordinary differen-
Encoder For each time-step t E {1... ... the corresponding video frame x, is first passed through small convolutional neural network (CNN) encoder (here, stack of five convolutional layers with
2

---

Published as a conference paper at ICLR 2022
ReLUs), where concatenate linear positional encoding at the second-to layer. The resulting grid of visual features is flattened into a set of vectors h; = fax( E R x Pax, where N is the size of the flattened grid (i.e., width and Dene is the dimensionality of the CNN feature maps. Afterwards, each vector is independently passed through a multi-layer perceptron (MI.)).
Slot Initialization SAVi maintains K slots, each of which can represent part of the input such as objects or parts thereof. We denote the set of slot representations at time t as S; = [s,,.... s51 E RKXD, where we use the calligraphie font to indicate that any operation on these sets is equivariant (or invariant) w.r.t. permutation of their elements. In other words, the ordering of the slots carries no information and they can be freely permuted (in consistent manner across all time steps) without changing the model output. We consider two types of initializers: conditional and unconditional. In the conditional case, we encode the conditional input either via a simple MI. (in the case of bounding boxes or center of mass coordinates) or via a CNN (in the case of segmentation masks). For slots for which there no conditioning information available (e.g. if K is larger than the number of objects), we set the conditional input to a fixed value (e.g.. '-1' for bounding box coordinates).
The encoded conditional input forms the initial slot representation for SAVi. In the unconditional case, either randomly initialize slots by sampling from Gaussian distribution independently for each video (both at training and at test time), or by learning a set of initial slot vectors.
Corrector The task of the corrector is to update the slot representations based on the visual features from the encoder. In SAVi this is done using the iterative attention mechanism introduced in Slot Attention (Locatello et al., 2020). Different from regular cross-attention mechanism (e.g. Vaswani et al., 2017) which is normalized over the inputs, Slot Attention encourages decomposition of the input into multiple slots via poftmax-normalizatiie an appealing choice for our video decomposition architecture. When using a single iteration of Slot Attention, the corrector update takes the following form:
over the output (i.e. the slots), which makes
N Ann @ ((he,)) ER A = =oofmmax K
U1
1
Σ n=1
k- ((S))) E ERNNK (1)
2,
where 2, that map to common dimension D. We apply LayerNorm (Ba et al., 2016) before each projection. The slot representations are then individually updated using Gated Recurrent Units (Cho et al., 2014) as 8 = CRU sf). Alternatively, the attention step (followed by the GRU update) can be iterated multiple times with shared parameters per frame of the input video. For added expressiveness, we apply an MLP with residual connection s + 8 + MLP(LN(35)) and LayerNorm (LN) after the GRU when using multiple Slot Attention iterations, following Locatello et al. (2020).
- -n= Ar. and @ denotes the Hadamard product. h. q. are learned linear projections
Predictor
The predictor takes the role of a transition function to model temporal dynamics, in- cluding interactions between slots. To preserve permutation equivariance, encoder (Vaswani et al., 2017). It allows for modeling of independent object dynamics as well as infor- mation exchange between slots via self-attention, while being more memory efficient than GNN-based models such as the Interaction Network (Battaglia et al., 2016). Slots are updated as follows:
Transformer
s (SL S.), S, LN LLMuuttHHHeddSslAAtnt (S.) + +.).
(2)
For MultiHeadSellAttn we use the default multi-head dot-product attention mechanism from Vaswani et al. (2017). We apply LayerNorm (L.N) after each residual connection.
The network output should be permutation equivariant (for per-slot outputs) or invariant (for global outputs) with respect to the slots. Slots can be read out either after application of the corrector or after the predictor (transition model). We decode slot representations after application of the corrector using slot-wise Spatial Broadcast Decoder (Watters et al., 2019) to produce per-slot RGB predictions of the optical flow (or reconstructed frame) and an alpha mask. The alpha mask is normalized across slots via a softmax and used to perform weighted sum over the slot-wise RGB reconstruction to arrive at a combined reconstructed frame:
Decoder
K
m: = softmax (m)),
m yt = Sacc (3!).
Yt
m
K
Our sole prediction target is optical flow for each individual video frame, which we represent as RGB images using the default conversion in the literature (Sun et al., 2018). Alternatively,
Training
3

---

Published as a conference paper at ICLR 2022
our framework also supports prediction of other image-shaped targets, such reconstruction of the original input frame. We minimize the pixel-wise squared reconstruction error (averaged over the batch), summed over both the temporal and spatial dimensions:
Lree
Vb
Vi
(4)
3 RELATED WORK
Object-centric representation learning There is a rich literature on learning object representa-
tions from static scenes (Greff et al., 2016; Eslami al., 2016; Greff et al., 2017; 2019; Burgess et al., 2019; Engelcke et al., 2020; Crawford & Pineau, 2019; Lin et al., 2020; Locatello et al., 2020; Du et al., a0 a) videos (van Steenkiste et al., 2018; Kosiorek et al., 2018; Stelzner al., 2019;
Kipf et al., 2020; Crawford & Pineau, 2020; Creswell et al., 2021) without explicit supervision. PSGNet (Bear et al., 2020) learns decompose static images or individual frames from a video into hierarchical scene graphs using motion information estimated from neighboring video frames. Most closely related to our work are sequential object- models for videos and dynamic environments, such as OP3 (Veerapaneni et al., 2020), R-SQAIR (Stanić & Schmidhuber, 2019), ViMON (Weis et al., 2020), and SCALOR (Jiang al., 2020), which learn an internal motion model for each object. SIMONe (Kabra et al., 2021) auto-encodes entire video in parallel and learns cemporally-abstracuee representations of objects. OP3 (Veerapaneni et al., 2020) uses the same decoder as SAVi and a related dynamics model, but less efficient inference process compared to Slot Attention.
In an attempt to bridge the gap to visually richer and more realistic environments, recent works in object-centric representation learning have explored integration of inductive biases related to 3D scene geometry. both for static scenes (Chen et al., 2020; Stelzner et al., 2021) and for videos (Du et al., 2021b; Henderson & Lampert, 2020; Harley et al., 2021). This is largely orthogonal our approach of utilizing conditioning and optical flow. A recent related method, FlowCaps (Sabour et al., 2020), similarly proposed use optical flow in a multi-object model. FlowCaps uses capsules (Sabour et al., 2017; Hinton et al., 2018) instead of a slot-based representation and assumes specialization of individual capsules to objects or parts of certain appearance, making it unsuitable for environments that contain a large variety of object types. Using a slot-based, exchangeable representation of objects allows SAVi to represent a diverse range of objects and generalize to novel objects at test time.
We discuss further recent related works on attention- based architectures operating sets of latent variables (Santoro et al., 2018; Goyal et al., 2021a;b;e; Jaegle et al., 2021 2021b;a), object-centric models for dynamic visual reasoning (Yi et al., 2020; Ding et al., 2021a; Bar et al., 2021), and supervised attention-base based object-centric models (Fuchs et al., 2019; Carion et al., 2020; Kamath et al., 2021; Meinhardt et al., 2021) in Appendix A.I.
Video object segmentation and tracking The conditional tasks we consider in our work are closely related to the computer vision task of semi-supervised video object segmentation (VOS), where segmentation masks are provided for the first video frame during evaluation. Different from the typical setting, which is addressed by supervised learning fully annotated videos related datasets (e.g. Caelles et al., 2017; Luiten et al., 2018), we consider the problem where models do not have access to any supervised information beyond the conditioning information on the first frame (e.g. a bounding box for each object). Several recent works have explored pre-training using self-supervision (Li al., 2019; Jabri al., 2020; Caron et al., 2021) or image classification (Zhang et al., 2020) for the semi-supervised VOS task. These models rely on having access to segmentation masks in the first frame evaluation time. We demonstrate that multi-object segmentation and tracking can emerge even when segmentation labels are absent at both training and test time.
There rich literature on using motion cues to segment objects in the computer vision community. These methods use motion information at test time to, for example, cluster trajectories in order to segment independently moving objects (Faktor & Irani, 2014) or estimate multiple fundamental matrices between two views (Isack & Boykov, 2012), to name few. Closest to our work is contemporary method (Yang et al., 2021) that trains a Slot Attention model on isolated optical flow data for bacoouuouddb segmentation of single object, independently for individual video frames and without using visual observations. Our method, on the other hand, supports multi object environments and only relies motion information as training signal but otherwise operates
4

---

Published as a conference paper at ICLR 2022
CATER
MOVi++
MOVi
Figure 2: Frame samples from the multi-object video datasets used in experiments. MOVi++ uses real-world backgrounds and 3D) scanned objects, which is significant step up in visual complexity object-centric learning.
compared to datasets used in prior work on unssperviedd/vakkly-seeeee
directly on textured visual information, which allows it to segment static scenes test time where optical flow unavailable and consistently represent and track multiple objects throughout a video.
4 EXPERIMENTS
We investigate 1) how SAVi compares existing unsupervised video decomposition methods, 2) how various forms hints (e.g., bounding boxes) can facilitate scene decomposition, and 3) how SAVi generalizes to unseen objects, backgrounds, and longer videos test time.
Metrics We report two metrics to measure the quality of video decomposition, object segmentation, and tracking: Adjusted Rand Index (ARI) and mean Intersection over Union (mloU). ARI is clustering similarity metric which we use to measure how well predicted segmentation masks match ground-truth masks in sermutation-invariae methods. Like in prior work (Greff et al., 2019; Locatello et al., 2020; Kabra et al., 2021), we only measure ARI based on foreground objects, which we refer to as FG-ARI. For video data, one cluster in the computation of ARI corresponds to the segmentation of single object for the entire video, requiring temporal consistency without object identity switches to perform well on this metric.
fashion, which makes it suitable for unsupervised
Training setup During training. we split each video into consecutive sub-sequences of 6 frames each, where we provide the conditioning signal for the first frame. We train for 100k steps (200k for fully unsupervised video decomposition) with batch size of 64 using Adam (Kingma & Ba, 2015) with a base learning rate of 10-4, We use a total of 11 slots in SAVi. For our experiments on fully unsupervised video decomposition iteration otherwise. Further architecture details and perraarameeters are provided in the appendix. On 8x V100 GPUs with 16GB memory each, training SAVi with bounding box conditioning takes approx. 12hrs for videos with 64 x 64 resolution and 30hrs for videos with 128 x 128 resolution.
use 2 iterations of Slot Attention per frame and single
4.1 UNSUPERVISED VIDEO DECOMPOSITION
We first evaluate SAVi in the unconditional setting and with a standard RGB reconstruction objective on the CATER (Girdhar& & Ramanan, 2019) dataset'. Dataset examples are shown in Figure 2. Results of SAVi and four unsupervised object representation learning baselines (taken from Kabra et al. (2021)) are summarized in Table la. The two image-based methods (Slot Attention (Locatello et al., 2020) and MONet (Burgess et al., 2019)) are independently applied to each frame, and thus lack built-in notion temporal consistency which is reflected their poor FG-ARI scores. Unsurprisingly, the two video-based baselines, S-IODINE (Greff et al., 2019) and SIMONe (Kabra et al., 2021). perform better. The (unconditional) SAVi model outperforms these baselines, demonstrating the adequacy of our architecture for the task of unsupervised object representation learning, albeit only for the case of simple synthetic data. For qualitative results, see Figure 5e and Appendix A.3.
4.2 MORE REALISTIC DATASETS
To test SAVi on more realistic data, we use two video datasets (see Figure 2) introduced in Kubric (Gr- eff et al., 2021), created by simulating rigid body dynamics of up ten 3D objects rendered in various scenes via raytracing using Blender (Blender Online Community, 2021). We refer to these as
We variant of CATER with segmentation masks (for evaluation), introduced by Kahra al. (2021).
5

---

Published as a conference paper at ICLR 2022
Table Segmentation results. Mean + standard error (5 seeds). All values in %.
(b) Conditional with optical flow supervision.
(a) Unsupervised.
MOVI
MOV++
CATER
Model Conditioning)
FG-ARI
mloUr
FG-ARIT
mloUt
Model
FG-ARI
Segmentation Propagation
61.5
4..
37.8
33.3
Slot Attention
7.3 0.3
SAVi Segmentation
93.7 0.2
72.0
20.4
43.0 1..6
MONet
0.5
SAVi Bounding Box
93.7 0.00
71.2 1.06
77 =03
45.9 1
S-IODINE
1.5
SAV Center of Mass
93.8 :
72.1 2.2
78.3 0.6
43.5 1:55:22.
SIMONe
91.8116
CRW (Jabri et al., 2020)
42.4
50.9
SAVi (uncond.)
92.8 0.8
T-VOS (Zhang et al,, 2020)
50.4
46.4
SAVi (ResNet) + Bounding Box
82.8
50.7 0.2
SAVi + Segm.
97.91
Flow k- K-Mcans + Center of Mass
30.2
5.8
32.6
9.3
MOVI++
100
MOVI
100
MOVI++
SAVi
80
FG (RI [1%]
75
75
SAVI (estimated flow)
660
SAVI (w/o flow)
50-
50
SAVI (unconditional)
40
FG-ARI
SAVI (uncond. w/o flow)
20
miau
SCALOR
10-2
10-1
10f
Noise [a/ mean objectisize
(a) Supervision ablation on MOVi & MOVi+.
(b) Robustness to noise.
Figure 3: (a) Supervision ablations for a SAVi model trained with bounding boxes conditional
input. (b) Robustness analysis with Gaussian noise added to the conditioning signal. In both figures, evaluate on the first six frames and report mean + standard error across seeds.
Multi-Object Video (MOVi) datasets. The first dataset, MOVi, uses the same simple 3D shapes the CATER dataset (both inspired by the CLEVR benchmark (Johnson et al., 2017)), but introduces more complex rigid-body physical dynamics with frequent collisions and occlusions. The second dataset, MOVi++ significantly increases the visual complexity compared to datasets used in prior work and uses approx. 380 high resolution HDR photos as backgrounds and of 1028 3D scanned everyday objects (Google Research, 2020), such as shoes, toys, or household equipment. Each dataset contains 9000 training videos and 1000 validation videos with 24 frames at 12 fps each and, unless otherwise mentioned, a resolution of 64 x 64 pixels for MOVi and 128 x 128 pixels for MOVi++.
Due to its high visual complexity the MOVi++ dataset is a significantly more challenging testcase for unsupervised and al., 2020) (using the reference implementation) was able to achieve 81.2 + 0.4% FG-ARI on MOVi, it only resulted in 22.7 + 0.9% FG-ARI on MOVi++. The SIMONe model also only achieved around 33% FG-ARI (with one outlier as high as 46%), using our own re-implementation (which approximately reproduces the above results CATER). We observed that both models, as well as SAVi, converged to a degenerate solution where slots primarily represent fixed regions of the image (often rectangular patch stripe), instead of tracking individual objects.
instance segmentation. In fact, while SCALOR (Jiang
4.3 CONDITIONAL VIDEO DECOMPOSITION
To tackle these more realistic videos we 1) change the training objective from predicting RGB image to optical flow and 2) condition the latent slots of the SAVi model on hints about objects in the first frame of the video, such as their segmentation mask. We showcase typical qualitative results for SAVi in Figure 4, and on especially challenging cases with heavy occlusion or complex textures in Figures 5a and 5b. In all cases, SAVi was trained on six consecutive frames, but we show results for full (24 frame) sequences at test time. Using detailed segmentation information for the first frame is common practice in video object segmentation, where models such as T- T-VOS (Zhang et al., 2020) or CRW (Jabri et al., 2020) propagate the initial masks through the remainder of the video sequence.

---

Published as a conference paper at ICLR 2022
t=1
13
17
10
13
16
19
22
Video
G.T.
Pred.
Pred. G.T.
Figure 4: Representative qualitative results of SAVi conditioned on bounding box annotations in the first video frame. We visualize predicted (Pred.) segmentations and optical flow predictions together with their respective ground-truth values (G.T.) for both MOVi (left) and MOVi++ (right). We visualize the soft segmentation masks provided by the decoder of SAVi. Each mask is multiplied with color that identifies the particular slot it represents.
Video
G.T. Video
G.T. Video
G.T.
Pred.
Pred.
Pred.
(a) Successfully handled challenging examples.
(b) Failure case.
(c) Unsupervised.
Figure 5: (a) SAVi successfully handles examples with complex texture (left) and occlusion with object of similar appearance (right). (b) In heavily occluded example SAVi can lose track of objects and swap object identities. (c) Unsupervised segmentation on CATER with RGB reconstruction.
Quantitative Evaluation On the MOVi and MOVi++ datasets T-VOS achieves 50.4% and 46.4% mloU respectively, whereas CRW achieves 42.4% and 50.9% mIoU (see Table lb). SAVi learns to produce temporally consistent masks that are significantly better on MOVi (72.0% mloU) and slightly worse than T-VOS and CRW on MOVi++ (43.0% mloU) when trained to predict flow and with segmentation masks as conditioning signal in the first frame. However, we hold that this comparison is not entirely fair, since both CRW and T-VOS use more expressive ResNet (He et al., 2016) backbone and operate on higher-resolution frames (480p). For T-VOS, this backbone is pre-trained on ImageNet (Deng et al., 2009), whereas for CRW re-train it on MOVI/MOVi++. For a fairer comparison, we adapt T-VOS (Zhang et al., 2020) to our setting by using the same CNN encoder architecture the same resolution in SAVi and trained using the same supervision signal (i.e. optical flow of moving objects). While this adapted baseline (Segmentation Propagation in Table 1b) performs similar T-VOS MOVi, there is significant drop in performance MOVi++. We verify that SAVi can similarly benefit from using a stronger ResNet backbone on MOVi++ (SAVi (ResNet) Table lb), which closes the gap to the CRW (Jabri al., 2020) baseline.
Conditioning Ablations Surprisingly, find that even simple hints about objects in the first video frame such as bounding boxes. or even just point in the center of a bounding box (termed as center of mass), suffice for the model establish correspondence between hint and visual input, and to retain the same quality in object segmentation and tracking. Conditioning on such simple signals is not possible with segmentation propagation models (Zhang et al., 2020; Jabri et al., 2020) as they require accurate segmentation masks input. To study whether an even weaker conditioning signal suffices, we add noise sampled from a Gaussian distribution. (000, o) - to the center-on oa-mass coordinates provided to the model in the first frame, both during training and testing. We find that SAVi remains robust to noise scales a up to 20% the average object size (Figure 3b), after which point mloU starts to decay (and faster than FG-ART), indicating that SAVi starts to lose correspondence between
7

---

Published as a conference paper at ICLR 2022
MOVI
MOVI
Default
abj. bg.
New obj.
MOVI
70
New bg.
FG-ARI 156
90.0
SAVi Segmentation
196]
60
SAVI + Segmentation
SAVi + Bounding Box
SAVI + Center of Mass
100-
MOVi+++
87.5
Bounding Box
Segmentation
FG [k]]
75
SAVI + Center of Mass
50
Propagation
SAVI
Init. SAVI + Learned Init.
82.5
25
5
11
15 10 21
15 17 Frame
21
Frame
Evaluation data
(a) Per-frame evaluation scores on MOVi.
(b) OOD generalization.
Figure 6: (a) Per-frame segmentation quality. Grey area highlights the number of frames used during training. (b) Video decomposition results on out-of-distribution evaluation splits with new objects and/or new backgrounds (6 frames). Mean + standard error for 5 seeds.
the conditioning signal and the objects in the scene. Full ablation of the conditioning signal leads to significant drop in performance (Figure 3), which demonstrates that conditional hints are indeed important to SAVi, though even very rough hints suffice.
Optical flow ablations Using ground-trute truth optical flow training target especially in the absence of camera motion - is a form of weak supervision that won't be available for real data. To investigate its benefits and importance, we first apply k-means clustering the optical flow signal (enriched with pixel position information) where we use the center of mass coordinates of objects to initialize the cluster centers. The results (see Table lb) show that simply clustering the optical flow signal itself is insufficient for accurate segmentation. Next, we perform the following ablations on a SAVi model with bounding boxes as conditional input, summarized in Figure 3: 1) instead of using ground-truth optical flow provided by the simulator/renderer, we use approximate optical flow obtained from a recent unsupervised flow estimation model (Stone et al., 2021) (estimated flow); 2) we replace optical flow with RGB pixel values training targets, i.e. using a reconstruction target (low)))w;; 3) we train SAVi without conditioning by initializing slots using fixed parameters that are learned during training (unconditional), We find, that estimated flow confers virtually the same benefits as ground truth-flow, and that flow is unnecessary for the simpler MOVi dataset. For MOVi++, the other hand, it is crucial for learning an accurate decomposition of the scene, and without access to optical flow (true or estimated) SAVi fails to learn meaningful object segmentations.
4.4 TEST TIME GENERALIZATION
Longer sequences For memory efficiency, we train SAVi on subsequences with a total of 6 frames, ie. shorter than the full video length. We find, however, that even without any additional data augmentation or regularization, SAVi generalizes well to longer sequences at test time, far beyond the setting used in training. We find that segmentation quality measured in terms of FG-ARI remains largely stable, and surprisingly even increases for unconditional models (Figure 6a). Part of the increase can be explained by objects leaving the field of view later in the video (see examples in the appendix), but models nonetheless appear to benefit from additional time steps to reliably break symmetry and bind objects to slots. The opposite effect can be observed for segmentation propagation models: performance typically decays over time as errors accumulate.
New objects and backgrounds We find that SAVi generalizes surprisingly well settings with previously unseen objects and with new backgrounds at test time: there is no significant difference in both FG-ARI and mloU metrics when evaluating SAVi with bounding box conditioning on evaluation sets of MOVi++ with entirely new objects or backgrounds. On an evaluation set where we provide both new backrounds and new objects at the same time, there a drop of less than 2% (absolute), see Figure 6b. Remarkably, SAVi model trained on MOVi+ also transfers well to MOVi at test time.
Part-whole segmentation Our dataset is annotated only at the level of whole objects, yet we find anecdotal evidence that by conditioning at the level of parts during inference the model can in fact successfully segment and track the corresponding parts. Consider, for example the two green fists in Figure 7 which in the dataset are treated single composite object. When conditioned with
8

---

Published as a conference paper at ICLR 2022
Whole
t=1
13 15
17
Video
Conditioning
signal for:
Who e
Slot
Parts
Slot
Slot 3
Slot
Parts
Slot
Figure 7: Emergent part-whole segmentation by querying the model at test time in different ways: The model attends to either both green fist with single slot or to each individual fist with separate slot, depending on the granularity of the conditioning signal. We visualize SAVi's attention masks.
single bounding box encompassing both fists (Whole), on two bounding boxes (Parts), SAVi either segments and tracks either the composite object or the individual parts (see appendix for an additional example). This finding is remarkable for two reasons: Firstly, it shows that the model can (at least to some degree) generalize to different levels of granularity. This kind of generalization is very useful, since the notion of an object is generally task-dependent and the correct level of granularity might not be known at training time. Secondly, this experiments demonstrates a powerful means for interfacing with the model: rather than producing a single fixed decomposition, SAVi has learned to adapt its segmentation and processing of the video based on the conditioning signal. We believe that this way of learning a whole hierarchy only implicitly and segmenting different levels on demand is ultimately more flexible and tractable than trying to explicitly represent the entire hierarchy at once.
4.5 LIMITATIONS
There are still several obstacles that have to be overcome for successful application of our framework to the full visual and dynamic complexity of the real world. First, our training set up assumes the availability of optical flow information at training time, which may not be available in real- world videos. Our experiments, however, have demonstrated that estimated optical flow from an unsupervised model such SMURF (Stone et al., 2021) can be sufficient train SAVi. Secondly, the environments considered in our work, despite taking a large step towards realism compared to prior works, are limited by containing solely rigid objects with simple physics and, in terms of the MOVi datasets, solely moving objects. In this case, optical flow, even in the presence of camera movement, can be interpreted as an indirect supervision signal for gureground-backgrourg per-object) segmentation. Furthermore, training solely with optical flow presents a challenge for static objects. Nonetheless, find evidence that SAVi can reliably decompose videos in constrained real-world environment: we provide qualitative results demonstrating scene decomposition and long-term tracking in robotic grasping environment (Cabi al., 2020) in Appendix A.3. Most real-world video data, however, is still of significantly higher complexity than the environments considered here and reliably bridging this gap is an open problem.
(but not
5 CONCLUSION
We have looked at the problem of learning object representations and physical dynamics from video. We introduced SAVi, an object-centric architecture that uses attention over latent of slots to discover, represent and temporally track individual entities in the input, and we have shown that video in general and flow prediction particular are helpful for identifying and tracking objects. We demonstrated that providing the model with simple and possibly unreliable position information about the objects in the first frame is sufficient to steer learning towards the right level of granularity to successfully decompose complex videos and simultaneously track and segment multiple objects. More importantly, we show that the level of granularity can potentially be controlled at inference time, allowing a trained model to track either parts of objects entire objects depending the given initialization. Since our model achieves very reasonable segmentation and tracking performance, this is evidence that object-centric representation learning is not primarily limited by model capacity. This way of conditioning the initialization of slot representations with location information also potentially opens the door for wide range of semi-supervised approaches.

---

Published as a conference paper at ICLR 2022
ETHICS STATEMENT
Our analysis, which is focused on multi-object video data with physically simulated common household objects, has no immediate impact on general society. As with any model that performs dynamic scene understanding, applications with potential negative societal impact such as in the area of surveillance, derived from future research in this area, cannot be fully excluded. We anticipate that future work will take steps to close the gap between the types of environments considered in this work, and videos of diverse scenes taken in the real world, which will likely enable applications in various societally relevant domains such robotics and general video understanding.
ACKNOWLEDGEMENTS
We would like thank Mike Mozer and Daniel Keysers for general advice and feedback on the paper, and David Fleet, Dirk Weissenbom, Jakob Uszkoreit, Marvin Ritter, Andreas Steiner, and Etienne Pot helpful discussions. We are further grateful Rishabh Kabra for sharing the CATER (with masks) dataset and to Yi Yang, Misha Denil, Yusuf Aytar, and Claudio Fantacci for helping get started with the Sketchy dataset.
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06455. 2016.
Amir Bar, Roei Herzig, Xiaolong Wang, Anna Rohrbach, Gal Chechik, Trevor Darrell, and Amir Globerson. Compositional video synthesis with action graphs. In International Conference on Machine Learning, 2021.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects. relations and physics. In Advances in Neural Information Processing Systems, pp. 4502-4510, 2016.
Daniel M Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Joshua Tenenbaum, al. Learning physical graph representations from visual scenes. In Advances in Neural Information Processing Systems, 2020.
Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2021. URL http: www. www.nee.................. org-
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPP programs, 2018. URL
http:/ github.. com/ comm//ogle/////oooooooooooooo jax.
Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised scene decomposition and representation. arXiv preprint arXiv: 13390, 2019.
Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikoy, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerik, Oleg Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyu Wang. Scaling data-driven robotics with reward sketching and batch reinforcement learning. In Robotics: Science and Systems, 2020.
Sergi Caelles, Kevis-Kokint Maninis, Jordi Pont- Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 221-230. 2017.
Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, KokitiK Maninis, and Luc Van Gool. The 2019 DAVIS challenge on vos: Unsupervised multi multi-object segmentation. arXiv preprint arXiv: / 9099900007777 2019.
10

---

Published as a conference paper at ICLR 2022
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zogoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104. 14294, 2021.
Chang Chen, Fei Deng, and Sungjin Ahn. Object-centric representation and rendering of 3D scenes. arXiv preprint arXiv:2006.06 2020.
Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan-Yee Yee Kenneth Wong, Joshua B Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. In International Conference on Learning Representations, 2021.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder- for statistical machine translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, 2014.
Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional neural networks. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 3412-3420, 2019.
Eric Crawford and Joelle Pineau. Exploiting spatial invariance for scalable unsupervised object tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 3684-3692, 2020.
Antonia Creswell, Rishabh Kabra, Chris Burgess, and Murray Shanahan. Unsupervised object-based transition models for 3D partially observable environments. arXiv preprint arXiv:2103.044: 046693, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2009.
David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, and Matt Botvinick. Attention over learned object embeddings enables complex visual reasoning, In Advances in Neural Information Processing Systems, 2021a.
Mingyu Ding, Zhenfang Chen, Tao Du, Ping Luo, Joshua B Tenenbaum, and Chuang Gan. Dynamic visual reasoning by learning differentiable physics models from video and language. In Advances In Neural Information Processing Systems, 2021b.
Yilun Du, Shuang Li, Yash Sharma, B. Joshua Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. In Advances in Neural Information Processing Systems, 2021
Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3D physical objects from video. In International Conference on Learning Representations, 2021b.
Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Generative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
SM Ali Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey Hinton, et al. Attend, infer, repeat: Fast scene understanding with generative models. In Advances in Neural Information Processing Systems, pp. 3225-3233, 2016.
Alon Faktor and Michal Irani. Video segmentation by non-local consensus voting. In Proceedings of the British Machine Vision Conference, 2014.
Fabian B Fuchs, Adam R Kosiorek, Li Sun, Oiwi Parker Jones, and Ingmar Posner. End-to-end recurrent multi object tracking and trajectory prediction with relational reasoning. arXiv preprint arXiv: 12877 2019.
11

---

Published as a conference paper at ICLR 2022
Rohit Girdhar and Deva Ramanan. Cater: A diagnostic dataset for compositional actions and temporal reasoning. arXiv preprint arXi::1910.04744 2019.
Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Google Research. Google scanned objects, 2020. URL htttttss: htt.:/ppp...nnttiioooooooooooo l ons ons//oogl (elooole@20Scammmnnn 82 0Ob jobjssts. org/ hewooogleeeeeeeeeeeeeemm wwwe/colllloooooooooooooo/oooooa (ooog//eeeeeearcc///////llllllee
Anirudh Goyal, Aniket Didolkar, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nico- las Heess, Michael Mozer, and Yoshua Bengio. Neural production systems. arXiv preprini arXiv:2103.01937, 2021a.
Anirudh Goyal, Alex Lamb, Phanideep Gampa, Philippe Beaudoin, Sergey Levine, Charles Blun- dell, Yoshua Bengio, and Michael Mozer. Factorizing declarative and procedural knowledge in structured, dynamic environments. In International Conference Learning Representations, 2021b.
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In International Conference on Learning Representations, 2021c.
Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems, pp. 4484 4492, 2016.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pp. 6691-6701, 2017.
Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. International Conference on Machine Learning, pp. 2424-2433, 2019.
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint urXiv:2012.05208. 2020.
Klaus Greff, Andrea Tagliasacchi, Derek Liu, Issam Laradji, Or Litany, and Luca Prasso. Kubric: A data generation pipeline for creating semi-realistic synthetic multi-object videos, 2021. URL
https: //tp::://tithbbbb cwmm /gonglgooleeeeeoooooooooeoooeeee rwwearch/kuurr......
Adam W Harley, Yiming Zuo, Jing Wen, Ayush Mangal, Shubhankar Potdar, Ritwick Chaudhry, and Katerina Fragkiadaki. Track, check, repeat: An EM approach to unsupervised tracking. arXiv preprint MXXi::2144004222 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http:/ g thub. com/ goog 1 e flax.
Paul Henderson and Christoph Lampert. Unsupervised object-centric video generation and decomposition in 3D. In Advances Neural Information Processing Systems, 2020.
Roei Herzig, Elad Levi, Huijuan Xu, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Globerson, and 2019.
Trevor Darrell. Spatio-temporal action graph networks. arXiv preprint
Geoffrey Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with EM routing. In
International Conference Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2022
Hossam Isack and Yuri Boykov. Energy-basee geometric multi-model fitting. International Journal of Computer Vision, 97(2):123-144 111223--7,,40 2012.
Allan Jabri, Andrew Owens, and Alexei A Efros. Space-time correspondence as a contrastive random walk. In Advances in Neural Information Processing Systems, 2020.
Andrew Jaegle, Sebastian Borgeaud, Jean-Raptiste Alayrac, Carl Doersch, Catalin loneseu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver IO: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795 2021a.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals. and Joao Carreira. Perceiver: General pereeption with iterative attention. In International Conference on Machine Learning, 2021b.
Ashesh Jain, Amir R Zamir, Silvio Savarese, and Ashutosh Saxena, Structural-RNe Deep learning on spatio-temporal graphs. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2016.
Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. SCALOR: Generative world models with scalable object representations. In International Conference Learning
Representations, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Putlern Recognition, 2017.
Rishabh Kabra, Daniel Zoran, Loic Matthey Goker Erdogan, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P. Burgess. SIMONe: View-invariant, temporally-abstracted object representations via unsupervised video decomposition. arXiv preprint arXiv:2106.03849 2021.
Daniel Kahneman, Anne Treisman, and Brian J Gibbs. The reviewing of object files: Object-specific integration of information. Cognitivee psychology, 24(2):175 219, 1992.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. MDETR - Modulated Detection for End-to-End Multi-Modal Understanding. arXiv preprint arXiv:2104.12763 2021.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevitex: A texture-rich benchmark for www.weenttofuteoeeooo
unsupervised multi-object segmentation, 2021. URL id=J4N1 2qRM rR.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Thomas Kipf, Elise van der Pol, and Max Welling. Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. Advances in Neural Information Processing Systems, pp. 1606-86166 2018
Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint- task self-supervised learning for temporal correspondence. Advances Neural Information Processing Systems, 2019.
Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng. Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised object-oriented scene representation via spatial attention and decomposition. In International Conference Learning Representations, 2020.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, 2020.
13

---

Published as a conference paper at ICLR 2022
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.
Iyya Loshchilov and Frank Hutter. SGDR:
Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
Jonathon Luiten, Paul Voigtlaender, and Bastian Leibe. Premvos:
refinement and merging for video object segmentation. In Asian Conference Computer Vision, pp. 565-580. Springer, 2018.
Aravindh Mahendran, James Thewlis, and Andrea Vedaldi. Cross pixel optical-flow similarity for self-supervised learning. In Asian Conference on Computer Vision, pp. 99-116. Springer, 2019.
Tim Meinhardt, Alexander Kirillov,, Laura Leal-Taixe, and Christoph Feichtenhofer. TrackFormer: Multi-Object Tracking with Transformers. arXiv preprint arXiv:2101.02702, 2021.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Dorkine-Hornung and Luc Van Gool. The 2017 DAVIS challenge video object segmentation. arXiv preprint arXiv:1704.00675. 2017.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in Neural Information Processing Systems, pp. 3856-3866 2017.
Sara Sabour, Andrea Tagliasacchi, Soroosh Yazdani, Geoffrey E Hinton, and David Fleet. Unsuper- vised part representation by flow capsules. arXiv preprint arXiv:2011.139200 2020.
Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 7299-7310, 2018.
Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96 2007.
Aleksandar Stanić and Jürgen Schmidhuber. R-SQAIR: Relational sequential attend, infer, repeat. arXiv preprint arXiv:1910.05233 2019.
Karl Stelzner, Robert Peharz, and Kristian Kersting. Faster attend-infer-repeal with tractable proba- bilistic models. In International Conference on Machine Learning, pp. 5966-5975. 2019.
Karl Stelzner, Kristian Kersting, and Adam R Kosiorek. Decomposing 3D scenes into objects via unsupervised volume segmentation. arXiv preprint arXiv:2104.01148, 2021.
Austin Stone, Daniel Maurer, Alper Ayvaci, Anelia Angelova, and Rico Jonschkowski. SMURF: Self-teaching multi-frame unsupervised RAFT with full-image warping. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2021.
Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 9933, 2018.
Aniruddha Kembhavi Derek Hoiem Tanmay Gupta, Amita Kamath. Towards general purpose vision systems. arXiv preprint arXiv: 1el::2204400445 2021.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural ex- pectation maximization: Unsupervised discovery of objects and their interactions. International Conference Learning Representations, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, pp. 5998-59988988 6008, 2017.
Rishi Veerapaneni, John D ReyRey,s, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual mrodel-basee reinforcement learning. In Conference Robot Learning, pp. 1439-1456, 2020.
14

---

Published as a conference paper at ICLR 2022
Jacob Walker, Abhinav Gupta, and Martial Hebert. Dense optical flow prediction from static image. In Proceedings of the IEEE International Conference on Computer Vision, 2015.
Nicholas Watters, Loic Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations VAEs. arXiv preprint arXiv:1901.000/1 2019.
Marissa A Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S Ecker. Unmasking the inductive biases of unsupervised object representations for video sequences. arXiv preprint arXiv:2006.00 07034, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference computer vision (ECCV), pp. 3-19, 2018.
Charig Yang, Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. arXiv preprint arXiv:2104.c 0.00000...8.... 2021.
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua Tenenbaum. CLEVRER: Collision events for video representation and reasoning. In International Conference on Learning Representations, 2020.
Yizhuo Zhang, Zhirong Wu, Houwen Peng, and Stephen Lin. transductive approach for video object segmentation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6949-6958, 2020.
Jiaojiao Zhao, Xinyu Li, Chunhui Liu, Shuai Bing, Hao Chen, Cees GM Snoek, and Joseph Tighe. TubeR: Tube- Transforree for Action Detection. arXiv preprint wrXi::2104.00969 2021.
15

---

Published as a conference paper at ICLR 2022
A APPENDIX
In Section A.I, we discuss additional related work. In Sections A.2-A.4, we show additional qualitative and quantitative results. Section A.4 further contains results for an ablation study on SAVi. Lastly, in Sections A.5-A.8, we report additional details on datasets, model architecture, and training setup (hyperparameters, baselines, metrics).
A.I ADDITIONAL RELATED WORK
Attention-based networks for sets of latent variables SAVi shares close connections with other attention-based modular neural networks. Relational Recurrent Networks (Santoro et al., 2018 (8) and the RIM model (Goyal et al., 20210) use related attention mechanisms to map from inputs to slots and from slots to slots, the latter of which (RIM) uses specialized neural network parameters for each slot. Follow-up works (Goyal et al., bba) introduce additional sparsity and factorization constraints on the dynamics model, which is orthogonal our contributions of conditional learning and dynamics modeling using optical flow. The Perceiver (Jaegle et al., 2021b) uses an architecture similar to Slot Attention (Locatello et al., 2020), i.e. using a of latent vectors which are (recurrently) updated by attending on a set of input feature vectors, for classification tasks. The Perceiver IO (Jaegle al., 2021a) model extends this architecture by adding an attention- based decoder component to interface with variety of structured prediction tasks. SAVi similarly maps inputs onto a latent set of slot vectors using an attention mechanism, but differs in the decoder architecture and in its sequential nature: different from Perceiver IO, SAVi processes sequence of inputs (e.g. video frames) of arbitrary length, encoding and decoding a single input (or output) element per time step while maintaining latent set of slot variables that is carried forward in time.
Supervised slot-based models for visual tracking Slot-based architectures have also been ex- plored in the context of fully supervised multi-ohject tracking and segmentation with models such as MOHART (Fuchs et al., 2019), TrackFormer (Meinhardt al., 2021), and TubeR (Zhao et al., 2021). In the image domain, GPV-I (Tanmay Gupta, 2021) and MDETR (Kamath et al., 2021; Carion et al., 2020) explore conditioning supervised slot-based models auxiliary information, in a form which requires combinatorial matching during training. SAVi learns to track and segment multiple objects without being directly supervised to do so, using only weak forms of supervision for the first frame of the video. Slot conditioning allows us to avoid matching and to consistently identify and track individual objects by their specified query.
Dynamic visual reasoning and action graphs Object-centric models for dynamic scenes have found success in other relational tasks in vision, such as supervised visual reasoning (Yi et al., 2020; Chen al., 2021; Ding et al., 202]a;b), activity/action recognition (Jain et al., 2016; Girdhar et al., 2019; Herzig et al., 2019), and compositional video synthesis with action graphs (Bar et al., 2021). Many of these approaches use pre-trained (Chen al., 2021) and sometimes frozen (Yi al., 2020; Ding et al., 202 la) object detection backbones. Combining these methods with an end-to-end video object discovery architecture such as SAVi using self supervise learning objectives is an interesting direction for future work.
A.2 REAL-WORLD ROBOTICS TASK
To evaluate whether SAVi can decompose real-world video data in a limited domain, we train and qualitatively evaluate an unsupervised SAVi model with RGB reconstruction on the Sketchy dataset (Cabi et al., 2020). The Sketchy dataset contains videos of a real-world robotic grasper interacting with various objects.
We use the human demonstration sequences as part of the "rgb30_all" subset of the Sketchy dataset (Cabi et al., 2020), resulting in a total of 2930 training videos of 201 frames cach. In Figure A.1 and in the supplementary videos (see supplementary material), we show qualitative results on unseen validation data.
Our qualitative results in Figure A.1 demonstrate that SAVi can decompose these real-world scenes into meaningful object components and consistently represent and track individual scene components over long time horizons, far beyond what is observed during training. SAVi is trained with per-slot
16

---

Published as a conference paper at ICLR 2022
MLP predictor model (with single hidden layer of 256 hidden units, skip connection, and Layer Normalization) for IM steps with otherwise the same architecture as the unsupervised SAVi model used for CATER described in the main paper.
11
21
31
41
51
61
71
81
91
101
Video
Pred. Recon.
Slot 10 Slot 11
Slot 9
Slot 8
Siot
Slot 6
Slot 5
Slot 4
Slot 3
Slot
Slot
Figure A.1: unllupresuee video decomposition on Sketchy (Cabi et al., 2020), a real-world robotic grasping dataset, with SAVi model trained using 11 object slots. In addition to the predicted segmentation of the scene (Pred.), we visualize reconstructions of individual slots (multiplied with their respective predicted soft segmentation mask). We assign all three discovered background slots (slots 2, 3, and 8) the color black to ease interpretation. Additional videos are provided in the supplementary material.
17

---

Published as a conference paper at ICLR 2022
A.3 ADDITIONAL QUALITATIVE RESULTS
Part-whole segmentation In Figure A.2 we show another example of steerable part-whole seg- mentation based the granularity of the hint provided for the first frame. In this example, the laptop in the foreground of the scene is either annotated with a single bounding box that covers the full object (which is the granularity of annotation originally provided in the dataset), or we re-annotate this object as two separate instances, where one bounding box covers the screen of the laptop and the other bounding box covers the base incl. the keyboard. Despite never having seen a separate "screen" object in the dataset, the model can reliably identify, segment, and track each individual part, as can be seen both by the predicted decoder segmentation masks and the corrector attention masks. Alternatively, if the conditional input covers the full object, the model tracks and segments the entire laptop using a single slot, here shown in green.
We further observe that the corrector attention masks much more sharply outline the object compared to the decoder segmentation masks, but have artifacts for large objects which is likely due to the limited receptive field of the encoder CNN used in our work (for reasons of simplicity). Both this gap in sharpness and the artifacts seen in the corrector masks for large objects can likely be addressed in future work by using powerful encoder and decoder architectures.
Whole
t-1
Video
Video
Whole
Whole
Parts
Parts
Parts
(a) Inputs.
(b) Decoder segmentation masks.
(c) Corrector attention masks.
Figure A.2: An example of query-dependent part vs. whole segmentation. (a) shows the input bounding boxes provided conditioning signal. (b) and (c) show the per-slot segmentations obtained
for each of the two conditioning cases in terms of the decoder segmentation masks and corrector attention masks, respectively.
Unsupervised video decomposition In Figure A.3 we show qualitative results for fully- unsupervised video decomposition with SAVi on the CATER dataset (on unseen validation data). Since SAVi is trained with RGB reconstruction loss in this setting, we can visualize reconstructions of individual objects over multiple time steps in the video.
Extrapolation to long sequences In Figures A.4-A.5 show several representative examples of applying SAVi with bounding box conditioning on full-length MOVi and MOVi++ videos of 24 frames, after being trained on sequences of only 6 frames.
Moving camera In Figure A.6 we show qualitative results for SAVi (with bounding box condition- ing) on more challenging variant of the MOVi+ dataset with (linear) camera movement, where optical flow encodes both object movement and movement of the background relative to the camera. This effectively leads to colored background the optical flow images, and weakens the usefulness of optical flow for snnerouud-bbackrooun conditioning achieves approx. 65.5 + 0.5% FG-ARI on this modified dataset, and we find that still able to segment and track objects, although at lower fidelity than in MOVi++.
segmentation. Nonetheless, SAVi with bounding box
Baselines: Segmentation Propagation In Figure A.7 we show qualitative results for the Seg- mentation Propagation baseline on couple of sequences each in the MOVi and MOVi+ datasets. In these image grids, 'Pred.' corresponds to the soft segmentation masks before argmax similar to the illustrations for SAVi above. We also show 'Pred. Argmax' which corresponds to the hard
18

---

Published as a conference paper at ICLR 2022
1
7
9
11
13
15
Video
Slot 10 Slot 11 Pred. G.T.
Siot 9
Slot 8
Slot 7
Slot 6
Slot
Slot 4
Siot 3
Slot
Siot 1
Figure A.3: Fully-unsupervieed video decomposition on CATER with a SAVi model trained us- ing 11 object slots. In addition the predicted segmentation of the scene (Pred.), we visualize reconstructions of individual slots (multiplied with their respective predicted soft segmentation mask).
19

---

Published as a conference paper at ICLR 2022
t=1 1
3
5
9
11
13
15
17
19
21
23
Video
G.T.
Pred.
G.T.
Pred.
t=1
9
11
13
15
17
19
21
23
Video
G.T.
Pred.
Pred. G.T.
15
21
G.T.
t=1
9
11
13
15
17
19
21
Video
G.T.
Pred.
G.T.
Qualitative extrapolation results for SAVi model with bounding box conditioning trained on sequences of frames on MOVi.
Figure A.4:
20

---

Published as a conference paper at ICLR 2022
3
5
9
11
13
15
17
19
21
23
Video
G.T.
Pred.
G.T.
Pred.
t=1
3
9
11
13
15
17
19
21
23
Video
G.T. Pred.
Pred.
21
G.T. Video
1
[=1
3
5
9
11
13
15
17
19
21
23
Video
G.T.
Pred.
G.T.
Qualitative extrapolation results for SAVi model with bounding box conditioning trained on sequences frames on MOVi++.
Figure A.5:
21

---

Published as a conference paper at ICLR 2022
3
5
9
11
13
15
17
19
21
23
Video
G.T.
Pred.
G.T.
Pred.
t=1
5
11
13
15
17
19
21
23
Video
G.T.
Pred.
G.T.
Pred.
Video
G.T.
11
13
15
17
19
21
Video
G.T.
G.T.
Figure A.6: Qualitative results for a SAVi model with bounding box conditioning trained on a varaint of MOVi++ with moving camera.
22

---

Published as a conference paper at ICLR 2022
11
15
19
21
Video
G.T.
Argmax Pred.
Pred.
G.T.
Pred.
11
13
15
17
19
21
23
Video
G.T.
Pred.
Argmax
G.T.
Pred.
t=1
7
11
15
17
19
21
23
Video
G.T.
Pred.
Argmax
Pred.
G.T.
Pred.
Figure A.7: Qualitative results for the Segmentation Propagation baseline with segmentation condi- tioning on the MOVi (top example) and on the MOVi++ (bottom two examples) datasets.
23

---

Published as a conference paper at ICLR 2022
segmentation masks post argmax. The last row in each image grid corresponds to optical flow predicted from a single image. This is expected to be poor since flow prediction from a single image a highly ambiguous task. We find that the Segmentation Propagation baseline struggles to retain small objects. These often disappear or become part of a bigger object. The soft masks also suggest that the model is struggling keep the background separate from objects. 9 continuous reference frames were used for label propagation in Segmentation Propagation. See Section A.7 for more details.
G.T. Video
t=1
9
11
13
15
17
19
21
23
Video
G.T.
Pred.
Figure A.8: Qualitative results for a SCALOR model trained on sequences of 6 frames on MOVi,
t=1
3
5
9
11
13
15
17
19
21
23
Video
G.T.
Pred.
21
G.T. Video
Figure A.9: Qualitative results for SCALOR model trained on sequences of 6 frames on MOVi++
Baselines: SCALOR Figures A.8 and A.9 show qualitative results for the SCALOR baseline on examples from the MOVi and MOVi+ datasets. Often, when small objects in MOVi get close each other, SCALOR merges them as one object. On MOVi++, SCALOR struggles with object boundaries and often picks up background texture extra objects.
A.4 ADDITIONAL QUANTITATIVE RESULTS
Transfer to static scenes key advantage of SAVi over methods that segment objects using optical flow input, is that SAVi can be applied to static images where optical flow is available. We
24

---

Published as a conference paper at ICLR 2022
demonstrate generalization to static images by evaluating 'SAVi Bounding box' models the first frame of all videos in the validation set. To this end, we repeat the first frame twice to form "boring" video to test whether the model can accurately segment and "track" objects even in the absence of motion. this setting, our model achieves a FG-ARI score of 88.7 + 0.2 on the MOVi++ dataset, i.e. it accurately segments the static scene.
Tracking subsets of objects We investigate interfacing with SAVi track subset of the objects present in the first frame. We achieve this by conditioning slots using bounding boxes/segmentation masks for some of the objects and ignoring the bounding boxes/segmentation masks for the re- maining objects. We found it beneficial, for these experiments, to initialize unconditioned slots with random vectors, so that they are free model any remaining objects. We also found beneficial to expose the model to this scenario at training time. We encoded whether slot re- ceived conditional input or not using binary indicator which we appended the initial slot value. Figure A.10 reports mloU across various experi- mental conditions. The top two plots study SAVi under bounding box conditioning and the bot- tom two plots study SAVi under segmentation mask conditioning.
60
70
mioU (96
40
mloU 196
60
Varying subset sizes evaluation: For the right two plots in Figure A0 we train SAVi with up to 6 objects and evaluate with varying subset sizes. We observe that, selecting and tracking subsets of objects is indeed possible, but the in- troduction of randomly initialized slots comes at price: with more objects without a condition- ing signal, therefore needing to be explained by randomly initialized slots, the model's 's ability to connect the provided hints to the correct objects decays, as can be observed by decrease in the mIoU score.
i
50
20
40
3
10
Max. Training objects
Max. Evaluation Objects
60
70
mlou [%)
mlou [9%]
40
60
20
Varying subset sizes for training: In the left two plots in Figure A.10, we train SAVi with various subsets of size E {1, 3, 6, 10} and evaluate on objects at test time. By training on sub- sets of objects, an unsupervised and a fully conditioned setting. As seen in the plots, when 'maximum training objects' is 1, that is 10 randomly initialized slots and conditioned slot are used, the model is not able to establish a correspondence between the conditioning hint and the corresponding object in the scene. In other words, it treats the conditioning hint similar to the randomly initialized slots. On the other hand, providing conditioning for objects at training time, that 'maximum training objects' is 10, leads to poor generalization to subsets at test time since the model was not exposed to unconditioned objects during training. Conditioning on up to 6 objects presents a reasonable trade off between these two extremes. Providing explicit supervision on the alignment of the hint and the responding object in the input (which we do not provide) is likely necessary for stabilizing this setup when only few hints provided. Alternatively, smarter initialization strategies for unconditioned slots can likely improve this setup, which we leave for future work.
50
SAVI
SAM
Soo. Prop.
Seg. Prop.
3
10
10 Max. Evaluation Objects
Max. Training objects
able to ablate between
Figure A.10: Subset selection experiments: We select a subset of objects at train and test time and measure tracking mloU on the MOVi dataset (first 6 frames only). Five seeds were used for each of these box-whisker plots.
Generalization to unseen objects and backgrounds To test for generlization capabilities, we evaluate a SAVi model with bounding box conditioning, which was trained MOVi++, the following evaluation splits (see Table A.
Unseen objects: This split only contains novel objects (approx. 100) not seen during training, which are also not part of the default evaluation split. Remarkably, SAVi generalizes without decrease in FG-ARI mloU scores this split.
25

---

Published as a conference paper at ICLR 2022
This split only contains novel backgrounds (approx. 40) not seen during training. but otherwise contains solely known objects. Generalization is comparable as to the default evaluation set.
Unseen objects:
Unseen objects & backgrounds: When evaluating with both novel objects and back- grounds, there is only a slight decrease in FG-ARI and mloU scores, i.e. the model still generalizes well.
MOVi: model trained MOVi++ also generalizes well to MOVi, here at resolution of 128 x 128 to be in line with the training setup on MOVi++.
Table A.I: Generalization results for SAVi + Bounding Box model trained on MOVi++. Evaluation on first 6 video frames. Mean and standard error (10 seeds). All values in %.
Evaluation split
FG-ARII
mloUt
MOVi++: Default
8..0
54.3
MOVi++: Unseen objects
8.1
0..4
MOVi++: Unseen backgrounds
8..2
54.0+0.3
MOVi++: Unseen objects & backgrounds
80.4
5..6+0.3
MOVi (trained on MOVi++)
83.7
53.7
Unconditional video decomposition on MOVi MOVi++ In Table A.2 we report results on unconditional video decomposition on the MOVi and MOVi++ datasets. For SIMONe, we use our own reimplementation
which we approximately reproduced the CATER results reported by Kabra et al. (2021). For SCALOR (Jiang et al., 2020), we use the implementation provided by the authors. Further baseline details are provided Section A.7.
Table A.2: Unconditional video decomposition on MOVi / MOVi++.
MOVi++ FG-ARI
MOVi
Model
FG-ARI
SCALOR
81.2
22.7 1.9
SIMONe
74.8
32.7
SAVi (uncond.)
78.2
47.6
Architecture ablations In this section, we report results on several architecture ablations on SAVi. In all cases, we evaluate on MOVi++ and condition SAVi on bounding box information in the first frame of the video. Results are reported in Table A.3. We consider the following ablations:
No predictor: Instead of Transformer- (Vaswani al., 2017) predictor, identity function. Modeling the dynamics of the objects is thus entirely offloaded to the corrector, which does not model interactions. We find that this can marginally increase FG-ARI scores, likely because the self-attention mechanism in the default model variant can have a regularizing effect. It, however, negatively affects the model's ability learn the correspondence between provided hints and objects in the video, as indicated by the drop in mloU scores, likely due to inability to exchange information between learned slot representations in the absence of the Transformer-basee predictor.
the
MLP predictor: In this setup, replace the predictor with MLP with residual con- nection and LayerNorm (Ba et al., 2016), applied independently on each slot with shared parameters. It has the same structure and number of parameters the MLP in the Trans- former block used in our default predictor. Similar to the no predictor ablation, this variant does not model interactions between slots. The effect on performance comparable to the no predictor ablation.
Inverted corrector attention: By default, we use Slot Attention (Locatello et al., 2020) as corrector. The attention matrix in Slot Attention is softmax-none normalized over the slots,
26

---

Published as a conference paper at ICLR 2022
which creates competition between slots so that two slots discouraged to attend to the same object or part (unless their representations are identical). In this ablation, we normalize the attention matrix instead over the visual input features, done in e.g. RIM (Goyal et al., 2021c). We observe a sharp drop in the model's is ability to learn the correspondence between hints and objects, indicated by the low mloU score in this setting.
Table A.3: SAVi ablations and model variants on MOVi++ with bounding box conditioning. Evalua- tion on first 6 video frames. Mean and standard error (10 seeds). All values in %.
Model variant
FG-ARI
mloUt
SAVi (default)
82.0
0.3
No predictor
44.7 8..
MIPP predictor
0.2
49.
Inverted corrector attention
78.3
1.
Comparison to semi-supervised training with matching While our approach of conditioning the model on set of hints, such as bounding boxes, in the first frame can be seen as a form of semi-supervised learning, SAVi is still solely trained with an unsupervised flow prediction (or reconstruction) objective. Instead of providing bounding boxes as conditional input to initialize the slots of SAVi, an interesting variant to instead initialize slots in an unconditioned way and provide the bounding boxes instead as supervision signal in the form of matching based loss during training. This variant receives the same information as our conditional setup, but now requires matching to associate slots bounding box labels. We train the model in the same way the supervised set prediction set up of Slot Attention using Hungarian matching and Huber loss as described by Locatello al. (2020) with labels provided only during the first frame of the video. Additionally we still decode optical flow predictions at every time step using the same loss as in our default model. We found that this setup requires more than a single iteration of Slot Attention to converge to sensible decomposition in the first frame where we apply matching. Results are summarized in Table A.4. We find that using matching as training signal results in significantly worse FG-ARI segmentation score at evaluation time compared to using conditioning both at training and at test time.
Semantic property readout To evaluate what degree slots of the SAVi model capture informa- tion about objects beyond their location and velocity, we train simple readout probe in the form of an MLP that is tasked to predict object properties given a learned slot representation in SAVi. For these experiments,
train SAVi with bounding box conditioning on MOVi and jointly train MLP readout head to predict object properties from each slot representation in the first frame. We do not propagate the gradients from the readout head through the rest of the model, i.e. the SAVi model does not receive any supervision information about object properties. The MLP readout head has a single hidden layer with 256 units and a ReLU activation. is tasked predict classification logits for (i) the object color out of 8 possible colors, (ii) the object shape out of 3 possible shapes, and (iii) the object material out of 2 possible materials. For each of the three classification targets we train with a cross- cross-entropy loss using all videos of the MOVi training set. For evaluation, we measure average accuracy where each classification target (color, shape, material) considered separately and the result is averaged over videos and classification targets.
In this setting, the readout probe achieves an accuracy of 54.6 1.5 (10 seeds, mean and standard error) on the MOVi evaluation set which is far above chance level, indicating that semantic information
Table A.4: SAVi on MOVi with bounding box conditioning vs. unconditional SAVi with bounding boxes provided as supervision during the first fame via matching- loss. Evaluation on first video frames. Mean and standard error (10 seeds). All values in %.
Model variant
FG-ARI
Conditional SAVi (default) SAVi + Learned init. + Matching loss
92.9
83.0+0.2
SAVi Gaussian init. Matching loss
82.3
27

---

Published as a conference paper at ICLR 2022
about objects is still to some degree retained in the slots even though our training objective (optical flow) is insensitive to color and material appearance. When training with RGB frame reconstruction instead, achieve significantly higher score of 85.7 7 =: u.7, suggesting that one could likely combine the benefits frame reconstruction and flow prediction by training model to jointly predict both targets, which is promising direction for future work.
A.5 DATASET DETAILS
The Kubric (Greff et al., 2021) dataset generation pipeline is publicly available under Apache 2.0 license. MOVi++ contains approx. 380 publicly available CC-0 licensed HDR backgrounds from https: / / / haven com/ The data does not contain personally identifiable information or offensive content.
The original CATER (Girdhar & Ramanan, 2019) dataset (without segmentation mask annotations) is publicly available under an Apache 2.0 license. The variant with segmentation masks was provided by the authors of SIMONe (Kabra et al., 2021).
A.6 ARCHITECTURE DETAILS AND HYPERPARAMETTER
All modules use shared parameters across time steps, except for the initializer, which is only applied the first time step.
Encoder Our encoder architecture is inspired by the one used in Slot Attention (Locatello et al., 2020). We summarize the architecture in Table A.5. We use the same linear position embedding as in Slot Attention, i.e. four-dimensional vector per spatial position encoding the coordinate along the four cardinal directions, normalized to [- 1]. We project it to the same size as the feature maps of the CNN encoder using learnable linear transformation. The projected position embedding is then added to the feature maps. For the modified SAVi (ResNet) model on MOVi++, we replace convolutional backbone (the first 4 convolutional layers) with a ResNet-34 (He et al., 2016) backbone. We use a modified ResNet root block without strides (i.c. 1 x stride), resulting in 16 16 feature maps after the backbone. We further use group normalization (Wu & He, 2018) throughout the ResNet backbone.
Table A.5: SAVi encoder architecture for MOVi and CATER with 64 x 64 input resolution. For MOVi++ with 128 128 input resolution, we use a larger input stride of 2 and 64 channels (numbers in parentheses).
Layer
Stride
#Channels
Activation
Conv 5 x 5
1 1 (2 2)
32 (64)
ReLU
Conv 5 x
1xL
32 (64)
ReLU
Conv 5 X 5
1x 1
32 (64)
ReLU
Conv 5 x 5
1
32 (64)
Position Embedding
32 (64)
Layer Norm
Conv 1 x
1x1
64
ReLU
Conv 1 x
1x1
64
Corrector For the Slot Attention (Locatello et al., 2020) corrector, we only use single iteration of the attention mechanism all experiments on MOVi and MOVi++, and iterations on CATER experiments. The query/key/value projection size is D 128. Before each projection, we apply Layer Norm (Ba al., 2016). The attention update followed by a GRU (Cho et al., 2014) that operates the slot feature size of 128 (unless otherwise mentioned). Locatello al. (2020) further describe placing an optional feedforward layer after the GRU, which we only use when using more than single iteration of Slot Attention: this MLP has single hidden layer of size 256.
Predictor
We use the default multi-head dot- produrr attention mechanism (MulltHHeadSelfAtn)) from Vaswani et al. (2017) part of our predictor, for which we use query/key/value projection
28

---

Published as a conference paper at ICLR 2022
size of 128 and total of attention heads. For the MLP, we use single hidden layer with 256 hidden units and ReLU activation. On CATER, we use the identity function as predictor, as objects do not interact in this dataset.
Decoder We use the same decoder architecture as in Slot Attention (Locatello et al., 2020), for which we specify the details in Table A. Each slot representation is first spatially broadcasted to an 8 x 8 grid (i.e., the same vector is repeated at each spatial location) and then augmented with position embeddings which are computed in the same way as in the encoder. We to arrive at representations with the same resolution as the input frame. Finally, the output of this CNN is processed by two projection heads in the form of x 1 convolutions, (1) read out the segmentation mask logits and (2) to read out the per-slot predicted optical flow (or reconstructed RGB image). The segmentation mask logits are normalized using a softmax (across slots) and then used to recombine the per-slot predictions into single image. The decoder uses shared parameters across slots and time steps. Note that the ground-truth optical flow computed w.r.t. the previous frame, using additional simulated frame before the first time step (which is only used for computation of ground-truth optical flow and later discarded).
small CNN
Table A.6: SAVi decoder architecture for MOVi and CATER with 64 x 64 resolution. For MOVi++ with 128 x 128 resolution, we use a larger output stride of 2 (numbers in parentheses).
Layer
Stride
#Channels
Activation
Spatial Broadcast 8 x 8
128
Position Embedding
128
Conv Transpose 5 x 5
2 x 2
64
ReLU
Conv Transpseee 5 5
2 x 2
64
ReLU
5 5
2 x 2
64
ReLU
Conv TonnTranseo 5 x 5
x (2 x 2)
64
Initializer To encode bounding boxes and center of mass coordinates we pass the coordinate vector for each bounding box or center of mass location through an MLP with single hidden layer of 256 hidden units, a Rel.U activation and an output size of 128. The MIP is applied with shared parameters on each bounding box (or center of mass location) independently. Bounding boxes are represented as [Mmin Tmin, 2/mxx -mmx]] coordinates and center of mass locations are represented as [v, x]. Any unconditioned slot is provided with [-1, -1, -, 1,,1--111,, 1] bounding box coordinates (l-1, -1] for center of mass conditioning).
When conditioning on segmentation masks, we use a CNN encoder applied independently on the binary segmentation mask for each conditioned foreground object. For each unconditioned slot, provide an empty mask filled with zeros as input to the CNN. The architecture is summarized in Table A.7.
Other hyperparameters As described in the main paper, we train for 100k steps with a batch size of 64 using the Adam optimizer (Kingma& & Ba, 2015) with learning rate of 2 10-4 and gradient clipping with maximum norm of 0.05. Like in previous work (Locatello et al., 2020), we use learning rate warmup and learning rate decay. We linearly warm up the learning rate for 2.5k steps and we use cosine annealing (Loshchilov & Hutter, 2017) to decay the learning rate to 0 throughout the course of training. Like prior work (Locatello et al., 2020), we use total of 11 slots in SAVi for our scenes which contain a maximum number of 10 objects. In the conditional setup, all unconditioned slots share the same initialization and hence effectively behave as one single slot, irrespective of their number. Our chosen hyperparamters are close to the ones used in prior work and we did not perform an extensive search. We chose learning rate and gradient clipping norm based on performance measured on separately generated instance of the MOVi++ dataset. We found that training with a substantially longer schedule (e.g. 1M steps) can still significantly improve results for SAVi, but we opted for shorter schedules to allow for easier and faster reproduction of our experiments. We only train the modified SAVi (ResNet) model on MOVi++ with a longer schedule of 1M steps, which we found to be especially beneficial for this significantly larger model.
29

---

Published as a conference paper at ICLR 2022
Table A.7: SAVi initializer architecture for segmentation mask conditioning signals. We use the same architecture for all datasets. After the initial convolutional backbone, we perform a spatial average, followed by Layer Norm (Ba et al., 2016) and small MLP, arrive the initial slot representations.
Layer
Stride
#Channels
Activation
Conv 5
2
32
ReLU
Conv 5 5
2 xx3 2
32
ReLU
Conv 5 x
2 x
32
ReLU
Conv 5 x
1 1
32
Position Embedding
32
Layer Norm
Conv 1 x
1
64
ReLU
Conv 1
1x1
64
Spatial Average
Layer Norm
Dense
256
ReLU
Dense
128
Training software and hardware We implement both SAVi and the T-VOS baseline in JAX (Brad- bury et al., 2018) using the Flax (Heek et al., 2020) neural network library. We train our models on TPU v3 hardware. We report training time GPU hardware in Section
A.7 BASELINE DETAILS
This baseline (Jiang et al., 2020) patches the image into d x grid and uses a discovery network to propose objects in each grid cell. Then based on a threshold 7 on the overlap of the proposed object with the propagated objects from the previous frames, the model rejects accepts the proposals. The image is reconstructed with a combination of discovered objects and a background decoder with an encoding bottleneck dimension of Dog-en- It is also important to have correct mean and variance of the scale and ratio for the proposed objects. We performed a syperr-prramete search over the mentioned parameters along with the priors and annealing schedules for discovery and for the proposal rejection. We found the foreground-bbacggruntt splitting behavior to be especially sensitive to hypee-parameters, frequently resulting in the reconstruction of the entire image including all (or some) foreground objects with the background decoder, opposed to the desired case where the background decoder only reconstructs the background of the scene. We also found that changing SCALOR's prior annealing schedule made the model highly unstable. We performed validation held out 1/10th of the training dataset, choosing the variants with best log likelihood and visual quality. All the hyper-parameters for the reported results are similar the original SCALOR, except we use a 4 x 4 grid, a rejection threshold of 0.2, glimpse size of 32, scale mean and variance 0.2 and 0.1, object height-width ratio mean and variance 1.0 and 0.5, and for MOVi we decrease the background encoding dimension from the default 10 to 2. We use the official implementation provided by the authors, Jiang al. (2020).
SCALOR
SIMONe SIMONe (Kabra al., 2021) is probabilistic video model which encodes frames using CNNs followed by a transformer (Vaswani et al., 2017) encoder. It subsequently produces per-frame embeddings and per-object embeddings by pooling latent representations over (per-frame) transformer tokens and over time, respectively. Each pixel (for each time step and for each object embedding) decoded independently using an MLP, conditioned on both time and object embedding (sampled from a learned posterior). The model is trained using a reconstruction loss in pixel space (in the form of mixture model likelihood) and regularization loss on the latent variables. SIMONe encodes all frames in a video in parallel, i.e. it differs from all other considered video baselines in that it is not auto-regressive and thus cannot generalize beyond the (fixed) clip length provided during training. We re-implement the SIMONe model in JAX (as there is no public implementation by the authors at the time of writing) with several simplifications: 1) we sample latent variables only once per latent variable and time step instead of re-sampling for every generated pixel, 2) we always decode all frames during training instead of sampling random sub-sets of frames, 3) we only train for 200k steps we found that performance did not significantly improve with longer schedule. We verified
30

---

Published as a conference paper at ICLR 2022
that our re- ce-implementation reaches comparable performance to the numbers reported by Kabra et al. (2021) on CATER. We otherwise utilize the same hyperparameters as reported by Kabra et al. (2021) for CATER in all our experiments, and re-size all video frames to 64 x 64 before providing them to the model.
Contrastive Random Walk (CRW) CRW (Jabri et al., 2020) is self-supervised technique that induces a contrastive loss between embeddings of patches in neighboring frames. These embeddings are effective for downstream tracking via label propagation. We used the open source implementation provided by the authors to train their model on our datasets, MOVi and MOVi++; and evaluated tracking performance using the Davis 2017 benchmark utilities (Caelles et al., 2019; Pont- et al., 2017) on the validation sets of MOVi and MOVi++ respectively. We resized all images to 256 x 256 for training and to 480 x 480 for evaluation. This matches the setup that CRW was originally trained and tested on (Kinetics at 256 x 256 and Davis at 480p). Label propagation is more accurate at high resolution. We tuned and set the following parameters for MOVi++: Edge-dropout rate 0.05, training temperature 0.01, evaluation temperature 0.05, frame sampling gap 1, number of training epochs 250 with a learning rate drop at 200 epochs. And for MOVi: Edge-dropout rate 0.05, training temperature 0.03, evaluation temperature 0.01, frame sampling gap 2, number of training epochs 40 with a learning rate drop at 30 epochs. Early stopping was necessary for MOVI as training longer hurt downstream performance. We used video clips consisting of 6 frames to train these models. All other hyper- parameters and the architecture, a ResNet-18 (He et al., 2016) encoder, were kept at their default values.
Transductive VOS Transductive VOS (T-VOS) (Zhang et al., 2020) propagates segmentation labels from the first frame into future frames, thus tracking objects and segmenting them. correlates per-pixel features of target frame with those of a history of previous frames, called reference. correlation weights are used to compute a convex combination of previous frame labels. This constitutes the prediction for each pixel in the target frame. Such predictions in turn get incorporated into the reference for subsequent label propagation. A simple motion prior is used to focus on neighboring regions when propagating labels. T-VOS uses references frames capture the past video context. For T-VOS, we use the official implementation by the authors, using an ImageNet pre-trained ResNet-50 (He et al., 2016) backbone excluding the last stage for better results. We upsample MOVi/MOVi++ frames to 480p resolution, as we found that T-VOS with a ResNet-50 backbone produces inferior results when applied on frames at the original resolution. Metrics were computed at the original resolution of the two datasets. We select the optimal T-VOS temperature hyperparameter (T for MOVi and T = 1.3 for MOVi++) based on evaluation performance.
Segmentation Propagation For a more direct comparison to SAVi, we apply T-VOS (Zhang et al., 2020) on a backbone that is pre- trained using the same data and optical flow supervision as SAVi. For this experiment, we use simplified re- re-implementation of T- in JAX, for which we use maximum of 9 reference frames of the immediate past. We do not make use of the sparse sampling method from (Zhang et al., 2020). This is similar to the "9 frames" column in Table 2 as reported by Zhang et al. (2020), but in addition also uses their motion prior. We trained self-supervised CNN backbone, in domain, using MOVi/MOVi++ frames. The self-supervision proxy task was that of single image optical flow prediction, inspired by prior work on predicting optical flow from static images (Mahendran al., 2019; Walker et al., 2015). This proxy task requires guessing object motion based on appearance, which in turn requires modeling the objects present in the scene.
In more detail, we regressed optical flow using fully convolutional network (Long et al., 2015) using the exact same loss function as SAVi. The image backbone was also identical to SAVi except for an extra 5 x 5 convolution layer to project features into 128 dimensions. This matches the dimensionality of slot vectors in SAVi making this a fairer comparison.
To pre-train the visual backbone, we use a batch size of 64 and we train for 100k steps using the Adam optimizer (Kingma & Ba, 2015) with a learning rate of 2 x 10-5, We use linear position embeddings in the same way as SAVi, adding them right before 1 x 1 convolution layer which regresses 3-dimensional RGB encoded optical flow. For evaluation of T-VOS,
their motion prior with oi = 2.3, d2 = 4.6, and temperature of 50 for MOVi and 200 for MOVi++. We use 7.2-normalized per-pixel embeddings in order make T-VOS less sensitive to temperature.
31

---

Published as a conference paper at ICLR 2022
A.8 METRIC DETAILS
For both FG-ARI and mloU, we skip the first frame during evaluation as we provide conditional information (e.g., bounding boxes) for in most cases.
FG-ARI For our experiments on CATER (unsupervised video decomposition) we follow prior work (Kabra al., 2021) and report FG-ARI for all frames.
Segmentation mloU This metric assumes strict alignment between ground truth and predictions. That is, if the first segment tracks ball then the first slot should track the same ball. No matching is used. Unlike ARI, this metric is sensitive slots tracking the object they were conditioned with. In frames were an object is missing, either due to occlusion or because has left the scene, mIoU = 1.0 if the object is missing in the prediction well, otherwise mloU 0.0. Thus even a single predicted pixel in such frames is penalized heavily. This setting corresponds to the Jaccard-Mean metric used the DAVIS-2017 object tracking challenge (Caelles al., 2019; Pont-Tuset al., 2017).
32


Reviews:
[{"review_id": "mn6C7-v8Ylc", "reviewer": "Reviewer_TdU6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like this paper overall, and believe that the two improvements would bring new knowledge to the object-centric learning community.\n\nStrengths\n1. Previous object-centric learning methods are mostly unsupervised. The visual results they generate on simple datasets like CATER are quite promising, but in fact, they often fail in outputting meaningful object segmentations. For instance, an object can be represented by two slots while we only want a single one. I think this can hardly be well-addressed under the pure unsupervised setting. To this end, this work shows giving a simple initialization of objects of interest leading to less ambiguous representations of object slots, and I think this is a promising direction for object-centric learning.\n2. Using optical flows as supervision makes it possible to perform object-centric learning on more realistic datasets. In such a dataset, the texture and color of objects are more diverse and the previous RGB reconstruction becomes an inappropriate pre-text task: We do not need to remember every detail of the objects' appearance in order to track them. Instead, predicting optical flows is an easier and more beneficial pre-text task.\n3. I really like the experimental setups and results. The extensive experiments successfully convince me about the effectiveness of the two improvements proposed.\n\nWeakness:\n1. I think the biggest limitation is that using optical flow as supervision cannot handle static objects. However, this is clearly discussed in the limitation section, and I agree that addressing this problem is out of the scope of this paper.\n\nMinor concerns:\n1. Section 2,  \"Inspired by ... ordinary differential equations\", missing references.\n2. In the legend of Figure 1, the text \"Gated Recurrent Units\" is not aligned.\n3. Section 2, the Encoder paragraph, the notation is not clear. What is N and D_{enc}?\n4. Section 2, the Slot Initialization paragraph, \"we set the conditional input to a fixed value indicating padding\": what does \"indicating padding\" mean here?\n\n\n", "summary_of_the_paper": "This paper considers the problem of learning object-centric representations from videos. Different from existing methods that mostly model this problem under a pure unsupervised setting by reconstructing video frames, this work introduces two improvements. The first is offering weak \"hints\" (can be seen as additional inputs) during training, which can avoid falling into a trivial solution like previous unsupervised approaches always did. The \"hints\" here can be in form of pixel-wise masks, bounding boxes, or even as simple as centers of mass.  The second improvement regards the supervision signals. Instead of reconstructing raw pixels, they propose to predict optical flows. In easy scenes the optical flows are actually similar to pixel-wise masks: values inside an object are prone to be consistent. Therefore the learning is eased, and then making it possible to train on more realistic datasets. Extensive experiments are conducted to validate the effectiveness of these two improvements. ", "summary_of_the_review": "This paper introduces two improvements upon existing object-centric learning methods: offering additional object-level \"hints\" as inputs and using motion cues (optical flows) as pre-text supervision. The idea is good and I believe will be beneficial to the object-centric learning community. The writing is good, easy to follow. Experimental results are sufficient to support their claims. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "j6RBAShz9QM", "reviewer": "Reviewer_o4c6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The weakly-supervised setting seems to be new and has not been used in the previous literacture, which adopts abstract hints of the first frame as input and achieves better performance.\n\n2. The idea to use optical flow as supervision signal seems to be new and has not been done before.\n\n3. The writing overall is easy to follow.", "summary_of_the_paper": "This paper learns the object-centric representation for videos by extending the previous static slot attention framework with two new considerations, 1. optical flow for temporal modeling and 2. using simple objects' location cues for better segmentation or tracking. Experiments are conducted on CATER, MOVi and MOVi++. It show SAVi has better performance than its baselines especially when using the objects' cues during training.", "summary_of_the_review": "The reviewer has some concerns on the novelty of the paper.\n\n1. The idea of using optical flow to help weakly-supervised or unsuperised object learning seems to be straight forward. Previous work like PSGNet[A] also adopts motions and depth as supervision signal and compare their performance with MoNet. It will be necessary to compare and disucss the PSGNet in the related work.\n\n2. The paper only compares with  Slot attention or MoNet in unsupervised object segmentation. Object-centric representation for video learning and reasoning has been widely studied in previous frameworks like [A, B, C, D]. Note that previous work like ALOE [B] and VRDP [C] has also been using Slot attention or MoNet for unsupervised video proposal segmentation. It will be interesting to replace the Slot attention or MoNet with the proposed segmentation model and see their performance on the CLEVRER[E] dataset. \n\n[A]. Bear D M, Fan C, Mrowca D, et al. Learning physical graph representations from visual scenes[J]. arXiv preprint arXiv:2006.12373, 2020.\n\n[B]. Attention over learned object embeddings enables complex visual reasoning. David Ding, Felix Hill, Adam Santoro, Malcolm Reynolds, Matt Botvinick, Arxiv, 2021.\n\n[C]. Chen Z, Mao J, Wu J, et al. Grounding physical concepts of objects and events through dynamic visual reasoning[J]. arXiv 2021.\n\n[D]. Ding M, Chen Z, Du T, et al. Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language[J]. arXiv 2021.\n\n[E]. A]. Yi K, Gan C, Li Y, et al. Clevrer: Collision events for video representation and reasoning[J], ICLR 2020.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XJz85lliglc", "reviewer": "Reviewer_1UV9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n- Adapting Slot Attention to produce coherent object-centric representations in video instead of static images is interesting and can lead to further development in the video-processing field. Moreover the idea is simple and sound.\n- I enjoy the experiments showing the transition of results/empirical findings from simple to more complicated scenarios and also from synthetic to more realistic ones. The detailed ablations (built incrementally) offer useful insight for what the current object-centric video models require in order to perform well on more challenging scenarios.\n- The experiments where the method is conditioned on a subset of objects (Appendix) set by the user are really nice and open a broad range of applications for the proposed method\n- Overall the paper is well written and easy to follow\n\n**Cons:**\n\n- When measuring FG-ARI for the frame-level methods (Slot Att and MONet in Table1) how is the matching between the consecutives frames computed? I agree with the authors that the low score could be due to the lack of temporal consistency, but I think that a basic matching algorithm such as Hungarian matching + pairwise similarity to reorder the slots in a more appropriate way could represent a better baseline.\n- It would be nice to see the results mentioned in the last paragraph of the Section 4.2 (Unsupervised settings on MOVi and MOVi++) in a table.\n- In the paper, it is mentioned that the comparison with original CRW or T-VOS is not entirely fair since they used different backbone and operate on higher-resolution frames. I agree with that. However, why did the authors decide to retrain those methods instead of using the better backbone/setup for their method? I agree that the results should improve in that situation, but it’s not necessarily a sure fact.", "summary_of_the_paper": "The paper proposes an object-centric method for video data, inspired from the slot-attention architecture. To enhance the learning (and to allow the users to specify the kind of entities that they are interested in), the slots are initialized from the first frame either a) random; b) with learnable initialisation; c) as the center of the bounding boxes; d) as bounding boxes or e) as segmentation maps.  The method uses a slot-attention mechanism to correct the slots from each frame, with query consisting in the current prediction of the slots, and keys and values consisting in some features extracted from the frame + GRU on each node, for temporal aggregation.  Then they apply a self-attention layer to propagate the information between slots. Finally, the model is optimise to predict either the optical flow or the rgb content.\n\nThe paper presents several ablations: using different types of conditioning, different levels of supervision (with or without optical flow or using estimated unsup optical flow instead of the real one); or by varying the amount of noise they inject in the initial positions of the slots (in the conditioned case).\n\nThe method is tested on CATER, MOVi and MOVi++ to quantify the quality of video decomposition and they also experiment with different types of generalization scenarios: various number of frames, new objects, new background, etc.", "summary_of_the_review": "Overall, I enjoy the idea presented in the paper and I consider the empirical part interesting enough to recommend the acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S3QVL3QUvFX", "reviewer": "Reviewer_r9zz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) Overall, this paper is well written, and the technical details are easy to follow. \n2) The main idea of learning object representations and physical dynamics from videos is interesting.\n3) I found the anecdotal evidence for segmenting and tracking corresponding parts of objects very interesting, opening the door for more hierarchical concepts of objects using self/semi-supervised approaches.\n\nWeaknesses:\n\n**Contributions.** [1] already proposed a Slot Attention model based on optical flow for segmenting a single object. Although the author’s method supports multi-object environments while [1] is not, this paper still did not evaluate the proposed approach on real-world data (unlike [1]), which is a concern to me. \n\t\nFew works have already shown that using learnable query vectors instead of Gaussian-initialized slots helps the slots to learn on a unique embedding. [1] shows that “learnable queries play a similar role as soft clustering, i.e., assigning each pixel to one of the motion groups.” [2] (and others) shows that 2D positional encoding and box query embeddings are essential for initializing the queries for other tasks. To summarize, these works share the same concept that initializing the slots with location or motion embeddings could play a significant role in different downstream tasks.\n\n[1] Self-supervised video object segmentation by motion grouping, ICCV 2021.\n\n[2] TubeR: Tube-Transformer for Action Detection.\n\n\n**Real-world Video Data.** As far as I see it, multi-object segmentation and tracking are essential for real-life datasets, and I am concerned that this approach could be relevant only for synthetic datasets. The main issue with the Slot attention model is the inability to capture natural texture/background and/or camera movement, and thus I believe this work will not be able to generalize in the real world.\n\n\n**Technical Novelty.** The proposed approach is heavily based on the Slot Attention [2] model. I could not find an apparent and exciting technical novelty that could be interesting for other domains or different tasks. I believe the extension for the video domain could reach the bar of the conference quality, but still, I expect the authors to bring something new to the table. \n\nCan the authors highlight 2-3 points of technical or architectural modeling that are unique and different from their perspective from [1] or [2]?\n\n[1] Self-supervised video object segmentation by motion grouping, ICCV 2021.\n\n[2] Object-centric learning with slot attention, NeurIPS 2020.\n\n\n**Relation to Prior Work.** There are some object-centric approaches that use object-centric representations for video understanding and might be worth considering citing them, such as:\n\n[*] Compositional Video Synthesis with Action Graphs, ICML 2021.\n\n[**] Spatio-Temporal Action Graph Network, ICCVW2019.\n\n\n\n\n\n", "summary_of_the_paper": "This paper introduces a sequential extension of Slot Attention to tackle the problem of unsupervised / weakly supervised multi-object segmentation and tracking in video data. The method demonstrates successful segmentation and tracking for synthetic video data on unsupervised object representation learning.\n", "summary_of_the_review": "My main concern is that I cannot see how the proposed approach can generalize to new domains and tasks. Overall, I like this paper and its contribution, but I think the authors should clarify their contributions and explain how the video slot attention could be leveraged in other domains and tasks. Otherwise, it feels like a perfect model for the CATER/ CLEVRER/ MOVi/ MOVi++ datasets, but the full potential is not entirely clear. \n\nI am open to the authors' feedback and other reviewers' opinions.\n\n\nAfter Rebuttal\n------------------\n\n\nAfter reading the authors' feedback and other reviewers' opinions, I would like to thank the authors for their rebuttal.\n\nThe rebuttal addresses most of my concerns. I am leaning towards acceptance of the paper since it maintains the high bar of the conference quality. I vote for 6.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new framework that can learn the object-centric representation for video. The authors did a good job during rebuttal and turned one slightly negative reviewer into positive ones. The final scores are 6,6,8,8. AC agrees that this work is very interesting and deserves to be published on ICLR. The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are also encouraged to make other necessary changes.
Decision: Accept (Poster)