SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos
Authors: Gamaleldin Fathy Elsayed
Aravindh Mahendran
Sjoerd van Steenkiste
Klaus Greff
Michael Curtis Mozer
Thomas Kipf
Venue: NeurIPS 2022 Accept (2022)
Keywords: Object-centric learning
Computer Vision
Segmentation
Video

Abstract: The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions. Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided. Slot-based models leveraging motion cues have recently shown great promise in learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos. In an effort to bridge this gap, we take inspiration from human development and hypothesize that information about scene geometry in the form of depth signals can facilitate object-centric learning. We introduce SAVi++, an object-centric video model which is trained to predict depth signals from a slot-based video representation. By further leveraging best practices for model scaling, we are able to train SAVi++ to segment complex dynamic scenes recorded with moving cameras, containing both static and moving objects of diverse appearance on naturalistic backgrounds, without the need for segmentation supervision. Finally, we demonstrate that by using sparse depth signals obtained from LiDAR, SAVi++ is able to learn emergent object segmentation and tracking from videos in the real-world Waymo Open dataset.

SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos
Gamaleldin F. Elsayed *t, Aravindh Mahendran Sjoerd van Steenkiste Klaus Greff, Michael C. Mozer & Thomas Kipf"
Google Research
Abstract
The visual world can be parsimoniously characterized in terms of distinct entities with sparse interactions. Discovering this compositional structure in dynamic visual scenes has proven challenging for end-to-end computer vision approaches unless explicit instance-level supervision is provided. Slot-based models leveraging motion cues have recently shown great promise learning to represent, segment, and track objects without direct supervision, but they still fail to scale to complex real-world multi-object videos. In an effort bridge this gap, we take inspiration from human development and hypothesize that information about scene geometry in the form of depth signals can facilitate object-centric learning. We introduce video model which is trained to predict depth signals
SAVi++, an object-centee from slot-based video representation. By further leveraging best practices for model scaling, we are able to train SAVi++ to segment complex dynamic scenes recorded with moving cameras, containing both static and moving objects of diverse appearance on naturalistic backgrounds, without the need for segmentation supervision. Finally, demonstrate that by using sparse depth signals obtained from LiDAR, SAVi++ is able to learn emergent object segmentation and tracking from videos the real-world Waymo Open dataset.
Project page: https: htt///lottsa//// wwwllllttateetiinnoo video. gi thub... io/ so/savi+//
Emergent segmentation
Introduction
The natural world consists of distinct entities oatities-people,, -people dogs, cars, trees, etc.- and its complexity emerges from the
Optional condatining signal
Init.
combined, mostly independent, actions of the entities. This compositional struc- ture must be appreciated predict fu- ture states of the world and to effect par- ticular outcomes. People have an in- trinsic understanding of objects: objects have spatiotemporal coherence, they in- teract when in close proximity, and they
1st f--frame bounding baxes
SAVi++
Figure 1: Emergent segmentation and tracking in SAVi++.
'Equal technical contribution. AAlphhhetical order. 'Correspondence to: gamaleldin@google com Author contributions: GFF, TK initiated and led the project. GFF, AM, SVS, TK developed the main model. GFE, AM. TK developed real-world driving data and model infrastructure. AM implemented data augmentation. GFE led ablation study. SVS led target signals analyses AM, SVS, KG, TK worked on baselines. KG, MCM provided advice at all stages and helped with project scoping. TK developed visualizations. GFE, MCM, TK worked on figure design. All authors wrote the paper.
metric design. GFE, AM, SVS, ran experiments.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

possess persistent, latent characteristics that determine their behavior over extended periods of time (25, 46]. Just as object-centric representations are critical human understanding, they have the potential in machine learning to greatly improve sample efficiency, robustness, visual reasoning, and interpretability of learning algorithms [15, 35]. For example, consider the challenge faced by an autonomous vehicle operating in diverse surroundings (Figure 1). Generalization across situations requires learning about recurring entities like cars, traffic lights, and pedestrians, and the rules that govern interactions among these entities.
In human brains, the ability to organize edges and surfaces into unitary. bounded, and persisting object representations develops through experience and/or maturation from infancy and without explicit instruction via a 'core system of object representation" [46], i.e.. form of cognitive inductive bias. In deep learning, such an inductive bias has been proposed in slot-based architectures which segregate knowledge about individual objects into nonoverlapping but interchangeable pools of neurons. The resulting representational modularity can facilitate causal reasoning and prediction for downstream tasks [15, 44].
A grand challenge in computer vision has been to discover the compositional structure of real-world dynamic visual scenes in an unsupervised fashion. By unsupervised, we mean no segmentation information is provided that specifies which pixels belong together as part of a single object. Initial efforts focused on single-frame, synthetic RGB images [13, 14, 36, 50], but extending this work to video and more complex scenes proved challenging. A key insight to further progress was the realization that a volor-intensity pixel array is not the only source of visual information readily available, at least not to human perceptual systems. The human perceptual system extracts motion and depth cues early in the processing stream [9-11, 20, 39]. These cues are correlated with object identities, and can therefore bootstrap the formation of object-centric representations [45].
The recently introduced Slot Attention for Video (SAVi) model [31] leveraged optical flow (frame- to-frame motion) as a prediction target to obtain object-centric representations of dynamic scenes involving complex 3D scanned objects and real-world backgrounds. However, motion prediction alone is insufficient to learn about the distinction between static objects and the background. Further, in real-world application domains such as self-driving cars, cameras themselves are subject to movement, which globally affects frame-to- to-frame motion as prediction signal in non-trivial ways.
In the present work, we describe enhanced slot-based model for video, referred as SAVi+ (Figure 2), which obtains qualitative improvements in object-centric representations by exploiting depth signals readily available from RGB-D cameras and LiDAR sensors. SAVi++ is the first slot- based, end-to-end trained model that successfully segments complex objects in naturalistic, real-world video sequences without using direct segmentation or tracking supervision.
summary of our contributions is as follows:
We introduce SAVi++; an object-centric slot-based video model that makes several key improve- ments SAVi [31] by utilizing depth prediction and by adopting best practices for model scaling in terms of architecture design and data augmentation.
On the multi multi-object video (MOVi) benchmark containing synthetic videos of high visual and dynamic complexity [16), we find that SAVi++ + is able to handle videos containing complex shapes and backgrounds, and large number of objects per scene. Improving on SAVi, our approach accommodates both static and dynamic objects and both static and moving cameras.
Finally, we demonstrate that SAVi++ trained with sparse depth signals obtained from LiDAR enables emergent object decomposition and tracking in real-world driving videos from the Waymo Open dataset [47].
2 Related work
Object-centric learning A growing body of research is addressing the problem of end-to-end learning of object-centric representations from raw perceptual data without direct supervision. Slot- based neural networks such as IODINE [14], MONet [4], and Slot Attention [36] rely on a factorized latent space and independent per-object decoders as inductive bias to enable object discovery in a simple auto-encoding setup. Architectures with stronger inductive biases using fixed object size, presence, or propagation priors have been explored in works such as SQAIR [33] and SCALOR [23], but generally these methods have faced challenges scaling to more complex real-world data when

---

relying on auto- aitoo-nnoding alone. Our work primarily builds on recent advances in object-centric generative models for video sequences [24, 31, 50, 52, 59]. Different from our approach, these methods have so far been unable to scale to complex real-world multi-object video data. An alternative class of methods using contrastive learning for object discovery [19, 30, 37, 56], most notably ViT [56] and ODIN [19]. has recently achieved some success in discovering semantic groupings in real-world images. However, neither Group ViT nor ODIN model dynamics and typically fail to separate semantically similar object instances in close proximity. In work, we follow generative approach, but instead of tasking the decoder to generate complex visual RGB pixel data, utilize depth information to bootstrap object-centric learning without direct supervision.
Object discovery in driving scenes A range of recent methods [1, 17, 49, 53] use a multi- stage pipeline of 1) obtaining pseudo ground truth (PGT) segmentation or detection labels via some heuristie, and 2) training a model in a supervised fashion on PGT labels. While this class of methods achieves some success in discovering and tracking objects in real-world driving scenes, crucially hinges on the quality of the PGT labels, requiring carefully engineered task task-specific heuristics to extract objects. Earlier methods solely use clustering heuristics to extract approximate segmentation masks directly from motion trajectories for moving objects [3, 40J. In our work, we instead demonstrate that object segmentation and tracking can emerge in an end-to-end setting on complex real-world data without relying on PGT label generation.
Cross-modal learning For self-supervised object-centric learning from visual data, range of target modalities and training signals have been explored in the literature. By using motion cues from optical flow as prediction targets, several recent methods [3] were able to overcome limitations of purely RGB pixel-level generative models, which frequently failed in the presence of complicated textures [27]. However, this advantage is primarily limited to discovery of moving objects. Utilizing depth targets from a simulator [2] or from sparse LiDAR [17. 49, 53], has been explored an effort to overcome these limitations. Different from prior works utilizing multi-stage pipelines and hand- crafted heuristics for extraction of pseudo-labels from LiDAR [17, 49, 53], we directly utilize the (sparse) depth signal as target and demonstrate that this can enable emergent object segmentation and tracking on real-world driving data without any additional regularizers or pseudo-labeling techniques.
Scaling strategies for vision models It is common practice to scale architectural capacity with dataset complexity and size, while making use of strong data augmentation when addressing various supervised computer vision tasks [7, 8, 18, 32]. Nonetheless, self self-supervised methods for end-to- end object discovery have primarily been relying on overly simplistic and low-capacity backbone architectures [14, 36, 59], likely due to the simplicity of datasets and tasks considered in prior work. By scaling object-centric methods to larger, visually more complex datasets, we find that utilizing stronger visual backbone architectures in combination with data augmentation- provide substantial benefits. For simplistic datasets with lower visual complexity (and same number of examples), found anecdotal evidence in preliminary experiments for the opposite effect: both architecture scaling and data augmentation can negatively affect object discovery performance, likely explaining why prior works have not explored these strategies.
Depth estimation Recent advances in supervised monocular depth estimation (see Ming et al. [38] for review) could be combined with our method in future work, for instance using ordinal regression losses [12], transformer architectures [43], or more complex instance wise decoder architectures [54].
3 Methods
We begin by providing brief introduction to Slot Attention for Video (SAVi), which is the starting point for our exploration. With SAVi++, we introduce several simple yet crucial improvements, which allow us to bridge the gap to complex real-world data. Our framework is summarized in Figure 2.
3.1 Background
Slot Attention for Video (or SAVi) is a recent state-of-the-ar architecture for learning object- ceneric representations from video with minimal supervision. We briefly highlight some of its key components below and refer the reader for complete details Kipf al. [3]].
SAVi can be viewed as an autoregressive encoder-decoder video model with structured latent state composed of K object slots. At a given time-step, an encoder first encodes the observed video frame
3

---

time
II--P
ENG
Depth prediction
sparse depth
time
Video frames
Augmented
N
SAM
encoder
Object slats
Slot decoder
Emergent segmentation
Figure 2: SAVi++ is an object-centric video model based on Slot Attention for Video [31], which encodes video into a set of temporally-come coopsssllycoosssteent latent variables (object slots). Input frames and prediction targets are augmented using random crop augmentations. Augmented frames are passed through the improved SAVi++ encoder and mapped onto object slots using an attention mecha- nism [36]. Slots are updated recurrently for each frame and subsequently decoded independently into a depth map and per-slot alpha masks. SAVi++ is trained using (sparse) depth targets, leading to emergence of camporally-consiiee object segmentation in the decoded alpha masks.
to yield high-level image features that are useful for learning about objects. This followed by Slot Attention [36] (the 'corrector'), which updates the slots using these features and encourages individual slots to specialize to different parts of the observation. The content of each slot is decoded separately using a decoder, which additionally outputs pixel-level alpha mask to indicate how the decoded values for each slot should be combined. Together, the mask and decoded slots determine the output of the model the current time-step from which a loss is computed, e.g., to train the model to predict frame-to- to-frame motion (optical flow) for this frame. Slots for the next time-step (for the corrector to update) are obtained by applying a predictor, which can model interactions between slots and learn about object dynamics to predict their future state.
In addition to optical flow prediction, SAVi introduces conditioning that helps reduce uncertainty about the part-whole division into objects by pointing the model to specific locations. Indeed, in the absence of a specific downstream task, scene decomposition can be ambiguous and providing additional information as a conditioning signal may help alleviate this. The conditioning takes place via the slot initializer, which initializes the slots used in the initial video frame. The initialization may be learned in an unconditional setting (i.e., learn the initial slot states) or obtained by conditioning the initial state on high-level cues such as bounding boxes of objects of interest in the first video frame. This direction of attention or input conditioning helped SAVi to succeed in decomposing more complex visual scenes.
3.2 SAVi+
As SAVirelics on optical flow prediction as its main training signal for object discovery, its application is primarily limited to settings where all objects in scene have independent motion. In addition, SAVi struggled to generalize to scenes with a moving camera, even though the optical flow field
encodes information about (static) scene geometry in this case.
Here, we identify two key directions for improving SAVi and bridging its capabilities to real-workd video data, while preserving its core foundation for learning object representations from video: (1) exploiting depth as a prediction signal, which is readily available many real-world settings, and (2) utilizing model scaling strategies in terms of encoder improvements and data augmentation, which, despite being commonly used for classic vision problems, are generally underutilized for object- centric learning. Our improved approach, called SAVi++, successfully segments complex objects in naturalistic, real-world video sequences without using direct segmentation or tracking supervision.
Exploiting depth information Training object-centric models solely using RGB image or video frame reconstruction proves challenging in the presence of complex visual textures. frequently leading to failure modes such as clustering by color or into object-agnostic spatial regions [14, 17]. In SAVi, optical flow was proposed as a prediction signal to mitigate this issue, while still operating on visual RGB inputs [31]. However, relying solely on optical flow as a prediction target for learning about objects has a clear disadvantage: static objects, which make up the vast majority of visual entities we encounter daily basis, are not captured in this signal unless the observer or the entire scene is in

---

Video
Flow
Depth
Video
Sparse depth
Complex texture
Moving objects
Moving camera
Aframes (train)
MOV-C
Static objects
MOV-D
MOVi-C /
/
X
x
234k
MOVi-D
/
/
x
234k
MOV-E
MOVi-E
/ /
/
234k
Waymo
159.6k
Open
(a) MOVi datasets.
(b) Waymo Open dataset.
(c) Dataset details and statistics.
Figure 3: We consider three synthetic Multi- Object Video (MOVi) datasets [16] and the large-scale real-world driving dataset Waymo Open [47]. All datasets contain complex textures and moving objects. The MOVi datasets increase in complexity from MOVi-C (moving objects only) over MOVi- D (+static objects) to MOVi-E (+ moving cameras). Waymo Open contains all these characterisics.
motion. As consequence, SAVi fails to represent objects that rest, and similarly struggles with scenes observed from a moving camera, as optical flow can prove challenging to model in this case.
Here, explore depth target signal, used in conjunction with flow even in isolation. Depth estimation has received little attention slot-based models, yet does not suffer from the limitation of optical flow in datasets with static objects and camera movement. We thus hypothesize that depth may greatly benefit obtaining emergent object decompositions of complex videos. In terms of practical applicability, note how depth is readily-available signal in many real-world settings thanks the prevalence of RGB-D cameras and LiDAR in settings like self-driving cars [47]. Even in the absence of depth sensing capabilities, this signal can be cheaply estimated from multi-camera systems [34].
In our implementation, we represent the depth signal in image space, which we encode using a log transformation log(1 + d), where d is the distance of pixel to the camera (see Figure 3a). This log-transform puts a stronger emphasis on close-by objects and-in early experiments we found this form of normalization crucial for reliably training object-centric models using depth targets. SAVi+ is then trained to minimize the squared difference between the decoder output and this target signal. In case of multiple available targets, such depth and flow, concatenate the target images along the channel dimension and predict them using an otherwise unchanged model.
For sparse targets such as depth obtained from LiDAR, we ignore any points in the image space for which no signal present the computation the loss. For LiDAR specifically, we obtain the x, y. coordinates of all the LiDAR points in the self-driving each of the points from the LiDAR sensor. We then use the camera and LiDAR calibration parameters to project the LiDAR point distances from the SDC domain to the camera frame. This projection represents a very sparse approximation of the ground-truth depth signal (Figure 3b).
(SDC) world and compute the distance of
Scaling strategies Visual complexity present in real-world videos necessitates a different class of encoders than those used for simple synthetic datasets. Inspired by successful visual backbone architectures for set-based supervised object detection models [6. 26], we use more capable encoder that utilizes the ResNet34 [18] architecture followed by a transformer encoder [51] (with layers, unless otherwise mentioned). To avoid computation of batch and/or temporal statistics, we replace the typical batch normalization in ResNet34 with group normalization [55]. We use a stride I convolution and use max-poot max-pooling in the ResNet root block. This results in overall backbone stride of 8 (as opposed 32), which was found to be important for retaining object dccomposition capabilities. Please see the appendix for further architectural details.
Drawing inspiration from training schemes commonly used for real-world vision models [48], we further apply Inception-ste -tyyle cropping as data augmentation. In particular, we randomly crop a region of each frame with aspect ratio E [0.75, 1.33) such that enough of the frame retained after cropping. The same crop is applied consistently across all frames and the resulting video is resized to the original resolution. Flow fields and depth maps are adjusted accordingly to keep them accurate and spatially aligned with the video frame.
5

---

4 Experiments
The goal of our experimental evaluation is twofold: 1) on synthetic video data of varying complexity we would like to analyze the potential advantages of utilizing a depth signal and model scaling strategies for learning emergent segmentation and tracking, and 2) would like to investigate whether these improvements enable bridging the gap to complex real-world video data.
Section 4.1 covers both qualitative and quantitative comparisons of SAVi++ against baselines on the synthetic MOVi datasets. In Section 4.2, we perform an ablation study on SAVi++. Finally, in Section 4.3 we demonstrate and analyze results for SAVi++ model applied to real-world driving videos from the Waymo Open [47] dataset.
Datasets As basis for our experiments, we use videos of different scene and camera complexities (Figure 3c). We use three synthetic Multi-Object Video (MOVi) datasets (Figure 3a) introduced in Kubric [16], which are created by simulating rigid body dynamics. We narrow our investigation to MOVi datasets with complex naturalistic backgrounds and 3D-scanned everyday objects (variants C. D. and E). MOVi-C is generated using static camera, and all objects (max. 10) are initialized to move independently. MOVi-D introduces more objects, some of which are dynamic (1-3) and the majority rests statically in the scene (10-20). Finally, MOVi-E introduces random, linear camera movement. Each video contains 24 frames sampled at 12 frames per second (fps).
We also train and evaluate SAVi++ in a real-world driving setting using the Waymo Open dataset (Figure 3b). Waymo Open is comprised of high resolution video data of 1280 1920 original resolution from multi-camera system collected by Waymo vehicles [47]. The dataset consists of 798 train and 202 validation scenes of 20s video each, sampled at 10 fps. We subsample the dataset at 5 fps both for training and validation. The dataset also includes LiDAR signals that we use to compute sparse depth maps as discussed in Section 3.
Training setup For all our experiments, unless stated otherwise, we resize frames to a height of 128 pixels while keeping the aspect ratio fixed, resulting in 128 128 resolution for MOVi datasets, and a resolution of 128 x 192 for Waymo Open. We train SAVi++ for 500k steps on Tensor Processing Unit (TPU) accelerators with batch size of 64 using Adam [29].
We train on randomly sampled sub-sequences of only frames using 24 slots for MOVi and 11 slots for Waymo Open. See appendix for further training details and hyperparameters.
4.1 SAVi++ improves object-centric learning on complex synthetic video data
We investigate whether the key changes introduced to SAVi [31], which constitute our improved SAVi++ model, allow us to overcome limitations of SAVi and address the most challenging synthetic multi-object video (MOVi) benchmarks introduced in Kubric [16].
Setup We train all models independently on each dataset variant. Both SAVi and SAVi++ are trained in a conditional setting where we initialize slots using ground-truth bounding box information in the first frame. We report the same segmentation metrics as in prior work, i.c. Foreground Adjusted Rand Index (FG-ARI) [21, 42] and Mean Intersection over Union (mloU). FG-ARI a permutation- invariant clustering similarity metric frequently used for evaluating scene decomposition quality. It compares discovered segmentation masks with ground -truthhttuthhhue masks while ignoring any pixels that belong to the background. It is sensitive to temporal consistency of masks, but insensitive to their ordering. The mIoU metric is a standard segmentation metric, here adapted for video as in [5]. We note that this implementation is sensitive to the correct ordering of masks, i.e. it also measures whether models used the conditioning signal (here, first-frame bounding boxes) correctly.
Baselines Besides comparing to SAVi [], the most representative prior method for the task we are interested in, we compare against a range of baselines aimed establishing the difficulty of the unsupervised, bounding nox-conditioned video object segmentation task: 1) bounding box copy (BBox copy) baseline, which simply repeats the first-frame boxes throughout the video, learned BBox propagation baseline that does not receive visual inputs, to test for easily exploitable biases in the datasets, 3) k-Means clustering baselines, that cluster the flow and/or depth signal across the video sequence (initialized using the ground-truth object centers in the first frame), and 4) label propagation baseline, that uses visual features to propagate the initial boxes (rendered as rectangular masks) across the video, based on Contrastive Random Walks (CRW) [22]. See appendix for further details.

---

Table 1: MOVi results in terms of mean score standard error seeds) from evaluating SAVi+ and baseline models on validation set video sequences of increased length (24 frames), *: we use the official implementation of CRW [22], which does not report FG-ARI.
mloU? (%)
FG-ARI (%)
Model
MOVi-C
MOVi-D
MOVi-E
MOVi-C
MOVi-D
MOVi-E
BBox copy
12.3
42.8
32.9
11.8
68.0
54.7
BBox propagation
2.1
26.7
24.1=111
9.6
24.9
1,9
K-Means (depth)
7.1
6.010.44
5.4
26.3
30.9
32.2 1
K-Means (flow)
1.5
7.410.4
6.0
26.5
3..
33.1 1.7.7
K-Means ddewh)
1..6
6.7
5.3+0.3
26.6+1.0
35.9 $
+4.8
CRW [22]
2..8
45.310.0
47.5
SAVi [31]
0.7
22.7
30.7
77.6
6..6
5
SAVi++ (ours)
45.2
48.3 1..3.....
47.1
81.9 1: 0.2
86.0 1.3
84.1 +: 0.9
Results Quantitative results can be seen in Table 1 and qualitative results on MOVi-E in Figure 4a. The BBox copy method serves as trivial baseline which a learningbbased approach should outperform. While the original SAVi model does so on MOVi-C, it clearly fails to model the more complex MOVi- D and -E datasets. The BBox copy baselines is ie-perhaps unsurprisingly--ttron strongest on MOVi-D, where most objects are static. SAVi+ outperforms this baseline on all datasets, indicating that it learns non-trin toivialiial segmentation and tracking capabílities. Indeed, this advantage does not solely come from fitting certain biases in the datasets, as learned BBox propagation baseline (using the same predictor as in SAVi++) that does not receive visual input, fails to generalize to unseen evaluation videos. It is worth noting that neither of the MOVi tasks can easily be solved by simply clustering the target signals, as the results for the k-Means baselines demonstrate.
Compared to CRW it can be seen how SAVi++ yields markedly better mloU on MOVi-C and D. while performance on MOVi-E is similar. Note that, unlike SAVi++, CRW is merely capable of propagating pixel-level annotations across frames in a video and does not by itself produce instance-level object segmentations or corresponding
reieetttpprssnnaationss that could be used for down-stream tasks. Finally, comparing SAVi++ and SAVi directly, we see that SAVi++ overcomes the primary limitations of SAVi on the harder MOVi-D and -E datasets, both quantatively (Table and qualitatively (Figure 4a), while also improving performance MOVi-C.
Discussion It is evident that small number of critical changes SAVi [31], namely utilizing depth targets, a stronger architecture, and data augmentation, can have dramatic consequences on the ability of this slot-based model to learn emergent object segmentation and tracking complex video sequences. The difference between SAVi++ and SAVi is especially evident for the more complex datasets in our study (e.g., improving the mloU score on MOVi-E from 30.7% 47.1% see also Figure 4a). These results demonstrate that SAVi++ is better suited for various data complexities in terms of object dynamics and camera movement, which are likely to exist in real-worid data.
4.2 Ablation study
In this section, report results of ablation study gauge the contribution of the different components of SAVi++. The three main ingredients of SAVi++ are 1) the use of depth as training target, 2) the extra capacity added to SAVi by including a transformer encoder, and 3) the use of data augmentation. Figure 4b shows a systematic ablation each of those components. Removing the transformer encoder reduces object segmentation quality. yet the degradation in performance is relatively limited. While data augmentation only has a mild effect on the simpler MOVi-C dataset, makes substantial difference on the more challenging datasets, MOVi-D and E. Finally, removing depth targets reduces performance further and particularly catastrophic on MOVi-E.
In fact, we find that training solely using depth targets without relying on predicting optical flow as well (see w/o Flow in Figure 4b) still allows the model to accurately segment and track objects, especially on the more complex MOVi-D and E datasets. This result is particularly strong on MOVi-E where jointly predicting optical flow presents a difficult task for scenes with camera movement. Further, training depth targets was very crucial obtain good performance on the most complex
7

---

Input frame
SAVi
SAV++
Ground truth
0.4
mlou
0.3
SAVi+
W/o Trans.
W/o Trans. & Aug.
W/o Trans. Aug. Depth
0.1
w/o
Depth
MOVI-C
MOVi-D
MOVI-E
(a) MOVi-E qualitative results.
(b) Ablation study
Figure 4: Left: Qualitative results of SAVi++ compared to SAVi [31] on the synthetic MOVi-E dataset with camera motion. Right: SAVi++ ablation study MOVi-C, D, and E. Bars reflect validation set mloU (mean standard error for 5 seeds). We ablate: 1) the transformer encoder (w/o Trans.). ). 2) data augmentation (w/o Trans. & Aug.), and 3) depth targets (w/o Trans. & Aug. & Depth). We further report results for training without flow, while only using depth targets (w/o Flow).
Input frame
SIMONe
SIMONe depth
SAV++ (unconditional)
SAV++ (conditional)
Figure 5: Qualitative segmentation comparison on the Waymo Open validation set. Naive application of SIMONe [24] baseline model to this dataset results in failure, while adapting SIMONe to predict (sparse) depth maps yields rough (but frequently misaligned) segmentation masks. SAVi++ generally produces highly accurate segmentation masks, while its unconditional results are promising. Here, we hide masks that occupy more than 1300 pixels on average per frame to ease interpretability.
synthetic data MOVi-E as demonstrated with the large drop in mloU when ablating depth and relying only on optical flow to train the model.
4.3 SAVi++ enables emergent segmentation on real-world driving data
In the previous section, we found that solely using depth as a training target can be sufficient to learn emergent object segmentation and tracking. This finding provides strong motivation for scaling this class of methods to real-world data, where the availability of optical flow relies on approximate and potentially inaccurate flow estimation methods, whereas depth can accurately measured using technologies like LiDAR. To investigate this possibility, we use the Waymo Open dataset [47], which includes videos obtained from cameras mounted on cars in various traffic environments.
Setup To obtain depth signal, we project 3D LiDAR points into the camera frame, resulting in very sparse depth image for each time step (see Figure 3b for examples). We exclude pixels that do not have valid LiDAR point when computing the L.2 loss in image space. We train +AVi++ with 11 slots on frames and evaluate the model on sequences of frames. Due to the absence of ground-truth segmentation labels in Waymo Open, we quantitatively measure performance compared to ground-truth bounding boxes using three metrics. The Center-of-Mas (CoM) distance measures the average Euclidean distance between the centroid of the predicted segmentation masks and the centers of the ground-truth bounding boxes. We report the centroid distance normalized by the maximum achievable distance in the video frame. Additionally, separately measure the fraction

---

Conditioning
10
25
Figure 6: Waymo Open qualitative results of SAVi++ (conditional) over long sequences.
of cases where any sort of segment is predicted when valid ground-truth box exists, denoted as bounding box recall (B. Recall). The Bounding Box mloU (B. mloU) is analog to mloU using predicted and ground-truth truth bounding boxes. The former are obtained by training readout MLP to predict bounding boxes from the slot representations. See appendix for further details.
Baselines We quantitatively compare to the subset of previous baselines that work with (sparse) depth. Further, we report qualitative results for SAVi++ in the unconditional setting, i.e. without providing first-frame bounding boxes to the model to initialize slots, and compare to SIMONe [24] as representative object-centric video model baseline from the literature.
Results Quantitative results can be seen in Table 2 and qualitative results in Figures 5-6. We find that SAVi++ markedly outperforms the BBox copy and propagation baselines, as well as the clustering baseline in terms of object tracking. Further, the bounding box recall is high indicating that valid objects are rarely ignored. The qualitative results in Figures 5-6 even better reflect the significance of SAVi++'s performance well as its potential utility for object-centric representation learning from real-world videos (for SAVi++ results divided per object category see appendix).
Our results using sparse depth targets suggest that SAVi++ does not need complete (i.e. dense) depth supervision. To investigate how accurate this signal needs to be, we explored the degree of sensitivity of SAVi++ noise in the depth signal. We trained SAVi++ with noisy depth targets by applying additive Gaussian noise to the ground-truth sparse LiDAR depth signals with standard deviations of 10cm, 20cm and 40em. We found that SAVi+ was able to retain its emergent tracking performance even at the highest considered noise scale of 40cm (see Table 3 in appendix).
We additionally experimented with removing the bounding box conditioning in SAVi++ in the initial frame. Removing this conditioning sig- nal and using a learned initialization together with simplified encoder also yielded good object decompositions (see SAVi++ (uncondi- tional) in Figure 5). Compared to using plain SIMONe [24], we observe that SAVi++ (uncon- ditional) performs markedly better. Interestingly, modifying the non-autoregressive SIMONe baseline similar to SAVi++ by predicting sparse depth instead of RGB also showed improve- ment in object emergence. This gives further evidence that using depth is suitable for learning object-centric representations from real videos. Quantiratively, SAVi++ achieves CoM distance of 6.9 + 0.5 while SIMONe (with depth loss) achieves 7.4 + 0.2 over sequence of 12 frames at test time, evaluated using Hungarian matching.
Table 2: Waymo Open results (mean + standard error in %, 3 seeds) from evaluating models on sequences of 10 frames. SAVi++ HR is a vari- ant trained on higher-resolution (256 x 384) video frames.
(%)
Model
CoMl
mloUt
Recall
BBox Copy
5.0
44.3
100
BBox Prop.
2.1
38.5
100
K-Means (depth)
13
SAVi (RGB)
2..5
7.9+08
95.8
SAVi (depth)
0.7
10.3
9..4
SAVi++
4.4=0.2
49.7
96.5
SAVi++ HR
3.9
51.9
96.2 0.4
Superviscd
1.1
67.6
We show qualitative results for longer sequences in Figure 6 and in video-format in the supplementary material. is worth noting that SAVi++ was only trained on frames and did not receive any tracking supervision. Interestingly, we find that objects are often consistently tracked until the moment they leave the scene. At this stage, slots are freed up again and tend to bind to previously unexplained or objects. This behaviour indicates that our reported tracking metrics an underestimation of the capabilities of the model, as such re-binding is not accounted for. It is, however, conceivable
'These baseline results are improved compared to an earlier version of the paper by using exactly the same depth target transformation for SAVi++.

---

that re-binding events could be identified post-hoc if one were to use the representations learned by SAVi++ for downstream tasks, which is an interesting avenue for future work.
4.4 Limitations
With SAVi++, we demonstrated the first proof of concept that an emergent object-centric decomposi- tion of real-world complex videos is possible with an end-to-end slot-based approach. Yet, there is still a lot of room for improvement.
Reliance on conditioning We focused our exploration on the conditional setup where we provided cues in the form of bounding boxes of objects in the first frame. Although the of such "object hints" may share some similarity to how human visual attention (and how humans parse a visual scene) can be directed via external signals (e.g., via gestures such as pointing), it ultimately limits the practical applicability of our approach. Preliminary results with unconditional SAVi++ suggest that this information may not be strictly necessary and could be removed in future research.
Reliance on ground-truth target signals In similar vein, the reliance of SAVi++ on ground- truth target signals for training is limitation that may affect its practical applicability. Fortunately. LiDAR sensors for depth estimation are readily available in many application domains (such as in robotics and self-driving), and there is also rich literature on monocular depth estimation. While estimated, depth (or flow) are expected to be noisier compared to the signals considered in our experiments, our experiment with "noisy depth" offers initial sign that this may not affect performance much.
Gap videos recorded in the wild is also important to point out that although Waymo Open offers a challenging real-world benchmark for learning about objects, its videos are relatively structured compared to real-world videos recorded "in the wild", and especially heavy on cars, roads, traffic signs, pedestrians, etc. Other datasets, such as DAVIS [41] or Kinetics [28] offer greater complexity in that regard and it is foresecable that further development of SAVi++ will be needed to truly support these. An example of this is that objects in Waymo Open usually do not re-appear, which is an aspect that is currently not explicitly modeled in SAVi++ (c.g. to ensure that the same object is re-captured by the same slot). More generally, there is substantial headroom to improve the modeling of disappearing and reappearing objects in future work, such as by explicitly modeling object presence [33], by explicitly attending to past latent states [58].
Gap to supervised approaches Finally, we note how both the conditional and the unconditional setting, the segmentation and tracking performance, though impressive given the minimal amount of supervision the model receives, still qualitatively lags behind supervised approaches. Improving on the temporal consistency of object tracks, especially in the unconditional setting, is another promising direction for future work.
5 Conclusion
We demonstrate that object tracking and segmentation can emerge from utilizing information about scene geometry in the form of depth signals in complex video data with slot-based neural architectures. We utilize a series of synthetic multi-object video benchmarks with increasing complexity to find a simple yet effective set of changes to an existing state-of- of-the-art object-centric video model (SAVi), allowing us to bridge the gap from synthetic to complex real-world driving videos.
Our work marks a first step towards building end-to-end trainable systems that Iearn to perceive the world in an object-centric, decomposed fashion without relying on detailed human supervision. While many open challenges remain, this result evidences that object-centric deep neural networks are not inherently limited to simple synthetic environments, and we are excited about the potential for this class of methods to radically reduce the need for human supervision in building scalable perceptual systems for the real world.
6 Acknowledgements
We would like to thank Ben Caine, Alex Bewley and Pei Sun for assistance with self-driving data. We are grateful to Jie Tan, Daniel Keysers, David Fleet, Matthias Minderer, Mehdi Sajjadi and Mario Lučié for general advice and feedback.
10

---

References
[1] Zhipeng Bao, Pavel Tokmakov, Allan Jahri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Discov- ering objects that can move. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2022.
[2] Daniel M Bear, Chaofei Fan, Damian Mrowca, Yunzhu Li, Seth Alter, Aran Nayebi, Jeremy Schwartz, Li Fei-Fei, Jiajun Wu, Joshua B Tenenbaum, et al. Learning physical graph representations from visual scenes. In Advances Neural Information Processing Systems, 2020.
[3] Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In European Conference on Computer Vision, pages 282-295. Springer, 2010.
[4] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised seene decomposition and representation. arXiv preprint arXiv: 1901.11390, 2019.
[5] Sergi Caelles, Jordi Pont-Tuset, Federico Perazzi, Alberto Montes, Kevis-Kokits Maninis, and Lue Van Gool. The 2019 DAVIS challenge on vos: Unsupervised multi-object segmentation. arXiv preprint 1wXiv:1905.00737720000 2019.
[6] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Europeun Conference Computer Vision, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruykn. End-to-end object detection with transformers. European Conference on Computer Vision, pages 213-229. Springer, 2020.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[9] J. Driver, P. McLeod, and Z. Dienes. Motion coherence and conjunction search: Implications for guided search theory. Perception and Psychophysics. 51:79-85, 1992.
[10] J. T. Enns and R. A. Rensink. Influence of scene-based properties on visual search. Science, 247:721-723, 1990.
[11] T. Enns and R. Rensink. Sensitivity to three-dimensional orientation visual search. Psychological Science, 1(5)3223-32,, 1990.
[12] Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In CVPR, June 2018.
[13] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems, pages 6691-6701, 2017.
[14] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In International Conference on Machine Learning, pages 2424-2433, 2019.
[15] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208, 2020.
[16] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo. Charles Herrmann, Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour. Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubrie: scalable dataset generator. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[17] Adam W Harley, Yiming Zuo, Jing Wen, Ayush Mangal, Shubhankar Potdar, Ritwick Chaudhry, and Katerina Fragkiadaki. Track, check, repeat: An EM approach to unsupervised tracking. In /EEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.

---

[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2016.
[19] Olivier J Hénaff,, Skanda Koppula, Evan Shelhamer, Daniel Zoran, Andrew Jaegle, Andrew Zisserman. João Carreira, and Relja Arandjelović. Object discovery and representation networks. arXiv preprint (iXiv:2003.08777.2 2022.
[20] Todd S Horowitz, Jeremy M Wolfe, Jennifer S DiMase, and Sarah B Klieger. Visual search for type of motion is based on simple motion primitives. Perception, 36(11):1624 1634, 2007.
[21] Lawrence Hubert and Phipps Arabie. Comparing partitions. Journal of Classification, (11):193-218, 1985.
[22] Allan Jabri, Andrew Owens, and Alexei A Efros. Space-time correspondence a contrastive random walk. In Advances in Neural Information Processing Systems, 2020.
(23] Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. SCALOR: Generative world models with scalable object representations. In International Conference on Learning Representations, 2020.
[24] Rishabh Kabra, Daniel Zoran, Loic Matthey Goker Erdogan, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P. Burgess. SIMONe: View-invariant, temporally-abstrrated object representations via unsupervised video decomposition. In Advances in Neural Information Processing Systems, 2021.
[25] Daniel Kahneman, Anne Treisman, and Brian Gibbs. The reviewing of object files: Object-specific integration of information. Cognitive psychology, 24(2):75--199 1992.
[26] Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Car- ion. MDETR- - Modulated Detection for End-to-End Multi-Modal Understanding. arXiv preprint arXiv:2104. 12763, 2021.
[27] Laurynas Karazija, Iro Laina. and Christian Rupprecht. ClevrTex: A texture-rich benchmark for unsuper- vised multi-object segmentation. In NeurlPS Truck on Datasets and Benchmurks, 2021.
[28] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.0699
2017.
[29] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[30] Thomas Kipf. Elise van der Pol. and Max Welling, Contrastive learning of structured world models. In International Conference on Learning Representations, 2020.
[31] Thomas Kipf, Gamaleldin F Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In International Conference on Learning Representations, 2022.
[32] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigeerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (BiT): General visual representation learning. In European Conference on Computer Vision, pages 451- Springer, 2020.
[33] Adam Kosiorek, Hyunjik Kim, Yee Whye Tch, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems, pages 8606-8616, 2018
[34] Hamid Laga, Laurent Valentin Jospin, Farid Boussaid, and Mohammed Bennamoun. A survey on deep learning techniques for stereo-based depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[35] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[36] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems, 2020.
[37] Sindy Löwe, Klaus Greff, Rico Jonschkowski, Alexey Dosovitskiy, and Thomas Kipf. Learning object- centric video models by contrasting sets. arXiv preprint arX2v:2011.10887, 2020.
12

---

[38] Yue Ming, Xuyang Meng, Chunxiao Fan, and Hui Yu. Deep learning for monocular depth estimation: A review. Neurocomputing, 2021.
[39] K. Nakayama and G. H. Silverman. Serial and parallel processing of visual feature conjunctions. Nature, 320:264-265, 1986.
[40] Peter Ochs and Thomas Brox. Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. In IEEE Conference on Computer Vision and Patter Recognition (CVPR), 2011.
[41] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine- Horung. benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pättern recognition, pages 724-732, 2016.
[42] William M Rand. Objective criteria for the evaluation of clustering methods. Journal of the American Stutistical Association, 66(336):846 84-000))466880 1971.
[43] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, October 2021.
[44] Bernhard Schölkopf. Francesco Locatello. Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5): 612-634, 2021.
[45] E. S. Spelke. Principles of object perception. Cognitive Science, 14:29-56, 1990.
[46] Elizabeth S Spelke and Katherine D Kinzler. Core knowledge. Developmental science, 10(1):89-96, 2007.
[47] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, al. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[48] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, pages 1-9, 2015.
[49] Hao Tian, Yuntao Chen, Jifeng Dai, Zhaoxiang Zhang, and Xizhou Zhu. Unsupervised object detection with lidar clues. IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2021.
[50] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In International Conference on Learning Representations, 2018.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017.
[52] Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsca Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model- -baed reinforeement learning. In Conference on Robot Learning, pages 1439-1456, 2020.
[53] Antonin Vobecky, David Hurych, Oriane Siméoni. Spyros Gidaris, Andrei Bursuc, Patrick Pérez, and Josef Sivic. Drive&segment: Unsupervised semantic segmentation of urban scenes via cross-modal distillation. arXiv preprint arXiv:2203. 160, 2022.
[54] Lijun Wang. Jianming Zhang, Oliver Wang. Zhe Lin, and Huchuan Lu. SDC-depth: Semantic divide-and- conquer network for monocular depth estimation. In CVPR, 2020.
[55] Yuxin Wu and Kaiming He. Group normalization. In European Conference Computer Vision (ECCV), pages 3-,9, 2018
[56] Jiarui Xu, Shalini De Mello, Sifci Liu, Wonmin Bycon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. arXiv preprint arXiv:2202. 1094, 2022.
[57] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proceedings of the IEEE International Conference on Computer Vision, 2021.
13

---

[58] Yi Zhou, Hui Zhang, Hana Lee. Shuyang Sun, Pingjun Li, Yangguang Zhu, ByungIn Yoo. Xiaojuan Qi. and Jae-Joon Han. Slot-vps: Object-centric representation learning for video panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093-3103, 2022.
[59] Daniel Zoran, Rishabh Kabra, Alexander Lerchner, Danilo Rezende. Parts: Unsupervised segmen- tation with slots, attention and independence maximization. In Proceedings of the IEEE International Conference on Computer Vision, pages 10439-10447, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Limitations section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Yes, see Societal Impact section appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] Instructions to reproduce experiments are included the main text and appendix sections. For our code release,
project website at http://llo-aattenioo-vdddd github.io/savit+// We only used publicly available datasets.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see experimental section and appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see experimental section and appendix.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We use publicly available data. For Waymo Open all faces and license plates are blurred and made unrecognizable by default.
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "y8LWgMxvfra", "reviewer": "Reviewer_FUXq", "summary": "The paper proposes a model to improve on top of the SAVI model. The key differences include: utilising image depth as a signal for supervision and adopting scaling techniques during training to further boost the performance. The model obtains superior results on both a challenging synthetic dataset (MOVi dataset) and also on real-world driving videos (Waymo Open dataset).\n", "questions": "- Is there any reason why no metrics are reported for the unconditional settings? There are a couple of qualitative comparisons in the paper but no quantitative ones (for e.g. to compare against unconditional SAVi and SIMONe ). I expect the performance to degrade a lot, but I still consider it important to know if this is a scenario where it is worth applying the proposed model.\n\n- Compared to the baselines, the model requires additional supervision (the depth). The author did a great job in explaining how this signal is often available in the real world. However, experimenting with signals coming from predicted depth rather than ground-truth one would be interesting to explore.\n\n- The results reported in Table 1 seem different from the ones reported in [16]. Is the setup in any way different from that one?\n", "limitations": "Since the slots are updated from one step to another, is the model able to “rediscover” an object if it leaves the scene and appears much later? Should we expect the model to attach that object to the original slot, or it will be attached to another random slot? Can this impact the final performance?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- With simple modifications, the model obtained clearly superior results compared to the SAVi model.\n- The ablation study is detailed and clearly highlights where the improvement comes from\n- Obtaining good performance on real-world video is a challenging, missing piece in video object-centric literature, and this paper brings good advancement in this direction.\n\n\nCons:\n- Even if the model produces strong results, the technical novelty is quite limited. Adding supervision from the depth signal and training techniques such as data augmentation were previously used on video data. However, I agree that clearly isolating the importance of adopting them by the object-centric community is a significant contribution.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dBZjrMSEQYy", "reviewer": "Reviewer_3HcR", "summary": "SAVi++ is a successor the SAVi, model, which learns to segment objects in video sequences without segment annotations. Both SAVi and SAVi++ take \"hints\" about where objects might be on the first frame (such as bounding boxes, though they can be trained without such hints.) They both reconstruct optical flow via a bottleneck of object slots.\n\nSAVi showed several key limitations, in particular its poor performance on complex, realistic (or real) datasets and a general inability to segment static objects. SAVi++ makes progress on both of these fronts by using an additional cue / reconstruction target: depth, which may be supplied in addition to or instead of optical flow. Using depth as an additional cue, SAVi++ can learn to segment complex and realistic scenes better than its predecessor, especially static objects. Using sparse depth signals alone from a real-world autonomous driving dataset, SAVi++ can segment many of the objects despite never receiving ground truth segment annotations.", "questions": "None", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper makes progress on a challenging problem -- unsupervised segmentation of objects in realistic or real-world videos. This field has been limited mainly to simple synthetic datasets, so pushing toward real scenes is an important contribution.\n\n2. Both key differences between SAVi++ and SAVi are well-motivated: depth is a useful cue for segmentation, and it makes sense that more expressive architectures (and data augmentations) would be important for scaling to real data.\n\nWeaknesses:\n\n1. The description of the literature on human grouping is a little misleading. It's not known whether perceptual grouping in humans is innate or not (L38). Some ability to group moving objects is present as early as two months, and possibly earlier -- it's nearly impossible to measure. What is clear is that grouping ability **matures** from only segmenting moving objects to being able to segment static objects as well. Whether this maturation is a learning process that depends on experience is unknown too, but that much at least seems plausible via a kind of unsupervised learning (and has been recently modeled to some degree in https://openaccess.thecvf.com/content/CVPR2022/html/Bao_Discovering_Objects_That_Can_Move_CVPR_2022_paper.html and https://arxiv.org/abs/2110.06562 and https://arxiv.org/abs/2205.08515.)\n\n2. SAVI's main limitation is that it can't be trained (in its best form) on raw video datasets that contain nothing but RGB frames: it substantially benefits from (and on challenging datasets, effectively requires) object \"hints\" on the first frame, and also estimates of optical flow. The latter are pretty easy to come by using pretrained models, but many large-scale video datasets do not have bounding boxes for all objects. Moreover, the authors' motivation for requiring these hints (L141-148) is pretty speculative -- I don't know of evidence that infants require people to point at objects in order to group them. Even if there were such evidence, it wouldn't answer the question of how an infant converts the pointing into something like spatial localization of objects. so the initial hints are a practical limitation, and it's likely that humans can get by without them.\n\nSAVI++ seems to still rely on these hints. There are some qualitative results of models trained without hints on the driving dataset, but I don't see quantification on any of them. It's important to know if the hints are still critical for SAVI++'s success -- it is a valuable contribution either way, but doing away with the need for those hints is a challenging problem in its own right.\n\n3. The driving dataset looks like it's very heavy on cars. Being able to segment real scenes is a huge improvement over most earlier unsupervised models, but as far as video datasets go this one probably doesn't have the diversity of objects that others do (e.g. the DAVIS datasets or Kinetics.) The paper would be much stronger if SAVi++ could be shown to work on one of these more diverse video datasets. Short of that, could the authors provide some evaluation of SAVI++'s performance broken down by category on the driving dataset?\n\n4. The reliance on depth signals restricts the use cases of SAVi++ more than SAVi. With optical flow, there are supervised and unsupervised models (e.g. RAFT and SMURF) that transfer well to real-world video datasets, so reliance on flow isn't much of a limitation. But is that true for depth? If the authors could show that sparse GT depth could be replaced with estimated depth (e.g. from stereo cameras, or a transfer from a pretrained monocular depth model) it would substantially relieve this restriction.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Sbjqw-H1l6", "reviewer": "Reviewer_GFoH", "summary": "Brief Summary: The paper proposes an extension of a previous work SAVI [30] which used optical flows as a training signal to further include depth maps obtained from RGB-D cameras / LiDAR. The main goal is to learn from videos which have depth-channels but no instance/semantic segmentation ground-truths. \n\nExperiments on MOVi dataset types C,D, and E which are created synthetically, as well as on Waymo dataset obtained from real world data, show their proposed method SAVI++ outperforms compared baselines.", "questions": "Q1. How important is the high resolution in waymo dataset? \n\nQ2. What is the main bottleneck in training time, given that the underlying datasets are not very large? Is it sequential time-steps for filling in the object slots?\n\nQ3. In L100, it is noted that no PGT data is needed. Does using PGT data give improvements, and if so how much?", "limitations": "Yes. Some more discussions on where the models are likely to fail such as low-data domain, as evidenced by L25 of suppl, could be useful. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n\n1. With ever increasing 3d input data information, it is imperative that models are able to utilize such information for semantic segmentation tasks. Further, 3d data enables learning about static objects. As such, there is a clear motivation for the problem at hand.\n\n2. The paper provides good visualizations such as Fig3, Fig5, as well those in supplementary material.\n\nCons:\n\n1. In my opinion, the paper makes too general claims without providing previous citations while their actual technical contributions and evidence is far more limited. For instance, L48 suggests extending previous work is more challenging but doesn't explain why. L63 suggests it is the first model for segmentation without direct supervision or tracking supervision. This is strictly not true, given literature in domain adaption where one trains on synthetic segmentation dataset (such as Synthia or GTA) and domain adaptation transfer to Cityscapes (see [Ref1] for an example). Similar assertion is made in L3, but this ignores literature in video-text pretraining (see [Ref2] as example). \n\n2. On novelty: the technical novelty in the paper is slightly incremental. given that it mainly suggests using 3d-depth map instead of flow maps used in previous work. \n\n3. On Model architecture: \n\n(i) The reason for using Resnet34 is unclear (L193). What metric is being used to decide this (\"more capable encoder\" is too vague). \n\n(ii) Why not use something like Vision-Transformer instead, and remove the additional transformer encoder? Design-wise that looks much simpler?\n\n4. On Experiments:\n\n(i) MOVI is synthetic dataset, and Waymo dataset appears to be quite small with only 200 validation videos. More datasets should be tried such as Cityscapes, Cityscapes-VPS, VIPER (see [Ref3]).\n\n(ii) The proposed method performs significantly worse on MOVI-A,B (as shown in Table 1 of supplementary material).  If the reason is indeed due to overfitting on simpler domains as claimed in suppl L26, there should be some experiment with same encoder.\n\n(iii) The compared baselines in Table 1. are quite weak. The authors should compare their model with other methods trained on depth maps, at least some naive extensions of other previous works to use 3d-maps (such as for ODIN [19]).\n\n(iv) The authors should show fine-tune performance on other segmentation tasks such as coco and pascal. \n\n(v) Some design decisions on the choice of visual encoders should be empirically justified. \n\n[Ref1]: Hoyer, Lukas, Dengxin Dai, and Luc Van Gool. \"HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation.\" arXiv preprint arXiv:2204.13132 (2022). \n\n[Ref2]: Miech, A., Alayrac, J. B., Smaira, L., Laptev, I., Sivic, J., & Zisserman, A. (2020). End-to-end learning of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9879-9889).\n\n[Ref3]: Kim, Dahun, Sanghyun Woo, Joon-Young Lee, and In So Kweon. \"Video panoptic segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9859-9868. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OoSNH64uIB6", "reviewer": "Reviewer_uKMC", "summary": "Learning the compositional structure (e.g. scene graph) in dynamic visual scenes without labels (e.g. object masks) has proven challenging.  Slot-based models leveraging motion cues have recently made progress in learning to represent, segment, and track objects without direct supervision. However, these methods only work on synthetic data and fails on complex real-world multi-object videos. This paper proposes to exploit depth signals to improve object-centric learning. It introduces SAVi++, an object-centric video model which is trained to predict depth signals from a slot-based video representation. The paper claims that by using sparse depth signals obtained from LiDAR, SAVi++ is able to learn emergent object segmentation and tracking from videos in the real-world Waymo Open dataset.", "questions": "For Waymo dataset, the authors should compare with SOTA self-supervised instance segmentation methods, e.g. SAVi and GroupViT. The paper should also mention the performance of SOTA supervised methods to clarify on the gaps between these two type of methods. \n\nCan SAVi++'s depth prediction module benefit from the large body of monocular depth estimation paper? \n\nIt seems that the feature learning can use a strong baseline, e.g.  Self-Supervised Representation Learning from Flow Equivariance, ICCV 2021.", "limitations": "The paper does not seem to have stated its limitations. The work has several major limitations, e.g. relying on depth information, far from being competitive with SOTA supervised models, and lacking experimental results on real-world videos. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nSAVi++ improves on SAVi with predicting depth signals, and utilizing stronger visual backbone architectures in combination with data augmentation.\n\nSAVi++ evaluates on Waymo Open dataset.\n\nWeaknesses\n\nDespite the claims \"End-to-End Object-Centric Learning from Real-World Videos\", SAVi++ has very limited results on only one real world video dataset, Waymo. The results on Table 2 do not compare with any significant prior work. For more convincing experimental results, the authors are encouraged to consider indoor datasets as well, SUN RGB-D.\n\nThe paper ignores a large body of work on monocular depth estimation, e.g. Monocular Depth Estimation Using Relative Depth Maps, CVPR, 2019. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept