SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Metrics We evaluate our approach in terms of the quality of the discovered slot masks (output by the decoder), using two metrics: video foreground ARI (FG-ARI) [2] and video mean best overlap (mBO) [49]. FG-ARI is a video version of a widely used metric in the object-centric literature that measures the similarity of the discovered objects masks to ground truth masks. This metric mainly measures how well objects are split. mBO assesses the correspondence of the predicted and the ground truth masks using the intersection-over-union (IoU) measure. In particular, each ground truth mask is matched to the predicted mask with the highest IoU, and the average IoU is then computed across all assigned pairs. Unlike FG-ARI, mBO also considers background pixels, and provides a measure of how accurately the masks fit the objects. Both metrics also consider the consistency of the assigned object masks over the whole video.
In addition, we also use image-based versions of those metrics (Image FG-ARI and Image mBO, computed on individual frames) for comparing with image-based methods.
Baselines We compare our method with two recently proposed methods for unsupervised object-centric learning for videos: SAVi [5] and STEVE [7]. SAVi uses a mixture-based decoder and is trained with image reconstruction. We use the unconditional version of SAVi. STEVE uses a transformer decoder and is trained by reconstructing discrete codes of a dVAE [50]. Similar to Seitzer et al. [9], we also add a regular block pattern baseline, corresponding to splitting the video into regular block masks of similar size that do not change over time. By showing the metric values for a trivial decomposition of the video, this baseline is useful to contextualize the results of the other methods. In addition to video-based methods, we compare our model to image-based methods, including DINOSAUR [9], LSD [36] and Slot Diffusion [37], showing that our approach performs well in both object separation and mask sharpness. Last, we also compare our model to two concurrent works discovering objects from real-world video, SMTC [41] and SOLV [42].
4.2 Comparison with State-of-the-Art Object-Centric Learning Methods
When comparing VideoSAUR to STEVE and SAVi, it is evident that VideoSAUR outperforms the baselines by a significant margin, both in terms of FG-ARI and mBO (see Table 1 and Fig. 4). On the most challenging synthetic dataset (MOVi-E), VideoSAUR reaches 73.9 FG-ARI. Notably, for the challenging YT-VIS 2021 dataset, both baselines perform comparable or worse than the block pattern baseline in terms of FG-ARI, showing that previous methods struggle to decompose real-world videos into consistent objects. We additionally compare VideoSAUR to image-based methods in App. A.1, including strong recent methods (LSD, SlotDiffusion and DINOSAUR), and find that our approach also outperforms the prior image-based SoTA. Finally, in App. A.2, we find that our method performs competitively with concurrent work.
Next, we report how well our method performs in terms of zero-shot transfer to other datasets to show that the learned object discovery does generalize to unseen data. In particular, we train VideoSAUR
Figure 5: Zero-shot transfer of learned object-centric representations on YT-VIS 2021 to the YT-VIS 2019 and DAVIS datasets for different number of slots.
on the YT-VIS 2021 dataset and evaluate it on the YT-VIS 2019 and DAVIS datasets. YT-VIS 2019 has similar object categories, but a smaller number of objects per image. The DAVIS dataset consists of videos from a fully different distribution than YT-VIS 2021. As the number of slots can be changed during evaluation, we test VideoSAUR with different number of slots, revealing that the optimal number of slots is indeed smaller for these datasets. We find that our method achieves a performance of  $41.3 \pm 0.9$  mBO on YT-VIS 2019 dataset and  $34.0 \pm 0.4$  mBO on DAVIS dataset (see Fig. 5), illustrating its capability to effectively transfer the learned representations to previously unseen data with different object categories and numbers of objects.
Long-term Video Consistency In addition to studying how VideoSAUR performs on relatively short 6-frame video segments from YT-VIS, we also evaluate our method on longer videos. In App. B.1, we show the performance for 12-frame and full YT-VIS videos. While, as can be expected, performance on longer video segments is smaller in terms of FG-ARI, we show that the gap between VideoSAUR and the baselines is large, indicating that VideoSAUR can track the main objects in videos over longer time intervals. Closing the gap between short-term and long-term consistency using memory modules [24, 51] is an interesting future direction that could be useful for video prediction [52] as well as for object-centric goal-based [53, 54] and model-based [55] reinforcement learning.
In this section, we analyze various aspects of our approach, including the importance of the similarity loss, the impact of hyperparameters (time-shift  $k$  and softmax temperature  $\tau$ ), and the effect of the choice of self-supervised features and decoder.
Choice of Loss Function (Table 2 and Table 3) We conduct an ablation study to demonstrate the importance of the proposed temporal similarity loss, comparing and combining it with the feature reconstruction loss [9]. We also consider predicting the features of the next frame (see App. C.4 for implementation details). For all datasets, feature reconstruction alone performs significantly worse than the combination of feature reconstruction and temporal similarity loss. Predicting the features of the next frame in addition to feature reconstruction also yields improved performance, but is worse than the temporal similarity, suggesting that the success of our loss can be partially explained by the integration of temporal information through future prediction. Interestingly, on MOVi-C, using the temporal similarity loss alone significantly improves the performance over feature reconstruction  $(+20$  FG-ARI,  $+7$  mBO). To provide insight into the qualitative differences between the losses, we analyze the videos with the most significant differences in FG-ARI (see Fig. E.4): unlike feature reconstruction, the temporal similarity loss does not fragment the background or large objects into numerous slots, and it exhibits improved object-tracking capabilities even when object size changes. To gain further insights, we also consider (ground truth) optical flow as a prediction target that only captures motion, but no semantic information (see App. B.2 for a detailed discussion). We find that only predicting optical flow is not enough for a successful scene decomposition, underscoring the importance of integrating both motion and semantic information for real-world object discovery.
Robustness to Camera Motion (Table 4) Next, we investigate if VideoSAUR training with the similarity loss is robust to camera motion, as such motion makes isolating the object motion more difficult. As a controlled experiment, we compare between MOVi-D (without camera motion) and
Table 2: Loss ablation on MOVi-C.
Loss Type	Metric
Feat. Rec.	Next Frame
Feat. Pred.	Temp. Sim.	FG-ARI	mBO
✓	40.2	23.5
✓	✓	47.2	24.7
✓	60.8	30.5
✓	✓	60.7	30.3
Table 3: Loss ablation on YT-VIS.
Loss Type	Metric
Feat. Rec.	Next Frame
Feat. Pred.	Temp. Sim.	FG-ARI	mBO
✓	35.4	26.7
✓	✓	37.9	27.3
✓	26.2	29.1
✓	✓	39.5	29.1
Table 4: Robustness to introducing camera motion (MOVi-D  $\rightarrow$  MOVi-E).
MOVi-D	MOVi-E
SAVi (optical flow) [12]	19.4	2.7
VideoSAUR (temporal sim.)	55.7	62.5
Table 5: Decoder comparison on MOVi-C and YT-VIS.
MOVi-C	YT-VIS	Memory
FG-ARI	mBO	FG-ARI	mBO	GB @24 slots
Mixer	60.8	30.5	39.5	29.1	24
MLP	64.2	27.2	39.0	29.1	70
MOVi-E (with camera motion), and train VideoSAUR using only the temporal similarity loss. We contrast with SAVi trained with optical flow prediction<sup>3</sup>, and find that VideoSAUR is more robust to camera motion, performing better on the MOVi-E dataset than on the MOVi-D dataset (+6.8 vs -16.7 FG-ARI for SAVi).
Choice of Decoder (Table 5) We analyze how our method performs with different decoders and find that both the MLP broadcast decoder [9] and our proposed SlotMixer decoder can be used for optimizing the temporal similarity loss. VideoSAUR with the MLP broadcast decoder achieves similar performance on YT-VIS and MOVi datasets, but requires 2-3 times more GPU memory (see App. C.1 for more details and Table B.3 for the detailed comparison of decoders on MOVI-E dataset). Thus, we suggest to use the SlotMixer decoder for efficient video processing.
Softmax Temperature (Figure 6a) We train VideoSAUR with DINO S/16 features using different softmax temperatures  $\tau$ . We find that there is a sweet spot in terms of grouping performance at  $\tau = 0.075$ . Lower and higher temperatures lead to high variance across seeds, potentially because there is not enough training signal with very peaked (low  $\tau$ ) and diffuse (high  $\tau$ ) target distributions.
Target Time-shift (Figure 6b) We train VideoSAUR with DINO S/16 features using different time-shifts  $k$  to construct the affinity matrix  $A_{t,t+k}$ . On both synthetic and real-world datasets,  $k = 1$  generally performs best. Interestingly, we find that for  $k = 0$ , performance drops, indicating that predicting pure self-similarities is not a sufficient task for discovering objects on its own.
Choices for Self-Supervised Features (Figures 6c and 6d) We study two questions about the usage of the ViT features: which ViT features (queries/keys/values/outputs) should be used for the temporal similarity loss? Do different self-supervised representations result in different performance? In Fig. 6c, we observe that using DINO "key" and "query" features leads to significantly larger mBO, while for FG-ARI "query" is worse and the other features are similar. Potentially, this is because keys are used in the ViT's self-attention and thus could be particularly good to compare with the scalar product similarity. Consequently, VideoSAUR uses "key" features in all other experiments. Moreover, we study if the temporal similarity loss is compatible with different self-supervised representations. In Fig. 6d, we show that VideoSAUR works well with 4 different types of representations, with MSN [39] and DINO [10] performing slightly better than MAE [11] and MOCO-v3 [40]. We also demonstrate that further fine-tuning the DINO features utilizing a self-supervised temporal-alignment clustering approach named TimeTuning [56] on unlabeled videos enhances the mask quality of VideoSAUR.
Pre-training Dataset (Table 6) All self-supervised methods we utilize are trained on the ImageNet dataset, which a) has a strong bias towards object-centricness as its images mostly contain single objects, and b) introduces a large number of additional images external to the dataset we are training
3SAVi results with optical flow are from Greff et al. [12].
(a) Softmax temperature  $\tau$
(b) Target time-shift  $k$  on MOVi-C and YT-VIS datasets.
(c) DINO ViT features.
(d) Self-supervised representation method.
Figure 6: Studying the effect of different parameters of the temporal similarity loss.
Table 6: Comparing VideoSAUR with features trained on MOVi-E (MAE+MOVi-E) to features trained on ImageNet (MAE+ImageNet). For MAE+MOVi-E, we pre-train a ViT-B/16 using the self-supervised MAE method on MOVi-E for 200 epochs. VideoSAUR is able to perform high-quality object discovery even without access to any external data.
MOVi-C	MOVi-E
FG-ARI	mBO	FG-ARI	mBO
VideoSAUR w/ MAE+ImageNet features	58.0	30.4	72.8	27.1
VideoSAUR w/ MAE+MOVi-E features	59.8	27.5	70.6	23.3
VideoSAUR on. An interesting question is whether a) and b) are actually required for the success of our method. To answer it, we train a ViT-B/16 encoder from scratch on the MOVi-E dataset using the MAE method, and then train VideoSAUR using the obtained features. Interestingly, we find that the features from MOVi-E yield similar results compared to ImageNet-trained features (although with slight drops in mask quality), demonstrating that VideoSAUR is able to perform high-quality object discovery even without access to external data. This result also has broader implications as it potentially increases the applicability of feature reconstruction-based object-centric methods to datasets fully out of the domain of ImageNet. It also raises a follow-up question: what properties of the pre-training dataset (and method) are important to obtain good target features for object discovery?
This paper presents the first method for unsupervised video-based object-centric learning that scales to diverse, unconstrained real-world datasets such as YouTube-VIS. By leveraging dense self-supervised features and extracting motion information with temporal similarity loss, we demonstrate superior performance on both synthetic and real-world video datasets. We hope our new loss function can inspire the design of further self-supervised losses for object-centric learning, especially in the video domain where natural self-supervision is available.
Still, our method does not come without limitations: in longer videos with occlusions, slots can get reassigned to different objects or the background (see Fig. B.5 for visualizations of failure cases). VideoSAUR also inherits a limitation of all slot attention-based method, namely that the number of slots is static and needs to be chosen a priori. Similar to DINOSAUR [9], the quality of the object masks is restricted by the patch-based nature of the decoder. Finally, while the datasets we use in this work are significantly less constrained compared to datasets used by prior work, they still do not capture the full open-world setting that object-centric learning aspires to solve. Overcoming these limitations is a great direction for future work.
We thank Martin Butz, Cansu Sancaktar and Manuel Traub for useful suggestions and discussions. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Maximilian Seitzer. Andrii Zadaianchuk is supported by the Max Planck ETH Center for Learning Systems. Georg Martius is a member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany's Excellence Strategy - EXC number 2064/1 - Project number 390727645. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039B).
[1] Christopher P. Burgess, Loic Matthew, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Representation. arXiv:1901.11390, 2019. URL https://arxiv.org/abs/1901.11390.
[2] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-Object Representation Learning with Iterative Variational Inference. In ICML, 2019. URL https://arxiv.org/abs/1903.00450.
[3] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-Centric Learning with Slot Attention. In NeurIPS, 2020. URL https://proceedings.neurips.cc/paper/2020/file/8511df98c02ab60aea1b2356c013bc0f-Paper.pdf.
[4] Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. SCALOR: Generative World Models with Scalable Object Representations. In ICLR, 2020. URL https://openreview.net/pdf?id=SJxrKgStDH.
[5] Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object-centric Learning from Video. In ICLR, 2022. URL https://openreview.net/forum?id=aD7uesX1GF_.
[6] Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf.  $\mathrm{SAV_i + + }$  : Towards End-to-End Object-Centric Learning from Real-World Videos. In NeurIPS, 2022. URL https://openreview.net/ forum?id=fT9W531LxNS.
[7] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos. In NeurIPS, 2022. URL https://openreview.net/forum?id=eYfIM88MTUE.
[8] Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Discovering Objects that Can Move. CVPR, 2022. URL https://arxiv.org/abs/2203.10159.
[9] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. In ICLR, 2023. URL https://openreview.net/forum?id=b9tUk-f_aG.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. ICCV, 2021. URL https://arxiv.org/abs/2104.14294.
[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dólár, and Ross Girshick. Masked Autoencoders are Scalable Vision Learners. In CVPR, 2022. URL https://arxiv.org/abs/2111.06377.
Supplementary Material for Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities
A Comparison with Additional Baselines 16
A.1 Comparison with Image-Based Object-Centric Methods 16
A.2 Comparison with Concurrent Work on Real-World Videos 16
B Additional Experiments 17
B.1 Long-Term Video Consistency 17
B.2 Optical Flow as Self-Supervised Target 18
B.3 Comprehensive Ablation Study on the MOVi-E Dataset 19
B.4 Stability of Mixer Decoder 19
B.5 Image-Based Feature Similarity on COCO 20
B.6 Sensitivity to Number of Slots During Evaluation 20
B.7 Effect of ViT Architecture on Final Performance 20
C Architectural Details and Hyperparameters 20
C.1 SlotMixer Decoder 21
C.2 ViT Encoders as Dense Features Extractors 21
C.3 Other Modules 23
C.4 Next-Frame Feature Prediction Details 24
C.5Baselines 24
C.6 Hyperparameters 24
C.7 Compute Requirements 25
D Dataset Details 25
E Additional Examples 26
A Comparison with Additional Baselines
A.1 Comparison with Image-Based Object-Centric Methods
In this section, we evaluate how effective our model is for unsupervised image segmentation from videos. In addition to reporting results for the video-based object-centric methods SAVi and STEVE, we compare with several recent image-based object-centric learning methods. SLATE [26] is an image-based object-centric model that trains a discrete VAE [50] as a dense feature extractor and uses a Transformer decoder conditioned on slots to reconstruct discrete representations of VAE features. LSD [36] replaces the Transformer decoder with a latent diffusion model conditioned on the object slots. Next, DINOSAUR [9] incorporates dense DINO features as targets and reconstructs the features itself. We report the Image FG-ARI and Image mBO metrics. They measure how well the predicted segmentation matches the ground-truth segmentation of a given single image (frame), thus consistency over the video is not taken into account.
The results for MOVi datasets are presented in Table A.1. VideoSAUR surpasses both previous image- and video-based methods, showing the benefits of using motion information in combination with semantically coherent self-supervised features. Interestingly, VideoSAUR performs well on both Image FG-ARI and Image mBO metrics, whereas DINOSAUR and LSD seem to improve either in quality of split (measured in Image FG-ARI) or in the sharpness of masks (measured in Image mBO) at the cost of performing worse on the second metric. We also note that LSD results are from larger resolution of MOVi images  $(256 \times 256)$ . We expect that our method can additionally improve if we also use larger resolution videos; however, to be comparable with other baselines, we use  $128 \times 128$  resolution in this work.
In addition, we also compare VideoSAUR with DINOSAUR and other video-based baselines on the more challenging YouTube-VIS dataset (see Fig. A.1). VideoSAUR outperforms DINOSAUR (+4 FG-ARI) and also surpasses video-based STEVE and SAVi methods. This underscores the benefit of our temporal similarity loss over mere feature reconstruction for challenging real-world datasets.
Figure A.1: Image-based comparison on YouTube-VIS (mean  $\pm$  standard dev., 3 seeds).
Table A.1: Comparison with state-of-the-art methods on the MOVi-C, MOVi-E image datasets. Both metrics are computed for individual frames. The results for SLATE and DINOSAUR are from Seitzer et al. [9], while LSD results are from Jiang et al. [36]. We report mean  $\pm$  standard dev. over 5 runs for our model.
MOVi-C	MOVi-E
Image FG-ARI	Image mBO	Image FG-ARI	Image mBO
Block Pattern	42.7	19.5	41.9	20.4
SAVi [5]	41.8	25.9	50.3	20.3
STEVE [7]	51.9	41.6	59.5	34.4
SLATE [26]	43.6	26.5	44.4	23.6
LSD [36]	50.5	46.3	53.4	39.6
SlotDiffusion [37]	-	-	60.0	30.2
DINOSAUR [9]	68.6	39.1	65.1	35.5
VideoSAUR	75.5 ± 0.9	46.0 ± 0.6	78.4 ± 0.7	41.2 ± 0.4
A.2 Comparison with Concurrent Work on Real-World Videos
In concurrent work, two more slot attention-based methods were proposed that learn object-centric representations on real-world videos: SMTC [41] and SOLV [42]. SMTC learns to extracts objects from videos by enforcing semantic and instance consistency over time using a student-teacher approach. SOLV extracts per-frame slots using invariant slot attention [30], applies a temporal consistency module and merges slots using agglomerative clustering; the model is trained using
Table A.2: Comparison of VideoSAUR with DINOv2 ViT B/14 features to SMTC [41] on the DAVIS-2017-Unsupervised validation set. The results for SMTC are from Qian et al. [41].
Method	J	F	J & F
SMTC	36.4	44.6	40.5
VideoSAUR	36.8	21.1	29.0
Table A.3: Comparison of VideoSAUR with DINOv2 ViT B/14 features to OCLR [57] and SOLV [42] on YT-VIS 2019. The results for OCLR and SOLV are from Aydemir et al. [42].
Method	mIoU
OCLR	32.5
SOLV (w/o slot merging)	39.9
SOLV (w/ slot merging)	45.3
VideoSAUR	40.3
DINOSAUR-style feature reconstruction on masked out intermediate frames. In this section, we compare to them using the respective evaluation protocols in their papers.
First, we compare to SMTC in Table A.2 on the DAVIS-2017-Unsupervised dataset [47]. Specifically, we assess our method's transfer performance when trained on YT-VIS 201, and follow the evaluation procedure outlined in [47] for matching ground truth masks to predictions using mean  $\mathcal{J}$  &  $\mathcal{F}$ . We report the Jaccard index  $\mathcal{J}$  (equivalent to IoU), the boundary F-score  $\mathcal{F}$  and their average as  $\mathcal{J}$  &  $\mathcal{F}$ . While the contours of VideoSAUR's segmentation masks (as measured by  $\mathcal{F}$ ) are not as accurate due to processing images and predicting masks at a lower patch resolution ( $37 \times 37$  for VideoSAUR trained with DINOv2 B14 features on original DINOv2 resolution), the Jaccard index  $\mathcal{J}$  is comparable to SMTC.
Second, we compare to SOLV in Table A.3 on the YT-VIS 2019 dataset. We also list the performance of OCLR [57], which uses synthetic data with ground-truth optical flow. As SOLV, we report the mIoU metric matched over the entire video. The results for VideoSAUR are obtained using random 200 videos from the YT-VIS 2019 train split  ${}^{4}$  . Our results surpass OCLR, showing VideoSAUR ’s effectiveness in extracting motion information directly from video data using dense self-supervised features. Additionally, our performance matches that of SOLV without using agglomerative clustering, while SOLV with slot merging outperforms VideoSAUR (+5 mIoU). This highlights the importance of correctly determining the number of slots. While the main concern in this paper is to integrate motion information from video, we see determining the number of slots as an important orthogonal direction. Thus, combining our method a solution such as the one from Aydemir et al. [42] is an interesting direction for future work.
B.1 Long-Term Video Consistency