Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities
Authors: Andrii Zadaianchuk
Maximilian Seitzer
Georg Martius
Venue: NeurIPS 2023 poster (2023)
Keywords: object-centric learning
video
representation learning
self-supervised learning
unsupervised learning

Abstract: Unsupervised video-based object-centric learning is a promising avenue to learn structured representations from large, unlabeled video collections, but previous approaches have only managed to scale to real-world datasets in restricted domains.
Recently, it was shown that the reconstruction of pre-trained self-supervised features leads to object-centric representations on unconstrained real-world image datasets.
Building on this approach, we propose a novel way to use such pre-trained features in the form of a temporal feature similarity loss.
This loss encodes semantic and temporal correlations between image patches and is a natural way to introduce a motion bias for object discovery.
We demonstrate that this loss leads to state-of-the-art performance on the challenging synthetic MOVi datasets.
When used in combination with the feature reconstruction loss, our model is the first object-centric video model that scales to unconstrained video datasets such as YouTube-VIS.

https://martius-lab.github.io/videosaur/

Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities
Andrii Zadaianchuk 1.2+
Maximilian Seitzer
Georg Martius'
Max Planck Institute for Intelligent Systems, Tübingen, Germany 2 Department of Computer Science, ETH Zurich andrii zalaaianchukttuuebbngg......
Abstract
Unsupervised video-based object-centric learning is promising avenue to learn structured representations from large, unlabeled video collections, but previous approaches have only managed scale real-world datasets in restricted domains. Recently, it was shown that the reconstruction of pre-trained self-supervised fea- tures leads object-centric representations on unconstrained real-world image datasets. Building on this approach, we propose a novel way to use such pre- trained features in the form of temporal feature similarity loss. This loss encodes semantic and temporal correlations between image patches and is a natural way introduce a motion bias for object discovery. We demonstrate that this loss leads to state-of-tt tate--of-the-art performance on the challenging synthetic MOVi datasets. When used in combination with the feature reconstruction loss, our model is the first object-centric video model that scales to unconstrained video datasets such YouTube- VIS. https: htt////aattussieeeeeeeeeeeeeeeet github. so/videosaur/
1 Introduction
Autonomous systems should have the ability to understand the natural world in terms of independent entities. Towards this goal, unsupervised object-centric learning methods [1-3] learn to structure scenes into object representations solely from raw perceptual data. By leveraging large-scale datasets, these methods have the potential obtain a robust object-based based understanding of the natural world. Of particular interest in recent years have been video-based methods [4-7], not least because the temporal information in video presents a useful bias for object discovery [8]. However, these approaches are so far restricted to data of limited complexity, successfully discovering objects from natural videos only on closed-wort datasets in restricted domains.
In this paper, we present the method Video Slot Attention Using temporal feature similaRity, VideoSAUR, that scales video object-centric learning unconstrained real-world datasets cov- ering diverse domains. To achieve this, we build upon recent advances in image-based object-centric learning. In particular, Seitzer self-supervised methods like DINO [10] or MAE [11] leads to state-of ofetth-artt object discovery on complex real-worid images. We demonstrate that combining this feature reconstruction objective with video object-centric model [5] also leads to promising results on real-world You'll You'Tube videos.
[9] showed that reconstructing pre-trained features obtained from
We then identify weakness in the training objective of current unsupervised video object-centric architectures [4, 7]: the prevalent reconstruction loss does not exploit the temporal correlations existing in video data for object grouping. To address this issue, we propose novel self-supervised loss based on feature similarities that explicitly incorporates temporal information (see Fig. 1). The loss works by predicting distributions over similarities between features of the current and future
equal contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Zt
Slots
P Pec++b
Self self-superiiee
VIT
ht
Group
Predict
60
0.
Cn
cn
Cosine Distance
Similarity Loss
Bit+
A
P t,t+
LeB
Selfosueeeee sepd
VIT
hith
06
3.0
CO
0.
02
0.
DG 0B 30
C.
00
0:37
0.05
Figure 1:
We propose self-supervised temporal similarity loss for training object-centric video models. For each patch time t, the model has to predict distribution Pr.++ indicating where all semantically-simme
patches have moved to steps into the future. The target distribution Pr,+ is computed with softmax on the affinity matrix A containing the cosine distance between all patch features h, h4+E- The loss incentivizes the model to group areas with consistent motion and semantics into slots.
frames. These distributions encode information about the motion of individual image patches. To efficiently predict those motions through the slot bottleneck, the model is incentivized to group patches with similar motion into the same slot, leading to better object groupings as patches belonging to an object tend to move consistently. our experiments, we find that such a temporal similarity loss leads to state-of-the-art performance on challenging synthetic video datasets [12], and significantly boosts performance on real-world videos when used in conjunction with the feature reconstruction loss.
In video processing, model efficiency is of particular importance. Thus, we design an efficient object- centric video architecture by adapting the SlotMixer decoder [13] recently proposed for 3D object modeling for video decoding. Compared to previous decoder designs [3], the SlotMixer decoder scales gracefully with the number of slots, but has a weaker inductive bias for object grouping. We show that this weaker bias manifests in optimization difficulties in conjunction with conventional reconstruction losses, but trains robustly with our proposed temporal similarity loss. We attribute this to the reee-suueee nature of the similarity loss: compared to reconstruction, it requires predicting information that not directly contained in the input; the harder task seems compensate for the weaker bias of the SlotMixer decoder.
To summarize, our contributions are as follows: (I) we propose novel self self-supervised loss for object- centric learning based on temporal feature similarities, (2) we combine this loss with an efficient video architecture based on the SlotMixer decoder where synergistically reduces optimization difficulties, (3) we show that our model improves the state-of-the-ar on the synthetic MOVi datasets by large margin, and (4) we demonstrate that our model is able learn video object-centric representations on the YouTube-ViI dataset [14], while staying fully unsupervised. This paper takes a large step towards unconstrained real-world object-centric learning on videos.
Related Work
Video Object-Centric Learning There is rich body of work on discovering objects from video, with two broad categories of approaches: tracking bounding boxes [4, 15-17] or segmentation masks [2, 5-7, 18-25]. Architecturally, most recent image-based models for object-centric learning [3, 9, 26] are based on an auto-encoder framework with a latent slot attention grouping module [3] that extracts set of slot representations. For processing video data, a common approach [5-7,2] 21, 24] then to connect slots recurrently over input frames; the slots from the previous frame act as initialization for extracting the slots of the current frame. We also make use of this basic framework.
Scaling Object-Centric Learning Most recent work has attempted increase the complexity of datasets where objects can successfully be discovered, such the synthetic ClevrTex [27] and MOVi

---

datasets [12]. On natural data, object discovery has so far been limited to restricted domains with lim- ited variety of objects, such as BouTube-Aquaiiue and -Cars [7], or autonomous driving datasets like WaymoOpen or KITTI [28]. On more open-ended datasets, previous approaches have struggled [29].
To achieve scaling, some works attempt to improve the grouping module, for example by introducing cquivariances to slot pose transformations [30], smoothing attention maps [3 IJ, formulating grouping as graph cuts [32] or stick-breaking process [33], or by overcoming optimization difficulties by introducing implicit differentiation [34, 35J. In contrast, we do not change the grouping module, but use the vanilla slot attention cell [3].
Another prominent approach is to introduce better training signals than the default choice of image reconstruction. For example, one line of work instead models the image as distribution of discrete codes conditional on the slots, either autoregressively by a Transformer decoder [7, 26], or via diffusion [36, 37]. While this strategy shows promising results synthetic data, it so far has failed to scale to unconstrained real-world data [9].
An alternative is to step away from uully-unsupervised representation learning by introducing weak supervision. For instance, SAVi [5] predicts optical flow, and SAVi+ [6] additionally predicts depth maps as signal for object grouping. Other works add auxiliary loss that regularizes slot attention's 's masks towards the masks of moving objects [8, 38]. Our model also has a loss that focuses on motion information, but uses unsupervised formulation. OSRT [13] shows promising results synthetic 3D datasets, but is restricted by the availability of posed multi- multi-camera imagery. While all those approaches improve the level of data complexity, it has not been demonstrated that they can scale to unconstrained real-world data.
The most promising avenue so far in terms of scaling to the real-world is to reconstruct features from modern self-supervised pre-training methods [10, 11, 39, 40J. Using this approach, DINOSAUR [9] showed that by optimizing in this highly semantic space, it is possible to discover objects on complex real-world image datasets like COCO or PASCAL VOC. this work, we similarly use such self- supervised features, but for learning on video instead of images. Moreover, we improve upon reconstruction of features by introducing a novel loss based on similarities between features.
Concurrent Work Parallel to this work, two more slot attention based methods were proposed that learn object-centric representations on real-world videos: SMTC [41] and SOLV [42]. SMTC learns to extracts objects from videos by enforcing semantic and instance consistency over time using a student-teacher approach. SOLV extracts per-frame slots using invariant slot attention [30], applies temporal consistency module and merges slots using agglomerative clustering; the model is also
trained using DINlsAUR-style feature reconstruction, but on masked out intermediate frames.
Method
In this section, we describe the main new components of VideoSAUR our proposed object-centric video model - and its training: a pre-trained self-supervised ViT encoder extracting frame features (Sec. 3.1), temporal similarity loss that adds motion bias object discovery (Sec. 3.2), and the SlotMixer decoder to achieve efficient video processing (Sec. 3.3). See Fig. 2 for an overview.
3.1 Slot Attention for Videos with Dense Self-Supervised Representations
VideoS is based on the modular video object-centric architecture recently proposed by SAVi [5] and also used by STEVE [7]. Our model has three primary components: (1) a pre-trained self- supervised VIT feature encoder, (2) a recurrent grouping module for temporal slot updates, and (3) the SlotMixer decoder (detailed below Sec. 3.3).
We start by processing video frames æt, with time steps L E (I,... into patch features hy:
h fa(x)) h4 E RLXD
where fo is a self-supervised Vision Transformer encoder (ViT) [43] with pre-trained parameters 6, and v, is the input at time step l. The ViT encoder processes the image by splitting it to L non- overlapping patches of fixed size (e.g. 16 x 16 pixels), adding positional encoding, and transforming them into I. feature vectors h4 (see App. C.2 for more details on ViTs). Note that the i'th feature retains association to the "th image patch; the features thus can be spatially arranged. Next,
3

---

Recurrent Attention
SlotMixer Decoder
Slots 5t-1
Slots
Reconstructed Patch Features
Frame 2;
Patch Features
h
Reconstruction
Self-supervised ViT
Patch Features h
Frame Zeth
Patch Transition
Predicted Probabilities Ai
Probabilities
Similarity Loss
P(h., h.**)
Y
Figure 2: Overview of VideoSAUR. Object slots 8t are extracted from patch features h, of self- supervised ViT using time-recurrent slot attention, conditional on slots from the previous time step t- 1. The model is trained by reconstructing the patch features h2 of the current frame 3t, and by predicting the similarity distribution over patches of future frame zi+ (see also Fig. 1). The predictions y, and y s are decoded efficiently using SlotMixer decoder.
we transform the features from the encoder with a slot attention module [3] to obtain a latent set [s]] st RM with K slot representations:
8, {
st SAg $t-1)-
Slot attention recurrently initialized with the slots of the previous time step 1, with initial slots So sampled independently from Gaussian distribution with learned location and scale. Slot attention works by grouping input features into slots by iterating competitive attention steps; we refer to Locatello et al. [3] for more details. To train the model, we use SlotMixer decoder Gæ (see Sec. 3.3) transform slots 8, to outputs y: = (8.). Those outputs are used as model predictions for the reconstruction and similarity losses introduced next.
3.2 Self-Supervised Object Discovery by Predicting Temporal Similarities
We now motivate our novel loss function based on predicting temporal feature similarities. Video affords the opportunity to discover objects from motion: pixels that consistently move together should be considered as one object, sometimes called the "common fate" principle [44]. However, the widely used reconstruction objective whether of pixels [5], discrete codes [7] or features [9] does not exploit this bias, as to reconstruct the input frame, changes between frames do not have to be taken into account.
Taking inspiration from prior work using optical flow prediction target [5], we design self- supervised objective that requires predicting patch motion: for each patch, the model needs to predict where all semantically-simieen supervised features describing the patches, we integrate both semantic and motion information; this in contrast to optical flow prediction, which only relies on motion. Specifically, we construct an affinity matrix A with the cosine similarities between all patch features from the present frame h, and all features from some future frame h
patches have moved steps into the future. By comparing self-
he [hell
A
A E (-1,1))xA
As self-supervised features are highly semantic, the obtained feature similarities are high for patches that share the same semantic interpretation. Due the ViT's positional encoding, the similarities also take spatial closeness of patches into account. Figure shows several example affinity matrices.
Because there are ambiguities in our similarity-based derivation of feature movements, we frame the prediction task as modeling a probability distribution over target patches - instead of forcing the prediction of an exact target location, like with optical flow prediction. Thus, define the

---

Affiity Mateic A Truseikes Paisaldlay P
Afbait Mates
Truniiis Paisadlay P
05
6.0 0i2
001
010
05
10
00
0002
Figure 3: Affinity matrix A and transition probabilities Pe.++ values between patches (marked by purple and green) of the frame In and patches of the future frame 24+k MOVi-C (left) and YT-VIS (right). Red indicates maximum affinity/probabiitty. Also see Fig. B.4 for more examples, and our website for an interactive visualization of temporal feature similarities.
probability that patch moves patch j by normalizing the rows of the affinity matrix with the softmax, while masking negative similarity values (superscripts refer to the elements of the matrix):
exp(A>/-)
if A 5 2 20,
Σ exp(A*>>) kE
p
if A <0,
where 7 is the softmax temperature. The resulting distribution can be interpreted the transition probubilities of a random walk along graph with image patches as nodes [45]. Then, we define the similarity loss the cross entropy between decoder outputs and transition probabilities:
CE(P),44
(5)
1=1
Figure illustrates the loss computation for an example pair of input frames.
Why is this Loss Useful for Object Discovery? Predicting which parts of the videos move consistently is most efficient with an object decomposition that captures moving objects. This is similar to previous losses predicting optical flow [5]. But in contrast, our loss (Eq. 5) also yields useful signal for grouping when parts of the frame are not moving: as feature similarities capture semantic aspects, the task also requires predicting which patches are semantically similar, helping the grouping into objects e.g. by distinguishing fore- and background (see Fig. 3). Optical flow for grouping also has limits when camera motion is introduced; in experiments, is more robust in such situations. Methods based on optical flow or motion masks can also struggle with inaccurate flow/motion mask labels unlike our method, which does not require such labels. This is of particular importance for in-the-wild video, where motion estimation is challenging.
find that loss
Role of Hyperparameters. and the softmax temperature r. The optimal time shift depends on the expected time scales of movements in the modeled videos and should be chosen accordingly. The temperature 7 controls the concentration of the distribution onto the maximum. Thus, it effectively modulates between two different tasks; accurately estimating the patch motion (low r), and predicting the similarity of each patch to all other patches (high 7). In particular in scenes with little movement, the latter may be important to maintain meaningful prediction task. In our experiments, we find that the best performance is obtained with a balance between the two, showing that both modes are important.
The loss has two hyperparameters: the time shift into the future
Final Loss. While the temporal similarity loss yields state-of-the-art performance on synthetic datasets, as shown below, we found that on real-world data, performance can be further improved by adding the feature reconstruction objective introduced in Seitzer et al. [9]. We hypothesize this is because the semantic nature of feature reconstruction adds another useful bias for object discovery. Thus, the final loss is given by:
T
L
isim (Pr,+ ye + a.g. y1 100).
(6)
5

---

MOVi-C
MOVi-E
YouTube- VIS
Impot
SAV
VideoSAUR STEVE
Figure 4: Example predictions of VideoSAUR compared recent video object-centric methods.
where y/: = m RIX, yt BRAP is the output of the SlotMixer decoder Iy and is weighting factor used to make the scales of the two losses similar (we use a fixed value a = 0.1 for all experiments real-world datasets). Like in Seitzer et al. [9], do not train the VIT encoder fø-
3.3 Efficient Video Object-Centrie Learning with the SlotMixer Decoder
In video models, resource efficiency is of particular concern: recurrent frame processing increases the load on compute and memory. The standard mixture-based decoder design [3] decodes each output K- times, where K the number of slots, and thus scales linearly with both compute and memory. This can become prohibitive even for a moderate number of slots. The recently introduced SlotMixer decoder [13] for 3D object-centric learning instead has, for all practical purposes, constant overhead in the number of slots, by only decoding once per output. Thus,
propose to SlotMixer decoder gv for predicting the probabilities Pt.++ from the slots st. To adapt the decoder from 3D to 2D outputs, change the conditioning on 3D query rays L learned positional embeddings, corresponding to L patch outputs y!- See App. C.1 for more details on the SlotMixer module.
As consequence of the increased efficiency of SlotMixer, there also is increased flexibility of how slots can be combined to form the outputs. Because of this, this decoder has a weaker inductive bias towards object-based groupings compared to the standard mixture- based decoder. With the standard reconstruction loss, we observed that this manifests training runs in which no object groupings are discovered. But in combination with our temporal similarity loss, these instabilities disappear (see App. B.4). We attribute this to the self sulf-supe nature the similarity loss?; having to predict information that is not directly contained in the input increases the difficulty of the task, reducing the viability of non-object based groupings.
Experiments
We have conducted number of experiments answer the following questions: (1) Can object- centric representations be learned from large number of diverse real-world videos? (2) How does VideoSAUR perform in comparison to other methods well-established realistic synthetic datasets? (3) What are the effects of our proposed temporal feature similarity loss and its parameters? (4) Can we transfer the learned object-groo grouping to unseen datasets? (5) How efficient the SlotMixer decoder in contrast to the maxxure-based decoder?
4.1 Experimental Setup
To investigate the characteristics of proposed method, utilize three synthetic datasets and three real-world datasets. For synthetic datasets, we selected the MOVi-C, MOVi-D
Datasets
Novel- -iew synthesis. the original task for which SlotMixer was proposed, is similarly a self self-superrvied prediction task. This may have contributed to the success of SlotMixer in that setting.

---

Table 1: Comparison with state-of the methods on the MOVi-C, MOVi-E, and YT-VIS datasets.
We report foreground adjusted rand index (FG-ARI) and mean best overlap (mBO) over 5 random seeds. Both metrics are computed for the whole video (24 frames for MOVi, frames for YT-VIS).
MOVi-C
MOVi-E
YT-VIS
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
Block Pattern
24.2
11.1
36.0
16.5
24
14.9
SAVi [5] 22.2
13.6 1.6
0.9
16.0
5
12.7
2.3
STEVE [7]
36.1 + 2.3
26.5 + 1.1
50.6 1.7
26.6 0.9
20.0 + 1.5
20.9
- 0.5
VideoSAUR
64.8 # 1.2
0.6
35.6 7.19 0.5
0.6
29.1
0.4
and MOVi-E datasets [12] that consist of numerous moving objects on complex backgrounds. Additionally, evaluate the performance of our method on the challenging YouTube Video Instance Segmentation (YT-VIS) 2021 dataset [14] as an unconstrained real world dataset. Furthermore, we examine how well our model performs when transferred from YT-VIS 2021 to YT-VIS 2019 [46] and DAVIS [47]. Finally, we use the COCO dataset [48] to study our proposed similarity loss function with image-based object-centric learning.
Metries We evaluate our approach terms of the quality of the discovered slot masks (output by the decoder), using two metrics: video foreground ARI (FG-ARI) [2] and video mean best overlap (mBO) [49]. FG-ARI is a video version of a widely used metric in the object-centric literature that measures the similarity of the discovered objects masks to ground truth masks. This metric mainly measures how well objects are split. mBO assesses the correspondence of the predicted and the ground truth masks using the intersection-over-ueee mask is matched to the predicted mask with the highest loU, and the average IoU is then computed across all assigned pairs. Unlike FG-ARI, mBO also considers background pixels, and provides measure of how accurately the masks fit the objects. Both metrics also consider the consistency of the assigned object masks over the whole video.
(IoU) measure. In particular, each ground truth
In addition, we also use image-based versions of those metrics (Image FG-ARI and Image mBO, computed individual frames) for comparing with image-based methods.
Baselines We compare our method with two recently proposed methods for unsupervised object- centric learning for videos: SAVi [5] and STEVE [7]. SAVi uses mixture-based decoder and trained with image reconstruction. We use the unconditional version SAVi. STEVE uses a transformer decoder and is trained by reconstructing discrete codes of dVAE [50]. Similar to Seitzer et al. [9], we also add a regular block pattern baseline, corresponding to splitting the video into regular block masks of similar size that do not change over time. By showing the metric values for trivial decomposition of the video, this baseline useful to contextualize the results of the other methods. In addition video-based methods, we compare our model to image-based methods, including DINOSAUR [9), LSD [36] and Slot Diffusion [37], showing that our approach performs well in both object separation and mask sharpness. Last, we also compare model to two concurrent works discovering objects from real-world video, SMTC [41] and SOLV [42].
4.2 Comparison with State-of-the-Art Object-Centric Learning Methods
When comparing VideoSAUR to STEVE and SAVi, it is evident that VideoS outperforms the baselines by significant margin, both in terms of FG-ARI and mBO (see Table I and Fig. 4). On the most challenging synthetic dataset (MOVi-E), VideoSAUR reaches 73.9 FG-ARI. Notably, for the challenging YT-VIS 2021 dataset, both baselines perform comparable or worse than the block pattern baseline in terms of FG-ARI, showing that previous methods struggle decompose real-world vidcos into consistent objects. We additionally compare VideoSAUR to image- based methods in App. A.1, including strong recent methods (LSD, SlotDiffusion and DINOSAUR). and find that approach also outperforms the prior image-based SoTA. Finally, in App. A.2, we find that our method performs competitively with concurrent work.
Next, we report how well our method performs terms of zero-shot transfer to other datasets to show that the learned object discovery does generalize to unseen data. In particular, we train VideoSAUR
7

---

YouTube- VIS 2019
DAVIS
VideoSAUR
mB0
STEVE
Random pattern
1
3
4
5
7
8
2
4
5
7
8
Number of slots
Number of slots
Figure 5: Zero-shot transfer of learned object-centric representations on YT-VIS 2021 to the YT-VIS 2019 and DAVIS datasets for different number of slots.
on the YT-VIS 2021 dataset and evaluate it on the YT-VIS 2019 and DAVIS datasets. YT-VIS 2019 has similar object categories, but smaller number of objects per image. The DAVIS dataset consists of videos from a fully different distribution than YT-VIS 2021. As the number slots can be changed during evaluation, we test VideoSAUR with different number of slots, revealing that the optimal number of slots is indeed smaller for these datasets. We find that our method achieves a performance of 41.3 = 0.9 mBO YT-VIS 2019 dataset and 34.0 + 0.4 mBO on DAVIS dataset (see Fig. 5), illustrating its capability to effectively transfer the learned representations to previously unseen data with different object categories and numbers of objects.
Long-term Video Consistency In addition to studying how VideoSAUR performs on relatively short 6-frame video segments from YT-VIS, we also evaluate our method on longer videos. In App. B.1, show the performance for 12-frame and full VT-VIS videos. While, can be expected, performance on longer video segments smaller in terms of FG-ARI, we show that the gap between VideoSAUR and the baselines is large, indicating that VideoSAUR over longer time intervals. Closing the gap between short-term and long-term consistency using mem- ory modules [24, 51] is interesting future direction that could be useful for video prediction [52] as well as for object-centric goal- goal-based [53, 54] and model-based [55] reinforcement learning.
track the main objects in videos
4.3 Analysis
In this section, we analyze various aspects of our approach, including the importance of the similarity loss, the impact of hyperparameters (time-shift &s and softmax temperature 7), and the effect of the choice of self-supervised features and decoder.
Choice of Loss Function (Table and Table 3) We conduct an ablation study to demonstrate the importance of the proposed temporal similarity loss, comparing and combining it with the feature reconstruction loss [9]. We also consider predicting the features of the next frame (see App. C.4 for implementation details). For all datasets, feature reconstruction alone performs significantly worse than the combination of feature reconstruction and temporal similarity loss. Predicting the features of the next frame in addition to feature reconstruction also yields improved performance, but is worse than the temporal similarity, suggesting that the success of our loss can be partially explained by the integration of temporal information through future prediction. Interestingly, on MOVi-C, using the temporal similarity loss alone significantly improves the performance over feature reconstruction (+20 FG-ARI, +7 mBO). To provide insight into the qualitative differences between the losses, we analyze the videos with the most significant differences in FG-ARI (see Fig. E.4): unlike feature reconstruction, the temporal similarity loss does not fragment the background or large objects into numerous slots, and it exhibits improved object-tracking capabilities even when object size changes. To gain further insights, we also consider (ground truth) optical flow as prediction target that only captures motion, but no semantic information (see App. B.2 for detailed discussion). We find that only predicting optical flow is not enough for successful scene decomposition, underscoring the importance of integrating both motion and semantic information for real-world object discovery.
Robustness to Camera Motion (Table 4) Next, we investigate if VideoS training with the similarity loss is robust to camera motion, as such motion makes isolating the object motion more difficult. As controlled experiment, compare between MOVi-D (without camera motion) and
8

---

Table 2:
Loss ablation MOVi-C.
Table 3: Loss ablation YT-VIS.
Loss Type
Metric
Loss Type
Metric
Next Frame Feat, Pred.
Next Frame Feat, Pred.
Feat. Rec.
Temp. Sim.
FG-ARI mBO
Feat. Rec.
Temp. Sim.
FG-ARI mBO
40.2
23.5
35.4
26.7
47.2
24.7
37,9
27.3
60.8
30.5
4
26.2
29.1
60.7
30.3
4
39,5
29.1
Table 4: Robustness to introducing camera motion (MOVi-D - MOVi-E).
Table 5: Decoder comparison on MOVi-C and YT-VIS.
MOVi-C
YT-VIS
Memory
MOVi-D MOVi-E
FG-ARI mBO
FG-ARI mBO GB @24 slots
SAVi (optical flow) [12] VideoSAUR (temporal sim.)
19.4
2.7
Mixer
60.8
30.5
39.5
29.1
24
55.7
62.5
MLP
64.2
27.2
39.0
29.1
70
MOVi-E (with camera motion), and train VideoSAUR using only the temporal similarity loss. We contrast with SAVi trained with optical flow prediction', and find that VideoSAUR is more robust to camera motion, performing better on the MOVi-E dataset than on the MOVi-D dataset (+6.88 vs -16.7 FG-ARI for SAVi).
Choice of Decoder (Table 5) We analyze how our method performs with different decoders and find that both the MLP broadcast decoder [9] and our proposed SlotMixer decoder can be used for optimizing the temporal similarity loss. VideoSAUR with the MLP broadcast decoder achieves similar performance on YT-VIS and MOVi datasets,
requires 2-3 times more GPU memory (see App. C.1 for more details and Table B.3 for the detailed comparison of decoders on MOVI-E dataset). Thus, we suggest use the SlotMixer decoder for efficient video processing.
Softmax Temperature (Figure 6a) We train VideoSAUR with DINO S/16 features using different softmax temperatures т. We find that there is a sweet spot in terms of grouping performance at 0.075. Lower and higher temperatures lead to high variance across seeds, potentially because there is not enough training signal with very peaked (low 7) and diffuse (high 7) target distributions.
Target Time-shift (Figure 6b) We train VideoSAUR with DINO S/16 features using different time-shifts & to construct the affinity matrix Ar, On both synthetic and real-world datasets, k = 1 generally performs best. Interestingly, we find that for k O, performance drops, indicating that predicting pure self-similarities is not a sufficient task for discovering objects on its own.
Choices for Sele-Super Features (Figures 6c and 6d) We study two questions about the usage of the VIT features: which ViT features gguerie//eys/vallus//uupp temporal similarity loss? Do different sulf--uper representations result in different performance? In Fig. 6c, we observe that using DINO "key" and "query" features leads to significantly larger mBO, while for FG-ARI "query" worse and the other features are similar. Potentially, this is because keys are used in the VIT's self-attention and thus could be particularly good to compare with the scalar product similarity. Consequently, VideoSAUR uses "key" features all other experiments. Moreover, we study if the temporal similarity loss is compatible with different self self-supervised representations. In Fig. 6d, we show that AadeoAUUR works well with differennt types of representations, with MSN [39] and DINO [10] performing slightly better than MAE [11] and MOCO-v3 [40]. We also demonstrate that further fine-tuning the DINO features utilizing a self-supervised approach named
should be used for the
clustering [56] on unlabeled videos enhances the mask quality of VideoSAUR.
Pre-training Dataset (Table 6) All self-supervised methods we utilize are trained on the ImageNet dataset, which has strong bias towards object-centricness its images mostly contain single objects, and b) introduces a large number of additional images external to the dataset we are training
SAVi results with optical flow from Greff al. [12].

---

MOVi-C
YT-VIS
60
30
30
MOVi-C
30
FG-ARI
60
Optimal temp.
FG-ARI
40
20
FG-ARI
20
mBO
20
40
20
10
10
0
0.01
0.05
0.2
0.5
1 2
3
1
0
2
1
2
3
(a) Softmax temperature 7.
(b) Target time-shift & on MOVi-C and YT-VIS datasets.
65
40
40
Query
MOCO-v3
EG-ARI
60
35
FG-ART
60
35
Value
MSN
55
30
55
30
Output
50
25
50
25
Key
DINO w/ TimeT
(c) DINO VIT features.
(d) Self supervised representation method.
Figure 6: Studying the effect of different parameters of the temporal similarity loss.
Table 6: Comparing VideoSAUR with features trained on MOVi-E MOAE+MOVi-E) to features trained on ImageNet (MAE+ImageNer), For MAE+MOVi-E, we pre-train a ViT-B/16 using the self-supervised MAE method on MOVi-E for 200 epochs. VideoSAUR is able to perform high-quality object discovery even without access to any external data.
MOVi-C
MOVi-E
FG-ARI
mBO
FG-ARI
mBO
VidcoSAUR w/ MAE+ImageNet features VidcoSAUR w/ MOV+MOVi-E features
58.0
30.4
72.8
27.1
59.8
27.5
70.6
23.3
VideoSAUR on. An interesting question is whether a) and b) are actually required for the success of method. To answer it, we train ViT-B/16 encoder from scratch on the MOVi-E dataset using the MAE method, and then train VideoSAUR using the obtained features. Interestingly, we find that the features from MOVi-E yield similar results compared ImageNet-trained' features (although with slight drops in mask quality), demonstrating that VideoS AUR able to perform high-quality object discovery even without access to external data. This result also has broader implications as it potentially increases the applicability of feature reconstruction-base datasets fully out of the domain of ImageNet. It also raises follow-up question: what properties of the pre-training dataset (and method) are important to obtain good target features for object discovery?
object-centric methods to
5
Conclusion
This paper presents the first method for unsupervised video-based object-centric learning that scales to diverse, unconstrained real-world datasets such as YouTTuee-VIS. By leveraging dense self-supervised features and extracting motion information with temporal similarity loss, we demonstrate superior performance on both synthetic and real-world video datasets. We hope our new loss function can inspire the design of further self-supervised losses for object-centric learning, especially in the video domain where natural self-supervision is available.
Still, our method does not come without limitations: in longer videos with occlusions, slots can get reassigned to different objects or the background (see Fig. B.5 for visualizations of failure cases). VideoSAUR also inherits a limitation of all slot attention-based method, namely that the the number of slots is static and needs to be chosen a priori. Similar to DINOSAUR [9], the quality of the object masks restricted by the patch-based nature of the decoder. Finally, while the datasets we use in this work are significantly less constrained compared to datasets used by prior work, they still do not capture the full open-world setting that object-centric learning aspires to solve. Overcoming these limitations is great direction for future work.
10

---

Acknowledgements
We thank Martin Butz, Cansu Sancaktar and Manuel Traub for useful suggestions and discussions. The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS- IS) for supporting Maximilian Seitzer. Andrii Zadaianchuk supported by the Max Planck ETH Center for Learning Systems. Georg Martius is member of the Machine Learning Cluster of Excellence, funded by the Deutsche Forschungsgemeinscaaif (DFG, German Research Foundation) under Germany's Excellence Strategy EXC number 2064/1 Project number 390727645. We acknowledge the support from the German Federal Ministry of Education and Research (BMBF) through the Tübingen AI Center (FKZ: 011S18039B
References
[1] Christopher P. Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. MONet: Unsupervised Scene Decomposition and Repre- sentation. arXiv:1901.11380 2019. URL https://rrzi org hargg/ab//ii /990 11390.
[2] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerehner. Multi-Object Representation Learning with Iterative Variational Inference. In ICML, 2019. URL https:
//arxiv. org/ httpaaas//11/000// 00450.
[3] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-Centric Learning with Slot Attention. NewrlPS, 2020. URL httptttp::/pppcoeeeiiiie Paper pdf.
neurips ctt.ppprcccpppe/////p/
[4] Jindong Jiang, Sepehr Janghorbani, Gerard de Melo, and Sungjin Ahn. SCALOR: Gener- ative World Models with Scalable Object Representations. In ICLR, 2020. URL https:
//openreview. het/ppffiid=SDJrrggttte
[5] Thomas Kipf, Gamaleldin Fathy Elsayed, Aravindh Mahendran, Austin Stone, Sara Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional Object- centric Learning from Video. In ICLR, 2022. URL /ttpss://ppenrriiiie wwtp:/nntttoo/ieeeemmmmm aD7uesX1GF_.
[6] Gamaleldin Fathy Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael Curtis Mozer, and Thomas Kipf. SAVi++: Towards End-to-End Cbjecc-Centric Learning from Real-Wer World Videos. In NeurIPS, 2022. URL https://openreviort net/ formmPid=-F9M5311:eN.
[7] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos. In NeurlPS, 2022. URL http::::///ppperrrriee net/ Forum?id=eYa IM88MTUE.
[8] Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Discovering Objects that Can Move. CVPR, 2022. URL https:///axxxit org/ abs/ 2203. 10159.
[9] Maximilian Seitzer, Max Hor, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl- Johann
Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, and Francesco Locatello. Bridging the gap to real-world object-centric learning. ICLR, 2023.
URL. WRR.htttps://oppneevieer seet/forumiddd-beee
a_a
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self- Self--uuervised Vision Transformers. ICCV, 2021. URL https: : /ttp://aa.vv.v//aa///a//eaaeaaaa org/ abs/ 2104. 14294.
[11] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders are Scalable Vision Learners. In CVPR, 2022. URL / //trxiv..vvv//////////eaeaaaa 2111. 06377.
http//axwv../aaeaaa/a//aa/ org/ abss/

---

httg/papers//22/21-11oe [12] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J. Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, Thomas Kipf. Abhijit Kundu, Dmitry Lagun, Issam Laradji, Hsueh-Ti (Derek) Liu, Henning Meyer, Yishu Miao, Derek Nowrouzezahrai, Cengiz Oztireli, Etienne Pot, Noha Radwan, Daniel Rebain, Sara Sabour, Mehdi S. M. Sajjadi, Matan Sela, Vincent Sitzmann, Austin Stone, Deqing Sun, Suhani Vora, Ziyu Wang. Tianhao Wu, Kwang Moo Yi, Fangcheng Zhong, and Andrea Tagliasacchi. Kubric: A Scalable Dataset Generator. In CVPR, 2022. URL https: arxivv org/abs/ 2200//333000..0 03570.
[13] Mehdi SM Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetic, Mario Lucic, Leonidas J Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer. In NeurIPS, 2022. URL https:/ http/vrxxvvv./////////ee//a.aav/ orggoos//////ooooo..eo.ooooeos// abw.aaaa...2////////////// 06922.
[14) Linjie Yang. Yuchen Fan, Yang Fu, and Ning Xu. The 3rd large-scale video object object segmentation challenge- video instance segmentation track, June 2021. URI htt.hhttt///ouuubbeeeeee org/ dataset/vis.
[15] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequen- tial Attend, Infer, Repeat: Generative Modelling of Moving Objects. In NeurlPS, 2018.
URL https://proceedingg wewpeeeddininnnnnnnieeeeee wetprips.cpppppep/001/ html..
[16] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convo- lutional neural networks. In AAAI, 2020. URL https://arxix: org/ abs/ /1911 1911.09033.
[17] Zhixuan Lin, Yi-Wu Fu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng, Jindong Jiang, and Sungjin Ahn. SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition. In ICLR, 2020. URL https https: //tpp//ei///eeeieee/// wet.eeeeewwwwweeeeeeeee..e...e.e forum?id=rk103ySYDI.
[18] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In NeurlPS, 2017. URL https://aartte http://aaxvvvv./ org/ abs/aa/s//// 03498.
[19] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In ICLR, 2018. URL httph:ttp//// htttap::///peereieess aet/forummid=rry22000
[20] Rishi Veerapaneni, John D. Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Ji- ajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity Abstraction in Visual Model- based Reinforcement Learning. In Conference on Robot Learning, 2019. URL https:
//arxiv.org wwwp:/ogg/abb///0/// abtp//aab///././////// 12827.
[21] Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo Rezende. PARTS: Unsuper- vised segmentation with slots, attention and independence maximization. In ICCV, 2021. URL
https:///eeexplores ieee. wrw//oouuegooooooooooooooooooooo
[22] Marissa A Weis, Kashyap Chitta, Yash Sharma, Wieland Brendel, Matthias Bethge, Andreas Geiger, and Alexander S. Ecker. Benchmarking Unsupervised Object Representations for Video Sequences. JMLR, 2021. URL /tttts://jmll.
html.
[23] Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matthew Botvinick, Alexander Lerchner, and Christopher P. Burgess. SIMONe: View-invariant, temporally-abstracter object representations via unsupervised video decomposition. In NeurIPS, 2021. URL https: http://pppeeeeeeeeeeeee hett/forunrIdd-=YyTTaatttt
[24] Manuel Traub, Sebastian Otte, Tobias Menge, Matthias Karlbauer, Jannik Thuemmel, and Mar- tin V. Butz. Learning What and Where: Disentangling Location and Identity Tracking Without Supervision. In ICLR, 2023. URL htlhttss///ppereeiiiee net forum?dd=MeD--Aa Ak-H
[25] Sadra Safadoust and Fatma Güney, Multi-object discovery by low-dimensional object motion. In ICCV, 2023. URL h //tp:i/a//ivv////a wwt.aaawwi.r...raaaaaaaaaea
www.ooo///aao... wwwgg/bbs/3300 08027.
[26] Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate DALL-EE Learns to Compose. In ICLR, 2022. URL https: htttpps://ppereeee http:///vpp//we.e//// http:///ppeeeieeeeeeeee/ertttttp hetf://orrmm//i--000000000e
12

---

[27] Laurynas Karazija, Iro Laina, and Christian Rupprecht. ClevrTex: A Texture-Rice Benchmark for Unsupervised Multi- Segmentation. In NeurIPS Track on Datasets and Benchmarks, 2021. URL hww.::tt////o //ttx//axvvvv///////a///aeaaaava org/ /ts///b///// /211.. 2121..1211.......
[28] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets Robotics: The KITTI Dataset. International Journal of Robotics Research, 2013, URL wwtph:tt////t cvlibs net www.ecaiiinnnngieeeeeeeee
pdff
[29] Yafei Yang and Bo Yang. Promising or Elusive? Unsupervised Object Segmentation from Real-world Single Images. In NeurlPS, 2022. URL httptht:/tst/eeeeeaeeeeee wwtpw/.rneeeee/edeeemmmm
DzPWTwfby5d.
[30] Ondrej Biza, Sjoerd van Steenkiste, Mehdi M. Sajjadi, Gamaleldin F. Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. In ICML, 2023. URL https: http://xxiv..rrgg/./ 22300 04973.
[31] Jinwoo Kim, Janghyuk Choi, Ho-Jin Choi, and Seon Joo Kim. Shepherding slots objects: Towards stable and robust object-centric learning. In CVPR, 2023, URL https://arxivv org/abs 17842.
[32] Adeel Pervez, Phillip Lippe, and Efstratios Gavves. Differentiable mathematical programming for object-centric representation learning. In ICLR, 2023, URL httpss// net/ forumPid-1J-2IT7aypy
[33] Martin Engelcke, Oiwi Parker Jones, and Ingmar Posner. GENESIS-VE Inferring Unordered Object Representations without Iterative Refinement. In NewrIPS, 2021. URL. https:// openreview aeearreview..nt/forumli.eeee
[34] Michael Chang, Thomas L. Griffiths, and Sergey Levine. Object representations as fixed points: Training iterative refinement algorithms with implicit differentiation. In NeurIPS, 2022. URL https://arxiv org/ ab2/ 00787.
[35] Baoxiong Jia, Yu Liu, and Siyuan Huang. Improving object-centric learning with query optimization. In ICLR, 2023. URL httph::////aaa httppp////wwwooooaaaaa/////a////
www.oogg//////ee/ooo//o//eee 2210 08990.
[36] Jindong Jiang, Fei Deng, Gautam Singh, and Sungjin Ahn. Object-centric slot diffusion. In NeurlPS, 2023. URL https: / //trxxaavvvvv//////////eeeeaaal/ org/ arg//bs//i
[37] Ziyi Wu, Jingyu Hu, Wuyue Lu, Igor Gilitschenski, and Animesh Garg. Slotdiffusion: Object- centric generative modeling with diffusion models. In NeurlPS, 2023. URL https://arxiv. arg/abs/2300 11281.
[38] Zhipeng Bao, Pavel Tokmakov, Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Object discovery from motion-guided tokens. In CVPR, 2023. URL https:///axtxs org/abs/ 2303. 15555.
[39] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael G. Rabbat, and Nicolas Ballas. Masked Siamese Networks for Label-Efficient Learning. ECCV, 2022. https/tts///axii org brrg/bb//an 2204. 0711
[40] Xinlei Chen, Saining Xie, and Kaiming He. An Empirical Study of Training Supervisd Vision Transformers. ICCV, 2021. URL attps://arxivv www.:org/abblo htt//bss/1000 22000.00010.......
[41] Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin. Semantics meets temporal correspondence: Self-supervised object-centric learning in videos. In ICCV, 2023. URL https: :/tp:://////vvi// org/ abs/2308. 09951..1..099.......1111
[42] Görkay Aydemir, Weidi Xie, and Fatma Güney. Self-supervised object-centric learning for videos. In NeurIPS, 2023. URL attps://arzie
06907.
[43] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition Scale, In ICLR, 2021. URI. https: /ttpr///eeeeereeeeee///e wwwp:iiwwwweeeetmmimeeiee YicbFaNTTy.
13

---

[44] Matthias Tangemann, Steffen Schneider, Julius von Kügelgen, Francesco Francese Locatello, Peter Gehler, Thomas Brox, Matthias Kümmerer, Matthias Bethge, and Berhhard Schölkopf. Unsupervised object learning via common fate. In CLeaR, 2023. URL htt.tttt///rra...oogo/abbb// 06562.
[45] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as contrastive random walk. In NeurIPS, 2020. URL https://arxiv, hrggabbs/2006 14613.
[46] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In ICCV, 2019. URL https: /ttp///axvvvv/// org/ /tlb//955.. 04804.
[47] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. 2017. URL httts:///aaaiiii org/ /0040/0000.000555
[48] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV, 2014. URL https://arxiv org.iii..rrrraaeeee htt//bbs//444 0312.
[49] Jordi Pont-Tuset, Pablo Arbeláez, Jonathan T. Barron, Ferran Marques, and Jitendra Malik. Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation. IEEE Transactions Pattern Analysis and Machine Intelligence, 39(1), 2017. doi: 10.1
TPAMI.201..2537320 URL
ieee. ieee..rr/eeeeee.eeeeeee...e wrgg/oocumnn//4231111
[50] Jason Tyler Rolfe. Discrete variational autoencoders. /openreview rett/foruuridrryhhtPPa...
In ICLR, 2017. URL https:
[51] Christian Gumbsch, Martin V Butz, and Georg Martius. Sparsely changing latent states for prediction and planning in partially observable domains. In NeurIPS, 2021. URL https: //arxiv. /2100. 15949.
[52] Ziyi Wu, Nikita Dvoriik, Klaus Greff, Thomas Kipf, and Animesh Garg. Slotformer: Un- supervised visual dynamics simulation with object-centric models. In ICLR, 2023. URL nett/orrmrid--Troooll..
[53] Andrii Zadaianchuk, Maximilian Seitzer, and Georg Martius. Self-supervised Visual Re- inforcement Learning with Object-centric Representations. In ICLR, 2020. URL https:
/ //openreview..
[54] Davide Mambelli, Frederik Träuble, Stefan Bauer, Bernhard Schölkopf, and Francesco Locatello. Compositional Multi-object Reinforcement Learning with Linear Relation Networks. In ICLR Workshop on the Elements of Reasoning: Objects, Structure and Causality, 2022. URL https://openrevil wwt./oormmi
[55] Fan Feng and Sara Magliacane, Learning dynamic autributeffactoree world models for efficient multi-object reinforcement learning. In NeurIPS, 2023. URL https: org/abs/ 2307 .92055.
[56] Mohammadreza Salehi, Efstratios Gavves, Cees G.M. Snoek, and Yuki M. Asano. Time does tell: Self-supervised time-tuning of dense image representations. In ICCV, 2023. URL https://arxil: wttp//owiiii....../////o/o//...r abs/ 11796.
[57] Junyu Xie, Weidi Xie, and Andrew Zisserman. Segmenting moving objects via an object-centric layered representation. In NeurIPS. 2022. URL attts:://axiiv org/ 2b2/ 02206.
[58] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- -Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision, 2023, URL https: // /arxivv.v.. org/ abs 2b3/ 231.///334..000......
14

---

Supplementary Material for Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities
Comparison with Additional Baselines
16
A.1 Comparison with Image-Based Object-Centric Methods
16
A.2 Comparison with Concurrent Work Real-World Videos
16
Additional Experiments
17
B.1
Long-Term Video Consistency
17
B.2 Optical Flow Self Sulf-Suuer Target
18
B.3
Comprehensive Ablation Study on the MOVi-E Dataset
19 ................................
B.4
Stability of Mixer Decoder
19
B.5
Image-Based Feature Similarity on COCO
20
B.6 Sensitivity Number of Slots During Evaluation
.e..............................
20
B.7 Effect of VIT Architecture on Final Performance
20
C Architectural Details and Hyperparameters
20
C.1 SlotMixer Decoder
C.2 VIT Encoders as Dense Features Extractors
21
C33 Other Modules
23
C.4 Next-Frame Feature Prediction Details
24
C,5 Baselines
24
C.6 Hyperparameters
24
C.7 Compute Requirements
25
Dataset Details
25
Additional Examples
26
15

---

A Comparison with Additional Baselines
A.1 Comparison with Image-Based Object-Centric Methods
In this section, evaluate how effective model is for unsupervised image segmentation from videos. In addition to reporting results for the video-based object-centric methods SAVi and STEVE, we compare with several recent image-based object-centric learning methods. SLATE [26] is an image-basee basee object-centric model that trains a discrete VAE [50] as a dense feature extractor and a Transformer decoder conditioned on slots to reconstruct discrete representations of VAE features. LSD [36] replaces the Transformer decoder with a latent diffusion model conditioned on the object slots. Next, DINOSAUR [9] incorporates dense DINO features as targets and reconstructs the features itself. We report the Image FG-ARI and Image mBO metrics. They measure how well the predicted segmentation matches the ground-truth segmentation of given single image (frame), thus consistency over the video is not taken into account.
The results for MOVi datasets presented in Table A.I. VideoSAUR surpasses both previous image- and video-based methods, showing the benefits of using motion information in combination with semantically coherent supf-super features. Interestingly, VideoSAUR performs well on both Image FG-ARI and Image mBO metrics, whereas DINOSAUR and LSD seem to improve either in quality of split (measured in Image FG-ARI) in the sharpness of masks (measured in Image mBO) at the cost of performing worse on the second metric. We also note that LSD results are from larger resolution of MOVi images (256 256). We expect that method additionally improve if also use larger resolution videos; however, to be comparable with other baselines, we use 128 x 128 resolution this work.
In addition, we also compare VideoSAUR with DINOSAUR and other video-based base-
Image FG-ARI
40
SAVi
30
lines on the more challenging YouTube- VIS dataset (see Fig. A.1). VideoSAUR outperforms DINOSAUR (+4 FG-ARI) and also surpasses video-based STEVE and SAVi methods. This underscores the benefit of our temporal simi- larity loss over mere feature reconstruction for challenging real-world datasets.
STEVE
20-
DINOSAUR
10
VideoSAUR
Figure A.1: Image-based comparison on YouTube-
VIS (mean standard dev., seeds).
Table A.I: Comparison with state-of-the-ar methods the MOVi-C, MOVi-E image datasets. Both metrics are computed for individual frames. The results for SLATE and DINOSAUR are from Seitzer al. [9], while LSD results are from Jiang al. [36]. We report mean + standard dev. over 5 runs for our model.
MOVi-C
MOVi-E
Image FG-ARI
Image mBO
Image FG-ARI
Image mBO
Block Pattern
42.7
19.5
41.9
20.4
SAVi [5]
41.8
25.9
50.3
20.3
STEVE [7]
51.9
41.6
59.5
34.4
SLATE [26]
43.6
26.5
44.4
23.6
LSD [36]
50.5
46.3
53.4
39.6
SlotDiffusion [37]
60.0
30.2
DINOSAUR [9]
68.6
39.1
65.1
35.5
VideoSAUR
75.5 # 0.9
46.0 # 0.6
78.4 # 0.7
41.2 + 0.4
A.2 Comparison with Concurrent Work on Real-World Videos
In concurrent work. two more slot attention based methods were proposed that learn object-centte representations on real-world videos: SMTC [41] and SOLV [42]. SMTC learns to extracts objects from videos by enforcing semantic and instance consistency over time using student-teacher approach. SOLV extracts per-frame slots using invariant slot attention [30], applies a temporal consistency module and merges slots using agglomerative clustering; the model is trained using
16

---

Table A.2: Comparison of VideoSAUR with DI- NOv2 ViT B/14 features to SMTC [41] on the 2017-Unsupervised validation set. The results for SMTC are from Qian et al. [41].
Table A.3: Comparison of VideoSAUR with
DINOv2 ViT B/14 features to OCLR [57] and
SOLV [42] on YT-VIS 2019. The results for
OCLR and SOLV are from Aydemir et al. [42].
Method
J
F
J & F
Method
mloU
SMTC
36.4
44.6
40.5
OCLR
32.5
VideoSAUR
36.8
21.1
29.0
SOLV (wlo slot merging)
39.9
SOLV (w/ slot merging)
45.3
VideoSAUR
40.3
DINOSAUR-style feature reconstruction on masked out intermediate frames. In this section, compare to them using the respective evaluation protocols in their papers.
dataset [47]. Specifically, we assess our method's transfer performance when trained on YT-VIS 201, and follow the evaluation procedure outlined in [47] for matching ground truth masks predictions using mean J & F. We report the Jaccard index T (equivalent to IoU), the boundary F-score F and their average as J & F. While the contours of VideoSAUR's segmentation masks (as measured by FF) are not as accurate due to processing images and predicting masks at a lower patch resolution (37 x 37 for VideoSAUR trained with DINOv2 B14 features on original DINOv2 resolution), the Jaccard index J is comparable to SMTC.
First, compare to SMTC in Table A.2 the DAVIS-
Second, we compare to SOLV in Table A.3 on the YT-VIS 2019 dataset. We also list the performance of OCLR [57], which uses synthetic data with ground truth optical flow. As SOLV, we report the mloU metric matched over the entire video. The results for VideoSAUR are obtained using random 200 videos from the YT-VIS 2019 train split 4. Our results surpass OCLR, showing VideoSAUR effectiveness in extracting motion information directly from video data using dense self-supervised features. Additionally, our performance matches that of SOLV without using agglomerative clustering, while SOLV with slot merging outperforms VideoSAUR (+5 mloU). This highlights the importance of correctly determining the number of slots. While the main concern in this paper is to integrate motion information from video, we see determining the number of slots as important orthogonal direction. Thus, combining our method a solution such as the one from Aydemir et al. [42] is an interesting direction for futuree work.
B Additional Experiments
B.1
Long-Term Video Consistency
Beyond the initial examination of how our VideoSAUR performs on the relatively brief 6-frame video segments from YouTube-Vis VIS 2021, we extend our evaluation to also assess its effectiveness on more substantial, longer video segments. and full YT-VIS video segments (see Fig. B.1 for the distribution of video lengths). Although the performance of VideoSAUR on extended video segments predictably decreases in terms of FG-ARI, the observed difference between VideoSAUR and the baseline models is significant. This suggests that VideoSAUR maintains its efficacy tracking the primary objects in videos across longer time intervals.
Table B. we show the performance for 12-frame
Additionally, we investigate if VideoSAUR benefits from using DINOv2 features [58] that are obtained by training DINO on the larger dataset and fine-tuning the representation on larger resolution (518 x 518). We show that VideoSAUR performance benefits from using such features as backbone. especially in terms of mask quality (+6 mBO points). Using those features with the original resolution VideoSAUR reaching 29.7 mBO on full-Iength YT-VIS 2021 videos. In addition to this quantitative evaluation, we visualize VideoSAUR predictions on long YT-VIS videos (longer than 30-frames) in Figure E.5 and Figure E.6.
"We use part of the train split, as validation labels are not not released. Exact indexes are available at val idation split of the Tensorflow version of the YT-VIS 2019 dataset: https: //www. tensorflow. org/datasets/ catalog/youtube htt.yyouuuuutttttuuuuuueeeeeeeee wwwpruuuuuuuivsssleeeeeeeeemeeee
with_labels train_split
17

---

Table B.1: Performance of VideoSAUR on the YT-VIS 2021 dataset, varying the length of the video segment (mean + standard dev., 5 seeds for VideoSAUR with DINO features and STEVE. VideoSAUR with DINOv2 features are one seed only due to computational limitations).
frames
frames
Full Videos
FO-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
Block Pattern
24.0 14.9
20.3
14.2
15.1
13.1
STEVE [7]
20.0 1.5
20.9
0.5
18.0 1.4
2.5
15.0 0.7
19.1 0.4
VURe with DINO B/16
39.5 =
29.1
0.4
35.8 + 0.3
29.4
28.9 0.4
26.3 = 0.2
ViReOSAUR with DINOv2 B/14 [58]
39.7
35.6
38.7
34.5
31.2
29.7
35
Number of videos
30
25
20
15-
10-
5-
T
20
30
40
50
60
Video length, frames
Figure B.I: Histogram of video lengths on the YT-VIS 2021 validation dataset.
B.2 Optical Flow as Self-Supervised Target
The choice of the self-supervised target plays important role in creating suitable inductive biases for object discovery and scene decomposition. As such, understanding the properties of the prediction target leading proposed in this work combines two different inductive biases: high-level semantic information and motion information. To elucidate the significance of both types of bias for scene decomposition, assess the performance of VideoS AUR with prediction targets that consist only of one of those biases.
effective scene decomposition is crucial. The temporal feature similarity prediction
In Table and Table 3 in the main text, compare predicting temporal similarities with predicting self-supervised features of the current frame. Such features only contain semantic information, but no information about motion. Depending the dataset, including temporal information brings small (YT-VIS) or large benefit (MOVi-C).
Subsequently, we study if motion cues alone (without the semantic information) are enough for successful scene decomposition.
particular, we compare self supervuee temporal similarity targets with (ground truth) optical flow targets (only motion information) on the MOVi-C and MOVi-E datasets. To this end, we train VideoSAUR by predicting optical flow targets, using a spatial broadcast decoder similar to SAVi [5] instead of the mixer decoder. All other components of VideoSAUR stay unchanged. The optical flow map is predicted at full image resolution (128 x 128).
The results are presented in Table B.2, We train VideoSAUR with GT optical flow for the best potential performance from optical flow alone. Yet, even on the MOVi-C datasets favoring optical flow (no camera motion, no static objects), VideoSAUR with temporal feature similarities significantly outperforme optical flow (+10 FG-ARI). This disparity is even greater on the MOVi-E dataset, highlighting VideoSAUR resilience to camera movements and static objects. Together, these results demonstrate that our temporal feature similarity targets, despite not requiring signals such as optical flow (which would need estimation in real-world scenarios like the You VIS dataset), excel over mere optical flow targets. We attribute this the enriched semantic bias inherent to the self-supervised feature similarities.
18

---

Table B.2: Comparing VideoSAUR predicting temporal similarities to predicting ground truth optical flow on the MOVi-C and MOVi-E datasets. We report Video FG-ARI of version of VideoSAUR with optical flow (both backward and forward) as well as the original VideoSAUR with temporal features similarity.
VideoSAUR
MOVi-C
MOVi-E
w/ GT Optical Flow (backward)
48.1
28.9
w/ GT Optical Flow (forward)
48.9
30.1
w/ Temporal Similarities
60.7
73.9
Table B.3:
Extended ablation of VideoSAUR components on MOVi-E. We compare VideoSAUR model with different choices of the decoder (Mixer vs MLP used by DINOSAUR) and loss types (temporal similarity loss vs feature reconstruction).
Decoder Loss Type
FG-ARI
mBO
MLP
Feature Reconstruction
68.6
27.6
MLP
Temp. Feat. Sim. Prediction
74.5
28.8
Mixer
Feature Reconstruction
62.3
20.6
Mixer
Temp. Feat. Sim. Prediction
74.1
34.1
B.3 Comprehensive Ablation Study the MOVi-E Dataset
In this section, we present the results of comprehensive ablation study conducted on the MOVi-E dataset. The purpose this study to investigate the impact of two key factors: decoder choice (MLP vs. Mixer) and loss function selection. Our goal is to gain a deeper understanding of how these choices affect the performance of our method. The results are summarized in Table B.3.
The similarity loss is beneficial for both decoders, pushing the FG-ARI to approximately 74, as compared to 69 when using the feature reconstruction loss. Notably, the Mixer decoder significantly enhances the clarity of the object mask, with an improvement of +5 mBO. When combined, the similarity loss and Mixer consistently outshine the MLP decoder equipped with the feature recon- struction loss. These insights provide valuable auxiliary information to our main paper ablations (see Table and Table 3), painting a more detailed picture of VideoSAUR's components and their respective performances.
B.4 Stability of Mixer Decoder
As mentioned in Sec. 3.3 of the main text, found that the mixer decoder sometimes exhibits training instabilities. For instance, Table B.4 shows that there is high variance over random seeds when training purely with feature reconstruction, i.e. some training runs fail discover an object grouping. These instabilities manifest in slot masks that follow a Voronoi-like decomposition of the image. When adding the temporal similarity loss, the instabilities disappear.
We hypothesize this is because the mixer decoder has increased flexibility in how to model the image with slots (compared to the conventional nixture-based decoder), and thus more failure modes
Table B.5: Loss ablation study on COCO dataset. Metrics are image-based ARI and mBO (mean, seeds).
Table B.4: Mixer decoder with smaller DINO features (S/16) on YT-VIS 2021 (mean + stan- dard dev., seeds).
Loss type
Metric
Loss type
Metric
Feat. Rec.
Self-Sim.
FG-ARI
mBO'
Feat. Rec.
Temp. Sim.
FG-ARI
mBO
34,8
23,9
14.9 12.0
12.9
V
28.5
25.6
37.0 + 3.5
29.1 = 0.6
/
38.0
25.9
19

---

(non-object groupings) the model can "fall into" during training. Increasing the difficulty of the task by adding the temporal similarity loss makes these failure modes less viable: by putting more pressure the slot bottleneck to encode information, object- basee slot groupings are more efficient representations than alternative groupings.
However, we found that the mixer decoder with feature reconstruction does not show instabilities in all settings. For example, training with DINO ViT Base/8 features on MOVi-C or DINO Base/16 features on YouTube-VIS 2021 (Fig. 2 and Fig. in the main text) is relatively stable, We attribute this the increased task difficulty when predicting ViT "Base" features instead of ViT "Small" features (as in Table B.4), Once more [5, 6, 9, 26, 29I, these findings demonstrate the central lesson of unsupervised object discovery: to be successful, the model needs to have sufficient inductive biases, whether they stem from the dataset, the decoder, the grouping module, or the training task.
B.5 Image-Based Feature Similarity on COCO
In this section, show that our proposed similarity loss can also be useful for image-basek datasets, and thus is not restricted to just the video setting. Note that by setting the time-shift k to O. the temporal similarities turn into self-similaritiess, that is, the target similarities are computed by comparing features from the same image. The resulting similarity maps highlight semantically and spatially related patches and thus could be useful targets to discover objects.
To test this, we train the DINOSAUR method with VIT S/16 [9] with the self-similarity loss on the real-world COCO dataset (see Table B.5). Similar to the results from the time-shift analysis (see = 0 in Fig. 6b), we find that using the self-similarity loss alone does not seem to carry enough signal to train the model and leads to degraded performance. However, combining the self self-similarity loss with feature reconstruction shows significant improvement over using only feature reconstruction. Even though the targets contain the same information overall, different transformations of the original targets (e.g. their relative similarity) create different biases - a combination of these different views into the targets appears be beneficial for object discovery.
B.6 Sensitivity to Number of Slots During Evaluation
One of the noteworthy properties of slot attention- based models with randomly sampled initialization" is that the number of slots can be adjusted during inference. As we demonstrate in Fig. 6b, this is helpful for successfully transferring to datasets that have a different average number of objects per video. For this purpose, it is important to examine how stable the model's performance is if it is used with a different number of slots than during training. Thus, we evaluate our model (trained on YT-VIS 2021 with k- = 7 slots) using varying number of slots (from I to 12) and present the results in Fig. B.2. We observe that while using fewer slots steadily deteriorates the performance, our method performs relatively well with larger number of slots. This suggests that method is relatively robust to the usage of a larger number of slots than needed. This property is useful for object discovery in images where the number of objects is unknown.
B.7 Effect of ViT Architecture on Final Performance
In addition to studying how the choice of self-superivsed method and ViT outputs affect the per- formance of VideoSAUR (see Fig. 6c and Fig. 6d in the main paper), we also explore the effect of the scale of VIT architecture on the performance. To this end, we compare VideoS AUR with DINO features of 2 different ViT sizes (Small and Base) and two different patch sizes (16 x 16 and 8 x 8 resolutions) on the MOVi-C dataset (see Table B.6). The results are presented in Fig. B.3. We find that smaller patch size is important for sharper masks (measured by the mBO metric), while both larger architecture and smaller patch size are important for a better split of the scene to object masks (measured by the FG-ARI metric).
C Architectural Details and Hyperparameters
Here we describe details about our model, its training and baselines that we use for comparison. We release code at https: // //gitttttt ggggthubbb... com martius-- -irtuluusabb/// laabvvideosau
'This is in contrast to the fixed learned initialization used in unconditioned SAVi [5] and SAVi++[6].
20

---

C.1 SlotMixer Decoder
We describe the SlotMixer decoder, and how we adapted for 2D decoding. We keep the original terminology from Sajjadi et al. [13] for consistency. SlotMixer performs three steps for decoding: the allocation step assigns slots to spatial positions, the mixing step creates slot mix for each spatial position, and the render step decodes the slot mix to the final output. See Fig. C.I for an overview and pseudocode implementing the decoder.
Allocation Step This step takes as input the slots 8; E RRKM and learned positional embedding p 4 and outputs feature vector f E RLXM using a cross-attention Transformer. In partieular, this Transformer iterates several cross-attention operations (and applies residual two-layer MLPs) using the position embeddings as queries to attend into the set of slots, where the position embeddings are residually updated using values from the slots. Importantly, the position embeddings are processed independently of each other. We utilize pre-normalization and also apply layer norm the slots before feeding them into the Transformer. In contrast to Sajjadi et al. [13], we use learned positional embedding initialized from a normal distribution instead of 3D encodings for the query rays, and do not apply a MLP to the positional embedding.
Mixing Step The mixing step is similar to single-head attention step using the features as queries and the slots 8, as keys, where the slots are averaged as the values to form the slot mix m E I2 4xM that is used for decoding to the final output:
norm ( Un
Ug E RRMM
norm (8) Uk
UR E MRM
A softmax qk T / //M
A E RL*K
m 8A
m E RLXM.
Render Step The render step takes the slot mix m E RL M adds the positional embedding p R LxM and applies MLP with ReLU activation independently to each position:
y MLP(m + +p)
Instead of adding the positional embedding, we also explored concatenating it to the slot mix; we did not find large differences from doing so.
C.2 VIT Encoders Dense Features Extractors
The Vision Transformer (ViT) [43] architecture takes input frame, denoted here 244x2244x which is divided into grid of non-overlapping contiguous patches of resolution N x N. Each of these patches is then passed through a linear transformation to generate a set of patch feature embeddings, ht BLXD
The set of patch tokens subsequently provided as input to a standard Transformer network. This Transformer network is comprised of sequence of self-attention and feed-forward layers, alongside
40
40
30
30
FG-ARI
20
10
10
2 3
4
5
7 8 9 101112
2 3 4 5 6 7 8 9 101112
Number of slots
Number of slots
VideoSAUR
STEVE
Random pattem
Figure B.2: Changing the number of slots during evaluation on the YT-VIS 2021 dataset (mean + standard dev., 5 seeds).
21

---

FG-ARI
60
Small 16x16
Table B.6: ViT networks configuration.
40
mB0
Base 16x 16
Small 8x8
Model Patch Size
Dim Heads
Tokens
Params
Base 8x8
Small
16
384
6
196
21M
Small
384
784
21M
Figure B.3: Effect ViT architecture choice and patch size VideoSAUR training the MOVi-C (mean + standard dev., seeds).
Base
16
768
196
85M
Base
8
768
12
784
85M
A
P
P
0.00.51.0
0.0 0.025 0.05
Figure B.4: Additional visualization of affinity matrix A, transition probabilities P and decoder predictions of transition probabilities P between patches (marked by purple and green) of the frame H; and patches of the next frame x1+1 for YouTube-VIR VIS 2021 validation vidcos. Red indicates maximum affinity/proaability.
residual connections for each layer. These residual connections are important for letting each patch representation h' keep a correspondence to the original image patch representation h" and thus making the patch representation a dense (with the resolution VL x vL) representation the image. In the self-attention layers, the token representations are updated through an attention mechanism that takes into account the representations all tokens:
|q' k' v
gkv
UgKV
F DX3D
softmax q'k /VD//D)
A E LXL
o' Av
Here q', k', v of are queries, keys, values and outputs of the self-attention layer In contrast to DINOSAUR [9], which uses the outputs o' as the target image representation, use attention keys
22

---

21
31 41 51
Input
VidcoSAUR
Figure B.5: Failure case for long video prediction. Note that the original videos in YouTube are resampled from the original 30 fps to 6 fps, thus the original length of the video is 250 frames. The slots are reassigned the background, while small objects are not recognized.
pos_ emb Paran (randon, Cor.ddo dim))
Slots
Position Embed. p
def mixer (slots):
ba. a_slats, poa_emb pos_enb. unaquesee (0)
slots. shape
expand (bs, -1, -1)
Allocation Transformer
# Step 1: Allocation Transforuer feate za_tranef (q=pos_enb.
10
kr=norn,kv(slots))
12
# Step Slot Mixing
Slot Mixing
13
q linear (aaaar_gnnoongtttattta linear_ (norn, (alota))
15
dots einsum ("a ("bpd, bad -> bps",
9. dots ** [-]] 17
18
atta softmax (dota, dim=-1) slot_mix einsum einaun("bpe, bed bpd", attn, alota)
Render MLP
19
30
22
N Step 3: Rendering with MLP slot pos_emb
Predicted Output y;
24
outputa mip(slot_nix)
26
Figure C.I: SlotMixer decoder. Left: SlotMixer performs three steps for decoding: the allocation transformer assigns slots 8, to spatial positions the slot mixing step creates slot mix for each spatial position, and the render MLP decodes the slot mix to the final output y: Right: PyTorch- pseudocode for the SlotMixer decoder.
k' from the last self-attention layer of ViT as the dense image representation that is provided to the temporal similarity loss (see Fig. 6e for detailed comparison of these representations). However, we still use the outputs o' as the input to the slot attention grouping module.
C.3 Other Modules
We group the dense encoder features with recurrent slot attention module similar to Singh al. [7] and Kipf et al. [5]. First, we transform the original features with a two-layer MLP with an output dimension equal the slot dimension. Second, we use slot attention module initialized with randomly sampled slots to group the first frame features, while for subsequent frames, we initialize the slot attention module with the slots of the previous frame, additionally transformed with predictor module. We use the GRU recurrent unit in the slot attention grouping, but not the residual MLP. Similar to SAVi [5] and STEVE [7], we use a one-layer transformer as the predictor module. In addition, propose to decouple the number of Slot Attention iterations in the first frame and other frames the video. This allows more iterations the first frames (we use iterations similar to image-basee based methods) and fewer iterations for the next frames where the initialization is much
23

---

Table C.I: Next-frame feature prediction with different decoders.
MOVi-C
MOVi-E
YT-VIS
FG-ARI
mBO
FG-ARI
mBO
FG-ARI
mBO
One decoder head
44.6
23.5
61.3
22.1
33.4
24.6
Two decoder heads
47.2
24.7
62.9
24.0
37.9
27.3
better (we use 2 iterations). For computational reasons, training relatively short 4-frame segments of original videos, i.e. T 4.
C.4 Next-Framm Feature Prediction Details
In this part, we cover implementation details for the next-frame frame feature prediction ablation presented in Table 2 and in Table 3. Reconstructing frame features from the current and next frame simultancously with single decoder is problematic because the decoder masks that are used for evaluation would be in reference to both the current and next frame. One way to overcome this problem is by using two decoder heads: deurren for the current frame and dnex for the next frame. Each head produces its own predictions and masks. In this case, masks from the deurren head can be used for evaluation. While more powerful, this approach also requires more memory and is slower than standard setting with only one head. In our experiments, we confirm that the version with two different Mixer decoders performs better than simultaneous reconstruction with one decoder (see Table C.I). We use this better version for our comparisons even though it heavier than our method which needs only one decoder.
C,5 Baselines
Block Pattern The block pattern baseline serves to show metrics for trivial decomposition of the video into regular blocks. is intended to show the difficulty of the dataset and how much object-centric methods improve upon such a trivial solution, the metrics values could be difficult to interpret without further calibration. To this end, we are splitting the video to spatial blocks consistently for all frames of the video. similar to how Seitzer al. [9] are splitting images into regular blocks.
SAVi We reimplement SAVi [5] close to the official implementation' In particular, we use the SAVi-L architecture for all experiments. This corresponds to the version using ResNet-34 encoder as described by Kipf et al. [5], and a CNN broadcast mixture decoder with 4 layers. We apply the unconditional version of SAVi, using fixed learned slot initialization instead. We train the model for 200 000 steps on all datasets, with batch size of 64, using image reconstruction as the training signal. For training, we use videos with 4 frames, with a single slot attention step per frame.
STEVE
We reimplement STEVE [7] close to the official implementation? We use the proposed configuration for the MOVi datasets and only change the number of slots to 11 for MOVi-C, 15 for MOVi-E and 7 for YT-VIS 2021. STEVE trains dVAB [50] the video frames to extract discrete latent code that is used as the reconstruction target. STEVE uses a CNN encoder with layers and a Transformer decoder with 8 layers. We train the model for 200 000 steps for MOVi-C and YT-VIS datasets and for 100 000 steps for MOVi-E datasets which resulted in optimal performance on this dataset, with a batch size of 24. Like Singh al. [7], for training, we videos with 3 frames, with two slot attention steps per frame
C.6 Hyperparameters
The full list of the hyperparameters used to train VideoSAUR on MOVi-C, -D, -E, and YouTube- VIS 2021 datasets are presented in Table C.2. Most of the hyperparameters are similar to previous methods, while new ones (such as softmax temperature a and time shift k) are optimized using a grid search.
24

---

Table C.2: Hyperparameters of VideoSAUR for the main results on MOVi-C, MOVi-E, and YouTube-
VIS 2021 datasets.
Dataset
MOVi-C
MOVi-E
YouTube- VIS
Training Steps
100k
100k
100k
Batch Size
128
128
128
Training Segment Length
4
4
4
LR Warmup Steps
2500
2500
2500
Optimizer
Adam
Adam
Adam
Pcak LR
0.0004
0.0004
0.0004
Exp. Decay
100k
100k
100k
VIT Architecture
ViT Base
VIT Base
ViT Base
Patch Size
8
8
Feature Dim. Dieat
768
768
768
Gradient Norm Clipping
0.05
0.05
0.05
Image/Crop Size
224
224
224
Cropping Strategy
Full
Full
Rand. Cent. Crop
Augmentations
Rand. Hori. Flip
Image Tokens
784
196
Slots
11
7
Iterations (first other frames)
3/2
3/2
3/2
Slot Attention
Slot Dim. Datass
128
128
64
Type
Transformer
Transformer
Transformer
Predictor
Layers
1
1
I
Heads
4
4
4
Type
Mixer
Mixer
Mixer
Allocator Transformer Layers
2
2
3
Decoder
Allocator Transformer Heads
4
Renderer MLP Layers
4
4
3
Renderer MLP Hidden Dim.
1024
1024
1024
Softmax Temperature т
0.075
0.075
0.25
Loss
Time-shift &
1
I
Feature reconstruction weight
0.1
C.7
Compute Requirements
We used cluster of A100 GPUs (with 40 and 80 Gb memory) for running the experiments. A single training run (100k steps) of VideoSAUR equipped with DINO B/16 with a batch size of 128 takes roughly 18 hours on one A100 GPU with 40 GB memory. We use seeds for the final results and 3 seeds for other experiments. Overall, we estimate the total compute spend on the whole project including training of baselines and the method development (including dead ends) to be around 800-1000 GPU days (this is rough estimate obtained from our cluster usage).
D Dataset Details
In this section, provide details about the datasets used in this work. See Table D.1 for overview.
MOVi datasets For MOVi-C, MOVi-D and MOVi-E, we use the standard training and test splits provided in the respective releases of MOVi datasets: 9750 training videos and 250 validation videos. For a fair comparison, all the methods are trained on images with the same initial resolution 128 x 128. Consistent with previous work [5-7, 9], we utilize the validation split of the MOVi dataset for our evaluations.
YouTube-VEE datasets The YouTube-Vil VIS datasets [14, 46] are benchmarks originally used for supervised video instance segmentation. The video instance segmentation involves segmenting, tracking and classifying instances throughout video. It is a challenging dataset because it contains
25

---

Table D.1: Overview of datasets used in this work. For the training process, solely utilize images or videos derived from the relevant datasets, with no reliance on labels. To generate central crops, we initially resize the mask so that its shorter dimension is 224 pixels. Subsequently, extract the most centrally located crop, maintaining a size of 224 by 224 pixels.
Dataset
Videos Images
Description
Citation
MOVi-C, -D, -E
9750
Train split videos
Greff et al. [12]
MOVi-C. DD. -E validation
250
Val. split w. instance segm. labels
Greff et al. [12]
COCO 2017 COCO 2017 validation
118
287
Train split
Lin et al. [48]
5000
Val split w. instance segm. labels
Lin et al. [48]
YouTube- VIS 2021
2785
Part of train split videos
Yang et al. [14]
YouTube- VIS 2021
210
Part of train split w. instance segm. labels
Yang ct al. [14]
YouTube- VIS 2019
200
Part of train split w. instance segm. labels
Yang et al. [46]
DAVIS-2017-Unsuppee
30
Val. split w. instance segm. labels
Pont-Tuset et al. [47]
various different classes of objects and the complexities of world video dynamics. To the best of our knowledge, this is the first work attempting unsupervised instance segmentation on YouTube-Vir VIS.
There are two different versions of this dataset: YouTube- 2019 and YouTube- VIS 2021. The YouTube VIS 2019 dataset is mainly derived from the Video Object Segmentation (VOS) dataset. It has limited number of individual instances (an average of 1.7 per video for the training set), and the categories of instances in the same video are usually different. In contrast, the 2021 edition of YouTube-VV VIS incorporates a higher quantity of objects with more difficult trajectories (average 3.4 per video for the additional videos in the train set) and thus are more interesting for object-centric learning. We sample both training and validation frames with a rate of 6 frames per second (each 5th frame of the original videos with 30 fps).
As the original validation dataset is publicly available, and the evaluation server does not compute the object-centric metrics we need, we split the original training set into two parts (210 videos for validation and the other videos for training). Overall, we 2775 videos for training, and 210 videos additionally added in YT-VIS 2021 train for validation.
DAVIS dataset The DAVIS-2017-B nosu-uusuper dataset [47] is used for video object segmentation and also contains videos with multiple objects per video (average is 1.97 for 30 validation videos). As those videos are not related to YouTube-VII videos, this dataset is useful for the evaluation of transfer abilities for real-world object-centric learning algorithms.
COCO dataset To test the properties of our proposed similarity loss for image-based object-centric learning we use the COCO dataset [48]. Similar to Seitzer et al. [9], we use 118287 training images (without labels) train and 5 000 validation images (with labels) for performance evaluation.
E
Additional Examples
We include additional example predictions of our model:
Figure E.1: comparing VideoSAUR to STEVE on Youtube- VIS 2021.
Figure E.2: comparing VideoSAUR to STEVE on MOVi-C.
Figure E.3: comparing VideoSAUR to STEVE on MOVi-E.
Figure E.4: comparing feature reconstruction and temporal similarity loss on MOVi-C.
Figure E.5: long-term video predictions on Youtube VIS 2021 (VideoSAUR with DINO B/16 features).
Figure E.6: long-term video predictions on Youtube- VIS 2021 (VideoSAUR with DINOv2 B/14 features).
26

---

t=0
3
9
12
15
Input
VideoSAUR STEVE
Input
VideoSAUR STEVE
Input
VideoSAUR STEVE
Figure E.I: Additional examples YouTube-Vii VIS 2021.
27

---

t-0
3
9
12
15
Input
VideoSAUR STEVE
Input
VideoSAUR STEVE
Input
VideoSAUR STEVE
Figure E.2: Additional examples MOVi-C.
28

---

3
9
12
15
Input
VideoSAUR STEVE
Input
VideoSAUR STEVE
Input
VideoSAUR STEVE
Figure E.3: Additional examples MOVi-E.
29

---

9
12
15
18
21
Input
Feat.
Sim.
Input
Feat.
Sim.
Input
Feat.
Sim.
Input
Feat.
Sim.
Input
Feat.
Sim.
Figure E.4: Difference between VideoSAUR train with feature reconstruction and temporal feature similarity losses on MOVi-C. We show videos with larger differences in performance between the two methods.
30

---

5
10
15
20
25
30
Input
VideoSAUR
Input
VideoSAUR
Input
VideoSA UR
Input
VideoSAUR
Input
VideoSAUR
Input
VideoSAUR
Figure E.5: Prediction of VideoSAUR with DINO B/16 features on longer videos from YouTube-Va VIS 2021.
31

---

5
10
15
20
25
29
Input
VideoSAUR
Input
VideoSAUR
Inpul
VideoSAUR
Input
VideoSAUR
Input
VideoSAUR
Input
VideoSAUR
Figure E.6: Prediction of VideoSAUR with DINOv2 B/14 features on longer videos from YouTube- VIS 2021.
32


Reviews:
[{"review_id": "C3Hd3hxTtx", "reviewer": "Reviewer_avhJ", "summary": "The paper considers the problem of unsupervised video-based object-centric learning. It incorporates a temporal feature similarity loss that encodes temporal correlations and introduces a motion bias for object discovery. This loss helps to achieve state-of-the-art performance on the synthetic MOVi dataset. The model is able to learn video object-centric representation on the YouTube-VIS dataset in a fully unsupervised way. ", "questions": "No additional questions.", "limitations": "The limitations are briefly discussed at the end of the paper. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The paper is very well-written and it is pleasant to read. The visualizations and graphics are particularly nice and done with good care.\n2) The proposed VideoSAUR method significantly improves performance on synthetic video datasets over related works of SAVi and STEVE. It also to learn video object-centric representation on unconstrained real-world videos of YT-VIS.\n3) The ablation study is extensively done. It considers all the main parts of the method along with the most important hyperparameters. ", "weaknesses": "I enjoyed the paper and I think that the paper is already in good shape for acceptance. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback! We are glad you find the paper to be “very well-written” and “pleasant to read”. We hope that the reviewer would find additional experiments requested from other reviewers interesting. Their description could be found in the general response. "}, {"review_id": "An4sj38eaO", "reviewer": "Reviewer_tzwP", "summary": "This paper proposes VideoSAUR for unsupervised video object segmentation / grouping. The key idea proposed in this paper is to use a temporal feature similarity loss, in combination with a feature reconstruction loss. The grouping is implemented with recurrent Slot Attention. Experiments are conducted on various benchmarks (MOVi-C, MOVi-D, MOVi-E, YT-VIS-2019, YT-VIS-2021, and DAVIS) with good improvement over existing methods. Written presentation is clear and easy to follow.", "questions": "* As mentioned above, I would be interested in seeing (1) how this proposed approach works on more challenging datasets; (2) how this method deals with extremely long videos (when will it fail); (3) any discussion on how to deal with long videos would be nice.", "limitations": "* The reviewer does not foresee any potential negative societal impact of this work.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper has strong experimental results: (i) solid improvements over existing methods; and (ii) a comprehensive set of ablations with different insights about the problem as well as the proposed method.\n\n* Written presentation is clear, consistent and easy to follow and understand.\n\n* The idea of using temporal feature similarity is not new (Vondrick et al ECCV'18, Vondrick et al CVPR'16, may be some others), but this paper shows an effective way to use it with self-supervised features, and recurrent Slot-Attention, yield strong results in unsupervised video-object segmentation. ", "weaknesses": "* I don't find any issue with this paper, may be further evaluate on more challenging dataset such as UVO will make this work srtonger.\n\n* There may be a stress test to push the method into an extreme setting, e.g., a very long video of hundreds of frames, to see when the recurrent Slot Attention will be failed. This may provide hints for future work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their very positive feedback! We are glad you like the paper for its “strong experimental results” and that you find the paper “clear, consistent and easy to follow”. We answer your questions below.\n\n> How does the proposed approach works on more challenging datasets?\n> \n\nWe agree that taking VideoSAUR to even more complex datasets is very interesting! Similarly, it would also be intriguing to see how the method scales with increasing dataset size. However, given that we already are the *first* unsupervised object-centric method to use the Youtube-VIS dataset, we think that this is out of scope for this paper.\n\n> Dealing with long videos? When will it fail? How to deal with long video?\n> \n\nWe thank the reviewer for the suggestion. Indeed better performance on long videos is one of the potential future directions for our method.\n\nAs we already discussed in Appendix B.1 and the Conclusions section there are several limitations of our current approach on long videos. First, the number of slots is static and needs to be chosen a priori, whereas, in the long videos, the objects frequently approach and disappear from the scene. Next, slots can get reassigned to each frame and can bind to different objects or the background. We visualize those failures on a long video (see in **Figure R1**). \n\nTo overcome these limitations, several innovations are needed. First, it is important to discover and maintain objects’ slots that are currently not visible in the image. This could be done with memory mechanisms and maintaining if each slot is active or not as an additional external variable (similar to $z_{present}$ in AIR-based [1] object-centric methods). Next, a slot re-identification module that matches any new objects to previously discovered, but not visible ones could further improve the performance on long videos. Finally, learning a generative latent object-centric world model (with set-based latent dynamics) could further improve the consistency of slots for object-centric representations on long videos.\n\n[1] Attend, Infer, Repeat: Fast Scene Understanding with Generative Models"}, {"review_id": "RXuWLKRrVK", "reviewer": "Reviewer_L1X2", "summary": "The paper aims to learn an effective object-centric feature representation for video segmentation. They build their method upon the slot attention-based framework and self-supervised ViT encoders (DINO), and propose a temporal feature similarity loss for object-centric learning. Their proposed method yields the best performance on several synthetic and real-world datasets, for both unsupervised object discovery and downstream video object segmentation tasks.", "questions": "Other questions about details:  \n- (L38) How does the model \"groups patches with similar motion into the same slot\"? Neither the temporal loss nor the reconstruction loss introduces such a grouping signal.\n- (L112) how are the slots $S_0$ initialized? Are they learnable, or randomly sampled from Gaussian?\n- Why the performance of VideoSAUR and several baselines have better performances on MOVi-E than MOVi-C as MOVi-E should be more challenging with both moving and static objects, and ego motion? Moreover, some previous works (e.g. [1,2]) have reported worse results of SAVI++ (unconditional version) on MOVi-E, given that SAVI should have even worse numbers compared with SAVI++, I am curious why the reported numbers of SAVI are higher? Especially for the comparison with [1] given that VideoSAUR is developed based on [1]. Is it because of the number of slots?  \n- Conceptually, what's the role of slot mixing under the SlotMixer? It is like an additional layer of the cross-attention union (Allocation Transformer)\n\nRef:\n[1] Bridging the Gap to Real-World Object-Centric Learning: https://openreview.net/forum?id=b9tUk-f_aG.  \n[2] Object discovery from motion-guided tokens: https://arxiv.org/abs/2303.15555  \n\n\n### Justification for rating\nThough the proposed method has some limitations, I do think the proposed VideoSAUR is interesting. Currently, I hold a borderline acceptance. If the authors could fully address my first concern in the \"Weakness\" section, I am willing to upgrade my rating. Will also refer to other reviewers' comments.\n\n---\nThe rebuttal has addressed my concern so that I upgrade my rating.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "+ The paper is well-organized and easy to follow\n+ The proposed temporal feature matching loss is sound and effective\n+ The experimental design is comprehensive and promising\n+ The proposed VideoSAUR model yields state-of-the-art performance on several datasets", "weaknesses": "I have two minor concerns regarding the method and experimental design:  \n- One of my concerns is the use of the pre-trained ViT encoder. The majority of the experiments are evaluated with a DINO encoder, however, DINO is pre-trained on ImageNet and DINO itself is \"object-centric\" (see https://github.com/facebookresearch/dino/). Though the authors have already clarified that they aim to bridge the gap between the pre-trained feature encoder and real-world video object discovery, it is still interesting to see if the proposed loss is useful *without* prior knowledge. It would be useful to add an experiment that uses a self-supervised ViT ***which is only pre-trained on the target dataset, e.g. MOVi-E***. Any self-supervised pre-training (DINO, MAE, MSN or MOCO) should be fine. This additional experiment could also give a fair comparison to other models as they, at least STEVE and SAVI, did not get access to extra data for pre-training.    \n- Novelty is the other concern. Though the authors claim the temporal feature matching loss is novel, the overall idea is adapted from contrastive random walk [41]. The proposed normalization function (Eq. 4) is different from the original one, but ignoring the negative-scored patches should not bring a big difference in the final similarity score. Besides the loss, the mixer decoder and reconstructing in the flow space have also already been verified to be helpful for object-centric learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your detailed review and interesting suggestions to improve our paper. Below we address your questions. We hope that our additional experiments will lead you to consider upgrading your rating.\n\n> One of my concerns is the use of the pre-trained ViT encoder. […] It is still interesting to see if the proposed loss is useful *without* prior knowledge. It would be useful to add an experiment that uses a self-supervised ViT *which is only pre-trained on the target dataset, e.g. MOVi-E*.\n> \n\nThanks for this suggestion. Indeed, a strength of our method is that it does not require any additional inputs other than images, so it is natural to test how much the ImageNet data bias is needed. To test this, we pre-train a ViT-B/16 encoder with the Masked Auto-Encoder (MAE) approach for 200 epochs on the MOVi-E dataset using standard hyperparameters, and then use it for training VideoSAUR (referred to as MAE+MOVi-E) with temporal features similarity loss.\n\nThe results are presented in **Table R4**. Interestingly, we observe that VideoSAUR is able to use such features for discovering objects in MOVi-E, reaching 70 FG-ARI, while not using any external data. We also use *MAE+MOVi-E* features to train VideoSAUR on MOVi-C where it reaches similar results (59.8 FG-ARI) as VideoSAUR trained with standard *MAE+ImageNet* features. Thus, even without any external data, VideoSAUR still outperforms the SAVi and STEVE baselines. Furthermore, we expect that tuning MAE hyperparameters could further improve the results. \n\n> Novelty is the other concern. Though the authors claim the temporal feature matching loss is novel, the overall idea is adapted from contrastive random walk [41].\n> \n\nWe indeed were inspired by a contrastive random walk (CRW), and we do use a similar construction of the affinities, as the reviewer correctly points out. But both the loss function and the overall learning setup in CRW are different from our work. While the supervised loss $L_{sup}$ in CRW (Eq. 3) does look similar to our loss $L^{sim}$ (Eq. 5), there are fundamental differences: in $L_{sup}$, *the affinity matrix is optimized*, while in our approach $L^{sim}$, the model is trained *to predict a fixed affinity matrix*. Thus, in CRW’s $L_{sup}$, the loss is used to maximize feature similarities between matching nodes, whereas in our approach, we train a slot representation by predicting pre-constructed self-supervised feature similarities, and use the structure in these similarities to induce an object grouping. We think that this is sufficiently different from CRW (both conceptually and mathematically) to form a novel contribution.\n\n> Besides the loss, […] reconstructing in the flow space have also already been verified to be helpful for object-centric learning.\n\nWhile optical flow prediction has indeed been shown to be beneficial for object-centric learning, we demonstrate the novel insight that temporal feature similarities can offer similar benefits but require no flow annotations, while working sensibly for static objects and being more robust to camera motion. The novelty over optical flow prediction thus lies in broadening the kind of datasets the method can be applied to.\n\n> How does the model \"groups patches with similar motion into the same slot\"?\n\nSlot attention-like models essentially learn groupings that lead to efficient reconstructions under the restricted capacity of the latent slot bottleneck. When reconstructing RGB images, these groupings e.g. capture areas of similar color, because the information needed to reconstruct such an area is efficient to represent in a single slot. Analogously, something similar happens under our temporal similarity loss: when groups of patches have similar motion patterns (temporal feature similarities), it is most efficient to represent them in the same slot in order to predict the similarities well. The same principle holds for optical flow prediction as well. Does this sufficiently address your question?\n\n> How are the slots initialized? Are they learnable, or randomly sampled from Gaussian?\n\nFor all VideoSAUR experiments, we use i.i.d. sampling from a Gaussian with learned mean and variance as slot initialization (see L113). This way learned object representation is invariant to the object order and also more flexible as the number of objects can be varied during inference. \n\n> Why the performance of VideoSAUR and several baselines have better performances on MOVi-E than MOVi-C [..] why the reported numbers of SAVI are higher? Especially for the comparison with [1] given that VideoSAUR is developed based on [1]. Is it because of the number of slots?\n\nAs we are using 15 slots for VideoSAUR and the other baselines on MOVi-E, their performance may differ from [1] and [2]. Secondly, [1] is evaluating a video-based model on 1-frame videos, which could be a disadvantage for SAVi as it was trained on many-frame videos. In contrast, we evaluate SAVi on the whole video, leading to better results. Similarly, for MOVi-C vs. MOVi-E, we think this also is due to the number of slots, as changing the number of slots can significantly change the results (e.g. Table 13 in [1]).\n\n> Conceptually, what's the role of slot mixing under the SlotMixer?\n\nSlot mixing creates a feature vector for each spatial position (patch) that is used to reconstruct that spatial position. This is done by taking a convex combination of the slots, using different weightings for different spatial positions. The weights are computed by a dot-product between the slots and the outputs of the allocation transformer. This operation is equivalent to a single-head attention step that uses the “slots” as “values”, so indeed this could be seen as just an additional (but special) layer of the allocation transformer. We give more details on this in Appendix C.1."}, {"review_id": "tg10luCHZr", "reviewer": "Reviewer_XQ9f", "summary": "The authors combine several recent approaches for object-centric learning. In particular, they use the overall framework which is a combination of SAVi with a slot mixer decoder from [13]. The main objective is optical flow prediction, like in SAVi, but they combine it with DINO feature reconstruction from Dinosaur [9]. They also adopt a pre-trained and frozen DINO backbone from Dinosaur. The only novel component in this framework is that they estimate the ground truth optical flow using the model itself by computing similarity between encoder features in consecutive frames. The way this is done is also adopted from prior work (CRW [41]). Another delta with respect to SAVi is that they predict the future optical flow t -> t + 1 at time time t, in contrast to the standard flow estimation objective (t -1 -> t), but the ablations don't show whether this makes a difference. \n\nIn the experimental evaluation on MOVi-C and MOVi-E they outperform some baselines but do not compare to the sota methods [9, 36]. They additionally report results on the real world YouTube-VIS benchmark where they outperform the same baselines, but the results are low for all the methods. Ablation demonstrates that on the real videos most of the performance is achieved by the Dino feature reconstruction objective from [9] and the flow prediction objective is fairly ineffective in the real world.", "questions": "Report results of [9] on YouTube-VIS. \n\nReport a full ablation on MOVi-E starting from Dinosaur and arriving to the proposed approach (ablate both the decoder, the new objective and all the changed hyper-parameters).\n\nReport results for the variant of the mode that uses gt/estimated optical flow directly in the objective.\n\nAblate whether predicting flow into the future makes a difference compared to the variant of the objective used in SAVi.", "limitations": "Limitations are addressed in the manuscript.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed approach seems sound.\n\nA relatively thorough ablation study is provided.\n\nThe paper is relatively well written in terms of the language (but not the notation or organization).\n\nState-of-the-art results are reported on the MOVi-E benchmark.", "weaknesses": "The proposed approach is merely a combination of several recent techniques for object centric learning. This wouldn't necessarily be a problem in itself, but the authors are not being fully upfront about it. In particularly, they do not explain that their 'feature similarity loss' is equivalent to optical flow prediction from SAVi (though they do show in Figure 3 that the learned feature similarities are effectively optical flow). In any case, the novelty of the proposed framework is minimal.\n\nThe authors do not compare to the state-of-the-art video-based and image-based approaches ([36] and [9]) in the main paper. They do compare to [9] on MOVi in the supplementary, but not on YouTube-VIS where that method is expected to be most effective (see ablation in Table 3). According to results on MOVi-C from the supplementary of [36] it outperforms the proposed approach. Thus it only seems to show  top performance on MOVi-E. \n\nThe results on YouTube-VIS are low for all the reported approaches. The numbers seem to indicate that the proposed methods simply fails less catastrophically than the baselines in the real world. \n\nAlthough a fairly thorough ablation study is provided it is not consistent enough to understand the main reason why this method outperforms prior work on MOVi-E. For example, the effect of replacing the decoder in [9] is not evaluated. At least the full version of the proposed approach with the decoder from [9] should be reported. Another important baseline is using actual optical flow (ground truth or estimated with a pre-trained model) in place of P_{t, t+1}.\n\nThe authors claim to show generalization of the proposed approach by evaluating a model trained on YouTube-VIS on DAVIS, but these experiments are not indicative of anything. It is well known in the video segmentation community that the data distributions of these two datasets are very close and models trained on YouTube-VOS/VIS perform very well on DAVIS without fine-tuning (fine-tuning brings only marginal improvements). It would be better to replace these results with a proper comparison to the state-of-the-art and a more thorough ablation analysis.\n\nThere are still quite a few issues with the presentation. In particular, the notation is not consistent: for example y_t denotes the similarity matrix in Eq. 5 but previously it is stated that y_t is the output of the model which is used both for reconstruction and for flow prediction. Later a distinction is made between y_sim and y_rec but until that point it's hard to understand what the authors are talking about. In addition, Figure 2 is never mentioned in the text and Figure 1 is only mentioned on page 5. Captions are also not self-sufficient to parse these figures. \n\nIn addition, some of the claims made in the paper are inaccurate. In particular, the authors claim that 'the approaches that are based on optical-flow/motion masks are not self-supervised because they may be hampered by the availability of labels'. This is not true, because self-supervised flow/motion segmentation estimation methods exist.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive suggestions to improve our paper. \n> The proposed approach is merely a combination of several recent techniques for object centric learning. […] The novelty of the proposed framework is minimal\n\nAll in all, we find this statement not a fair characterization of our contributions. We summarize them in the general response as supported by the other reviewers. In the following discussion the points will be made more specific.\n> 'Feature similarity loss' is equivalent to optical flow prediction from SAVi.\n\nOur feature similarity loss is *different from SAVi’s optical flow prediction conceptually, mathematically, and in terms of the resulting model performance.* \n- Conceptually: for the feature similarity loss, a successful prediction requires to predict where *all semantically similar patches* are located in the next frame (see e.g. Fig. 3). In contrast, the optical flow target only contains information on how much each pixel moves independently from each other. Thus, the temporal feature similarity target is still meaningful for non-moving objects, whereas the optical flow target is trivial.\n- Mathematically: for optical flow prediction, SAVi uses MSE against the RGB image, leading to a point prediction. For feature similarity, we use the cross entropy loss against the patch similarity distribution, predicting a categorical distribution. This allows modeling uncertainty especially when similar patches are prevalent or when objects enter or leave the scene.\n- Performance: following the reviewer's feedback, we compared optical flow targets to VideoSAUR's feature similarity targets (**Table R3**). We trained VideoSAUR with GT optical flow for the best potential performance from optical flow alone. Yet, even on the MOVi-C datasets favoring optical flow, VideoSAUR with feature similarity significantly outperformed optical flow (+10 FG-ARI). This disparity is greater in the MOVi-E dataset, highlighting VideoSAUR's resilience to static objects and camera movement. We show that our temporal feature similarity approach that doesn't need signals like GT optical flow. Additionally, we also don’t need extra datasets (**Table R4**) to perform well.\n\nThese new results now lead to a clearer picture of our contribution. We will adapt the text accordingly and rephrase the relation and comparison to optical flow in the paper. \n> … similarity between encoder features in consecutive frames is also adopted from prior work (CRW [41]). \n\nWe clarify that both the loss function and the overall learning setup in CRW are different from our work in our response to reviewer L1X2.\n> Report results of [9] on YouTube-VIS.\n\nThank you for the suggestion. To compare our method with DINOSAUR on the YouTube-VIS dataset, we train DINOSAUR and assess both approaches, along with STEVE and SAVi baselines, using image-based FG-ARI. **Table R1** displays the results. VideoSAUR outperforms DINOSAUR (+4 FG-ARI) and also surpasses video-based STEVE and SAVi methods, as shown in Table 1 of the main paper. This underscores the benefit of our temporal similarity loss over mere feature reconstruction.\n> […] why this method outperforms prior work on MOVi-E.\n\nWe conducted an in-depth ablation study on the MOVi-E dataset, assessing the impact of the decoder (DINOSAUR MLP vs Mixer) and loss function choices. As shown in **Table R2**, using similarity loss improves FG-ARI for both decoders and reaches around 74 FG-ARI (vs 69 for feature reconstruction loss). The Mixer decoder enhances object mask sharpness by +5 mBO. Consequently, using similarity loss and Mixer jointly outperforms the MLP decoder with feature reconstruction loss. These findings supplement our main paper, offering a clearer insight into VideoSAUR's components and performance.\n> The results on YouTube-VIS are low for all the reported approaches. […] simply fails less catastrophically than the baselines in the real world.\n\nWe respectfully disagree:\n- Qualitatively: we provide extensive visualizations (Figure 4, Figure E.1, Figure E.5) and on our website, demonstrating the successful operation on many different YT-VIS examples, which waves “simply fails […] catastrophically”.\n- Quantitatively: the regular pattern baseline is a good estimate of catastrophic failure. While both SAVi and STEVE perform worse or similar to that, VideoSAUR performs *twice* better, showing a clear and significant signal of “working”.\n\nBeing the first to attempt the task of fully unsupervised video-object discovery on YouTube-VIS, we present a meaningful step forward and a large improvement over all compared baselines. \n> […] evaluating a model trained on YouTube-VIS on DAVIS […] are not indicative of anything. \n\nThe distribution of objects is different in DAVIS and YouTube-VIS (e.g. number of objects). The transfer shows the flexibility of the object-centric model during inference. We note that generalizing results from supervised methods might be highly misleading. Nevertheless, we would consider moving this part to the supplementary. \n> Inaccurate claims\n\nThe quoted passage by the reviewer differs from our actual statement in L150-L152: “Another advantage of our loss is that it is fully self-supervised, whereas approaches based on optical flow or motion masks may be hampered by the availability of labels. This is of particular importance for in-the-wild video, where motion estimation is challenging.” It highlights our loss advantage, especially when optical flow/motion estimation is tough in in-the-wild videos.\nWe acknowledge the phrasing could be clearer and suggest: “Methods based on optical flow or motion masks can struggle with the need for accurate flow/motion mask labels (GT or estimated one), unlike our loss which doesn't require them.”\n> Notation and captions\n\nThank you. We will clarify the notations for the decoder and improve the descriptions of the figures. We also note that the figures were referenced earlier in the Introduction and Section 3 using “Fig.”"}]

Author rebuttal:
We thank the reviewers for their feedback and appreciate that they found our paper "well-written", "pleasant to read", and "well-organized". In addition, the reviewers recognized that our "comprehensive ablation study” is “extensively done” and that it brings insights into the method’s performance. In addition, the proposed approach was recognized as “sound” and “effective” and, as a result, yielded strong outcomes.

In addition, we also value the reviewer’s constructive feedback, with requests for additional experiments and ablations. Below, we list the main new experiments that we present in this rebuttal:

- We compare our method with the state-of-the-art image-based DINOSAUR method and several baselines on the YT-VIS dataset showing that VideoSAUR performs better than DINOSAUR on this dataset (**Table R1 in the Rebuttal’s PDF**).
- We provide an extended ablation study on the MOVi-E dataset, where we ablate both the decoder choice and the proposed loss, showing that both are needed for the state-of-the-art performance of our method. In addition, we showed that VideoSAUR significantly outperforms the extension of DINOSAUR to the video domain (**Table R2 in the Rebuttal’s PDF**).
- We study the performance of our method when GT optical flow is used as a target. We show that the motion information of each pixel alone is a worse grouping signal than the proposed temporal similarity of *highly semantic* self-supervised features (**Table R3 in the Rebuttal’s PDF**).
- We investigate our method's performance when no additional datasets are available for self-supervised pertaining. We showed that using MAE pre-trained only on MOVI-E allows VideoSAUR to achieve similar performance to the MAE pre-trained on ImageNet on both MOVi-E and MOVi-C datasets, showing that object-centric bias in ImageNet dataset is not necessary for successful object discovery with VideoSAUR (**Table R4 in the Rebuttal’s PDF**).
- Finally, we visualize and discuss how our method can fail on long videos and discuss potential future work needed to enable our method to work on long videos (**Figure R1 in the Rebuttal’s PDF**).

We are happy that the reviewers recognize the following contributions:

- This paper shows an effective way to use it **[temporal feature similarity] with self-supervised features** and recurrent Slot-Attention (tzwP)
- It incorporates a temporal feature similarity loss that encodes temporal correlations and introduces a motion bias for object discovery (avhj)
- The proposed VideoSAUR model yields state-of-the-art performance on several datasets (L1X2)

In addition, we would like to add:

- **Efficient video architecture integration**: We integrated this loss with the efficient SlotMixer decoder, where this loss synergistically reduces optimization difficulties.

________________________________________________
For convinience we also include the same tables as in PDF below: 

**Table R1. Image-based comparison on YouTube-VIS**

| Method     | Image FG-ARI    |
|------------|-----------------|
| SAVi       | 13.2 ± 5.0      |
| STEVE      | 25.3 ± 1.8      |
| DINOSAUR   | 39.2 ± 0.3      |
| VideoSAUR  | 43.1 ± 0.4      |


**Table R2. Extended ablation of VideoSAUR components on MOVi-E**
| Decoder | Loss        | FG-ARI | mBO  |
|---------|-------------|--------|------|
| Mixer   | Feature Reconstruction  | 62.3   | 20.6 |
| MLP     | Feature Reconstruction  | 68.6   | 27.6 |
| MLP     | Feature Similarity  | 74.5   | 28.8 |
| Mixer   | Feature Similarity  | 74.1   | 34.1 |

**Table R3. Ablation of VideoSAUR features similarity targets with GT optical flow**

| VideoSAUR                     | MOVi-C | MOVi-E |
|-------------------------------|--------|--------|
| w/ GT Optical Flow (backward) | 48.1   | 28.9   |
| w/ GT Optical Flow (forward)  | 48.9   | 30.1   |
| w/ Feature Similarity         | 60.7   | 73.9   |

**Table R4. Self-supervised method (Masked Auto-Encoder) pretraining on MOVi-E**


| VideoSAUR           | FG-ARI on MOVi-C| mBO on MOVi-C | FG-ARI on MOVi-E | mBO on MOVi-E |
|---------------------|---------------|------------|---------------|------------|
| w/ MAE+ImageNet   | 58.0          | 30.4       | 72.8          | 27.1       |
| w/ MAE+MOVi-E     | 59.8          | 27.5       | 70.6          | 23.3       |


Meta-review: Reviewers have come to a consensus on accepting this paper,  the authors are expected to incorporate the suggestions from reviewers in the final camera ready version.
Decision: Accept (poster)