Self-supervised Object-Centric Learning for Videos
Authors: Görkay Aydemir
Weidi Xie
Fatma Guney
Venue: NeurIPS 2023 poster (2023)
Keywords: Unsupervised Object Discovery
Unsupervised Video Object Segmentation
Object-Centric Learning
Unsupervised Video Multi Object Segmentation
TL;DR: In this paper, we introduce SOLV (Self-supervised Object-Centric Learning for Videos),  a self-supervised model capable of discovering multiple objects in real-world video sequences without using additional modalities.

Abstract: Unsupervised multi-object segmentation has shown impressive results on images by utilizing powerful semantics learned from self-supervised pretraining. An additional modality such as depth or motion is often used to facilitate the segmentation in video sequences. However, the performance improvements observed in synthetic sequences, which rely on the robustness of an additional cue, do not translate to more challenging real-world scenarios. In this paper, we propose the first fully unsupervised method for segmenting multiple objects in real-world sequences. Our object-centric learning framework spatially binds objects to slots on each frame and then relates these slots across frames. From these temporally-aware slots, the training objective is to reconstruct the middle frame in a high-level semantic feature space. We propose a masking strategy by dropping a significant portion of tokens in the feature space for efficiency and regularization. Additionally, we address over-clustering by merging slots based on similarity. Our method can successfully segment multiple instances of complex and high-variety classes in YouTube videos.

Self-supervised Object-Centric Learning for Videos
Görkay Aydemir' Weidi X
Fatma Güney 1,2
Department of Computer Engineering, Koç University KUIS Al Center CMIC, Shanghai Jiao Tong University fgaydemir23, fguney] @ku. edu. tr
"Shanghai Al Laboratory weidi@s Midii@jjun jtssjcu.eee edu. cn
Abstract
Unsupervised multi-object segmentation has shown impressive results on images by utilizing powerful semantics learned from self-supervised pretraining. An addi- tional modality such as depth motion is often used to facilitate the segmentation in video sequences. However, the performance improvements observed symthetic sequences, which rely the robustness of additional cue, do not translate more challenging real-world scenarios. this paper, we propose the first fully unsupervised method for segmenting multiple objects in real-world sequences. Our object-centric learning framework spatially binds objects to slots on each frame and then relates these slots across frames. From these temporally-aware slots, the training objective is to reconstruct the middle frame in high-level semantic feature space. We propose masking strategy by dropping significant portion of tokens in the feature space for efficiency and regularization. Additionally, we address overtelusg by merging slots based on similarity. Our method can successfully segment multiple instances of complex and high-variety classes in YouTube videos.!
Introduction
Given a video sequence of a complex scene, our goal is to train vision system that can discover, track, and segment objects, way that abstracts the visual information from millions of pixels into semantic components, i.e., objects. This is commonly referred to as object-centric visual representation learning in the literature. By learning such abstractions of the visual scene, the resulting object-centric representation acts as fundamental building blocks that can be processed independently and recombined, thus improving the model's generalization and supporting high-level cognitive vision tasks such as reasoning, control, etc. [13, 29].
The field of object-centric representation learning computer vision has made significant progress over the years, starting from synthetic images [34, 37], and has since shifted towards in in-the-wild image [20, 49] and real-world videos [65, 85, 87, 46, 60]. In general, existing approaches typically follow an autoencoder training paradigm [52, 26], i.e., reconstructing the input signals with certain bottlenecks, with the hope to group the regional pixels into semantically meaningful objects based on the priors of architecture or data. In particular, for images, low-level features like color, and semantic features from pretrained deep networks, are often used indicate the assignment of pixels to objects, while for videos, additional modalities/signals are normally incorporated, such optical flow [40], depth map [17], with segmentation masks directly available from the discontinuities. Despite being promising, using additional signals in vidcos naturally incurs computation overhead and error accumulation, for example, optical flow may struggle with static or deformable objects and large displacements between frames, while depth may not be readily available in generic videos and its estimation can suffer in low-light or low-contrast environments.
Project page: http:/kuii-ii.gihuuiooooe
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

First Frame
Middle Frame
Last Frame
First Frame
Middle Frame
Last Frame
Figure 1: We introduce SOLV, an object-centric framework for instance segmentation in real-world videos. This figure depicts instance segmentation results of first, middle, and last frames of videos on Youtube-VIS 2019 [87]. Without any supervision both in training and inference, manage to segment object instances accurately.
In this paper, we introduce SOLV, a self-supervised model capable of discovering multiple objects in real-world video sequences without using additional modalities [2, 36, 40, 17] or any kind of weak supervision such as first frame initialization [40, 17]. To achieve this, we adopt axial spatial-temporal slot attentions, that first groups spatial regions within frame, then followed by enriching the slot representations using additional cues from interactions with neighboring frames. We employ masked autoencoder (MAE)-type training, with the objective of reconstructing dense visual features from masked inputs. This approach has two benefits: first, it acts as an information bottleneck, forcing the model to learn the high-level semantic structures, given only partial observations; second, it alleviates memory constraints, which helps computational efficiency. Additionally, due to the complexity of visual scenes, fixed number of slots often leads to over-segmentation issue. We address this issue by merging similar slots with a simple clustering algorithm. Experimentally, our method significantly advances the state-of-ther toffthheaat art without using information from additional modalities on a commonly used synthetic video dataset, MOVi-E [25] and a subset of challenging Youtube- VIS 2019 [87] with in-the-will videos (Fig. 1).
To summarize, we make the following contributions: (i) We propose a self-supervised multi-object segmentation model on real-world videos, that uses axial spatial-temporal slots attention, effectively grouping visual regions with similar property, without relying on any additional signal; (ii) We present an object-centric learning approach based on masked feature reconstruction, and slot merging; (iii) Our model achieves state-of-the-ar results on the MOVi-E and Youtube- VIS 2019 datasets, and competitive performance on DAVIS2017.
2 Related Work
Object-centric Learning: The field of unsupervised learning of object-centric representations from images and videos has gained considerable attention in recent years. Several approaches have been proposed address this problem with contrastive learning [31, 41, 84] or more recently with reconstruction objectives. An effective of region identifier variables in the latent space, i.e. slots, that bind to distinctive parts corresponding to objects. Slot attention has been applied to both images [26, 5, 10, 12, 18, 50, 89, 19, 27, 15, 52, 69] and videos [35, 71, 28, 40, 33, 75, 90, 43). However, these methods are typically evaluated on synthetic data and struggle to generalize to real-world scenarios due to increasing complexity. To address this challenge, previous works explore additional information based on the 3D structure [8, 58, 32) or reconstruct different modalities such as flow [40] or depth [17]. Despite these efforts, accurately identifying objects in complex visual scenes without explicit guidance remains an open challenge. The existing work relies guided initialization from motion segmentation mask [1, 2] initial object locations [40, 17J. To overcome this limitation, DINOSAUR [67] performs reconstruction in the feature space by leveraging the inductive biases learned by recent self-supervised models [7]. We also follow this strategy which has proven highly effective in learning object-centric representations in real-world data without any guided initialization explicit supervision.
connstrrccion-baeed approach first divides the input into set
Object Localization from DINO Features:
The capabilities of Vision Transformers (ViT) [14]
have been comprehensively investigated, leading to remarkable findings when combined with self- supervised features from DINO [7]. By grouping these features with a traditional graph partitioning
method [57, 68, 81], impressive results can be achieved compared to earlier approaches [76, 77].

---

Recent work, CutLER [80], extends this approach to segment multiple objects with series of normal- ized cuts. The performance of these methods without any additional training shows the power of self-supervised DINO features for segmentation and motivates us to build on it in this work.
Video Object Segmentation: Video object segmentation (VOS) [86, 4, 22, 23, 42, 38, 44, 45, 56, 59, 61, 72, 78, 79, 88] aims to identify the most salient object in video without relying on annotations during evaluation unsupervised setting [73, 21, 47, 53] and only annotation of the first frame in semi-supervised setting [6]. Even if the inference is unsupervised, ground-truth annotations can be used during training in VOS [11, 16, 63, 54, 6]. Relying on labelled data during training might introduce a bias towards the labelled set of classes that is available during training. In this paper, we follow a completely unsupervised approach without using any annotations during training or testing.
Motion information is commonly used in unsupervised VOS to match object regions across time [36, 9, 51, 66]. Motion cues particularly come in handy when separating multiple instances of objects in object-centric approaches. Motion grouping [86] learns an object-centric representation to segment moving objects by grouping patterns in flow. Recent work resorts to sequential models while incorporating additional information [70, 40, 17, 1J. In this work, we learn temporal slot representations from multiple frames but we do not use any explicit motion information. This way, we can avoid the degradation in performance when flow cannot be estimated reliably.
3 Methodology
In this section, we start by introducing the considered problem scenario, then we detail the proposed object-centric architecture, that trained with self-supervised learning.
3.1 Problem Scenario
Given an RGB video clip input, i,.e., V/ (V--n..... Vt.... Vttn } E R(2n+1)xH*Xx3 our goal is to train an object-centric model that processes the clip, and outputs all object instances in the form of segmentation masks, i.e., discover and track the instances in the video, via self-supervised learning, we can formulate the problem as:
m, + (V:; θ) Dvis-dec o Pst-bind o Piis-enc (V:)
(1)
where m, E RKxxHxWW refers to the output segmentation mask for the middle frame with K discovered objects. After segmenting each frame, we perform Hungarian matching track the objects across frames in the video. 4 (:; refers to the proposed segmentation model that will be detailed in the following section. Specifically, it consists of three core components (see Fig. 2), namely, visual encoder that extracts frame-wise visual features (Section 3.2. pettial-emmpral axial binding that first groups pixels into slots within frames, then followed by joining the slots across temporal frames (Section 3,2.2), and visual decoder that decodes the spatial- patial-temporal slots to reconstruct the dense visual features, with the segmentation masks of objects as by-products (Section 3.2.3).
3.2 Architecture
Generally speaking, our proposed architecture is variant of Transformer, that can be trained with simple masked autoencoder, i.e., reconstructing the full signals given partial input observation. However, unlike standard MAE [30] that recovers the image in pixel space, we adopt an information bottleneck design, that first assigns spatial-temporal features into slots, then reconstructs the dense visual features from latent slots. As a result, each slot is attached to one semantically meaningful object, and the segmentation masks can be obtained as by-products from reconstruction. i.e., without relying on manual annotations.
3.2.1 Visual Encoder
Given the RGB video clip, divide each image into regular non-overlapping patches. Following the notation introduced in Section 3.1, i.e., V;-
(Ve-n..... Vt.... .. V(+n) RR (2 ()x(P))) where N HW/ p2 is the number of tokens extracted from each frame with patches of size P. The visual encoder consists of token drop and feature extraction, detailed below.
3

---

DINO
V
s-bind
V
F-bind
merge
Vdcc
Visual Encoder
Spatial-temporal Binding
Visual Decoder
Invariant
Slot
V
mapper
Atteution
Brondcasting
Marks &
Agglomerative Clustering #
Reconstrattoos
s-bind
V dee
Figure 2: Overview. In this study, we introduce SOLV, an autoencoder- based model designed for object-centric learning in videos. Our model consists of three components: (i) Visual Encoder for extracting features for each frame using ΦDINo: (ii) Spatial-temporal Binding module for generating temporally-aware object-centric representations by binding them spatially and temporally using bind and bind. respectively; (iii) Visual Decoder for estimating segmentation masks and feature reconstructions for the central frame with dede after merging similar slots using mergge
Token Drop: As imput to the encoder, we only sample a subset of patches. Our sampling strategy is straightforward: we randomly drop the input patches with some ratio for each frame,
V (V = drop (Ve-n) drop (vetn) x1)x++1"x""xx33X33
N < N
where N' denotes the number of tokens after random sampling.
We use frozen Vision Transformer (ViT) [14] with parameters initialized from DINOv2 [62], self-supervised model that has been pretrained on a large number of images:
Feature Extraction:
F (f.m........ f =
ODDIO
(Ve+n)) 1(2n+1)xN'
(3)
where refers to the dimension of output features from the last block of DINOv2, right before the final layer normalization.
Discussion: Our design of token drop serves two purposes: Firstly, masked autoencoding has been widely used in natural language processing and computer vision, acting as a proxy task for self-supervised learning, token drop effectively encourage the model acquire high-quality visual representation; Secondly, for video processing, the extra temporal axis brings few orders of magnitude of more data, processing sparsely sampled visual tokens can substantially reduce memory budget, enabling computation-efficien learning, as will be validated in our experiments.
3.2.2 Spatial-temporal Binding
After extracting visual features for each frame, we first spatially group the image regions into slots, with each specifying semantic object, i.e., discover objects within single image; then, we establish the temporal bindings between slots with a Transformer, i.e., associate objects within video clips.
Pst-bind (F) t'raiin (Vs-bind (fe-n) ... Ve-rind (fe+n)) E kggxxx Dan
Spatial Binding (2/s-nina): The process of spatial binding is applied to each frame independently. We adopt the invariant slot attention proposed by Biza et al. [3], with one difference, that is, we use shared initialization Z, at each time step E {t n. n... Specifically, given features after token drop at a time step t as input, we learn a set of initialization vectors to translate and scale the input position encoding for each slot separately with K slot vectors z & Ras K scale vectors

---

S E R², K position vectors S E R2, and one absolute positional embedding grid Gabs E IR" Nx2 We mask out the patches corresponding the tokens dropped in feature encoding (Section 3.2.1) and obtain absolute positional embedding for each frame as Gabs,7 = drop (aats) E R x2 Please refer to the original paper [3] or the Supplementary Material for details of invariant slot attention single frame. This results in the following set of learnable vectors for each frame:
2, = (z' Sy, Spp, ) K
(5)
Note that these learnable parameters are shared for all frames and updated with respect to the dense visual features of the corresponding frame. In other words, slots of different frames start from the same representation but differ after binding operation due to interactions with frame features. The output of bisd the updated slots corresponding to independent object representations at frame T:
{ j= 4/s-bind (f,) E R
Kx Daia
T {t n, ...... t n}
(6)
In essence, given that consecutive frames typically share similar visual context, the of learned slots inherently promotes temporal binding, wherehy slots with the same index can potentially bind to the same object region across frames. Our experiments demonstrate that incorporating invariant slot attention aids in learning slot representations that are based on the instance itself, rather than variable properties like size and location.
Temporal Binding (e'v-iia): Up until this point, the model is only capable of discovering objects by leveraging information from individual frames. This section focuses on how to incorporate temporal context to enhance the slot representation. Specifically, given the output slots from the spatial binding module, denoted as (z " { E xxx Pau, we apply a transformer encoder to the output slots with same index across different frames. In other words, the self-attention mechanism only computes a (2n + 1) x (2n + 1) affinity matrix across 2n + 1 time steps. This approach helps to concentrate on the specific region of the image through time. Intuitively, each slot learns about its future and past the temporal window by attending to each other, which helps the model to create more robust representation by considering representations of the same object at different times.
To distinguish between time stamps, we incorporate learnable temporal positional encoding onto the slots, Le., all slots from single frame receive the same encoding. As the result of temporal transformer, we obtain the updated slots at the target time step t:
c = Psc-bind (F) E R Dau
3.2.3 Visual Decoder
The spatial-temporal binding process yields a set of slot vectors c E R at target frame / that aware of temporal context. However, in natural videos, the number of objects within frame can vary significantly, leading to over-clustering when a fixed number of slots is used, i.e., as we can always initialize slots in an over-complete manner.
To overcome this challenge, we propose a simple solution for slot merging through Agglomerative Clustering. Additionally, we outline the procedure for slot decoding to reconstruct video features, which resembles a masked auto-encoder in feature space.
S vis-dec (c) v/dec
Vnerrg (c)
Slot Merging (emegre): As expected, the problem of object segmentation is often a poorly defined problem statement in the self-supervised scenario, as there can be multiple interpretations of visual regions. For instance, in an image of person, it is reasonable group all person pixels together or group the person's 's face, arms, body, and legs separately. However, it is empirically desirable for pixel embeddings from the same object to be closer than those of different objects. To address this challenge, we propose to merge slots using Agglomerative Clustering (AC), which does not require a predefined number of clusters. As shown in Fig. 3, we first compute the affinity matrix between all slots based on cosine similarity, then use this aftinity matrix to cluster slots into groups, and compute the mean slot for each resulting cluster:
Umerge (c) E 1R K.X KiXDam Dan,
K. S K
(9)
5

---

Figure 3: Assignments of slots to the input frame (left) and the pairwise similarity ma- trix of slots (right) where lighter colors in- dicate higher similarity. Slots corresponding to the parts of the same object (0 and 1) are highly similar each other while being dif- ferent from background slots (2, 3, 4, and 5),
Figure 4: Results without (left) and with (right) slot merging.
By merging semantically similar slots that correspond to the same object, we can dynamically determine the optimal number of slots. Our slot merging strategy, as shown in Fig. 4, effectively combines regions of the same object using solely learned visual features. Note that slot merging not a post-procssiing step, but rather an integral part of our training process.
Decoder (t'tec): We decode the merged slots c' with decoder t/ to obtain the corresponding segmentation mask m and the full reconstruction y:
m = dhe (c'), YER m E R
(10)
We reshape and upsample the masks m to the original input size to obtain the final segmentation. spatial broadcast decoder [82] to reconstruct the full feature map, i.e., 3 E RND of each slot j with their alpha weights a RR These alpha weights are converted to segmentation masks with softmax. As per standard procedure, we add learned positional encodings to identify spatial locations during decoding. Fach slot c²³, broadcasted to the shape of the input feature map. is decoded by series of linear layers, Vmapper. with shared weights across all slots. The reconstruction ultimately achieved through a weighted sum of the decoded slots:
Similar the MLP decoder design introduced in DINOSAUR [67], we
m
m' softmax (a"), a², yd M'mapper broadcast
(c')
(11)
At training time, we optimise the model by minimising the difference between the feature map computed from original unmasked tokens of the frame time and the reconstructed tokens y:
L = |IŌDINO (vi) - yl
(12)
4 Experiments
4.1 Experimental Setup
Our proposed method is evaluated on one synthetic and two real-world video datasets. For the synthetic dataset, we select MOVi [25], a widely-used benchmark for evaluating object-centric methods, particularly for multi-object segmentation in videos. To ensure rigorous evaluation, we use the challenging MOVi-E dataset with up to 20 moving and static objects and random camera motion, suggested by Bao et al. [2]. For the real-woril datasets, In addition, we evaluate our method on a subset of the Youtube- VIS 2019 (YTVIS19) [87] train set, because there is no official validation test set provided with ground-truth masks. Specifically, the first evaluation set consists of 30 videos out of 90 videos on DAVIS17, and the second evaluation set consists of 300 videos out of 2,883 high-resolution videos YTVIS19. We perform all the ablations on the 19 dataset.
Datasets:
use the validation split of DAVIS17 [65].
Metries: For our synthetic dataset evaluation, we use the foreground adjusted rand index (FG-ARI)
measure the quality of clustering into multiple foreground objects. To maintain consistency with prior studies, we calculate the per-frame FG-ARI and report the mean across all frames, done in

---

recent works such as Karazija et al. [36], Bao et al. [2], and Seitzer et al. [67]. This approach also allows to compare state-of-the-an image-based segmentation methods, such as DINOSAUR [67]. (mloU) metric, which is widely
For real-world datasets, we use the mean copeneectiooovve--nnieee accepted segmentation, by considering only foreground objects. To ensure temporal consistency of assignments between frames, we apply Hungarian Matching between the predicted and ground-truth masks of foreground objects in the video, following the standard practice [65].
4.2 Results on Synthetic Data: MOVi-E
The results on the MOVi-E dataset are presented in Table 1, with methods divided based on whether they use an additional modality. For example, the sequential extensions of slot attention, such as SAVi [40] and SAVi++ [17], reconstruct different modalities like flow and sparse depth, respectively, their performance falls behind other approaches despite additional supervision. On the other hand, STEVE [69] improves results with a transformer-basee decoder for reconstruction. PPMP [36] significantly improves performance by predicting probable motion patterns from an image with flow supervision during training. Mo'Tok [2] outperforms other methods, but relies on motion segmentation masks to guide the attention maps of slots during training, with a significant drop in performance without motion segmentation. The impressive performance of DINOSAUR [67] highlights the importance of self-supervised training for segmentation. Our method stands out by using the cnatial-temporal slot binding, and autoencoder training in the feature space, resulting in significantly better results than all previous work (see Fig. 5).
Table I: Quantitative Results MOVi-E. This table shows results in comparison to the previous work in terms of FG-ARI MOVi-E.
RGB
Model
+Modality
FG-ARI
SAVi [40]
Flow
39,2
GT
SAVi++ [17]
Sparse Depth
41.3
PPMP [36]
Flow
63.1
MoTokk [2]
Motion Seg.
66.7
MoTok [2]
52.4
Ours
STEVE [70]
54.1
DINOSAUR [67]
65.1
Ours
80.8
Figure 5: Qualitative Results on MOVi-E.
4.3
Results on Real Data: DAVIS17 and Youtube-VEL 2019
Due to a lack multi-object segmentation methods evaluated on real-world data, we present a baseline approach that utilizes spectral clustering the DINOv2 features [62]. To define the number of clusters, we conduct an oracle test by using the ground-trn number of objects. After obtaining masks independently for each frame, between frames based on the similarity
ensure temporal consistency by applying Hungarian Matching
the mean feature of each mask. However, as shown in Table 2, directly clustering the DINOv2 features produces poor results despite the availability of privileged information on the number of objects.
Table 2: Quantitative Results Real- Real-World World Data. These results show the video multi-object evaluation results on the validation split of DAVIS17 and a subset of the YTVIS19 train split.
DAVIS17
YTVIS19
Model
Supervision
mIoU t
FG-ART
mloU
FG-ARI 1
DINOv2 [62] Clustering
#Objects Flow
14.34
24.19
28.17
16.19
OCTR [83]
34.60
14.67
32.49
15.88
Ours
30.16
32.15
45.32
29.11

---

First Frame
Middle Frame
Last Frame
First Frame
Middle Frame
Last Frame
RGB
GT
OCLR
Ours
Figure 6: Qualitative Results of Multi-Object Video Segmentation YTVIS19. We provide the first, middle, and last frames along with their corresponding segmentation after the Hungarian Matching step. Our model is proficient in accurately identifying objects in their entirety, irrespective of their deformability or lack of motion. This capability is highlighted in contrast to OCLR [83], whose results displayed in the third row for comparison.
also compare our method to the state-of-the-art approach OCLR [83], which is trained using supervised objective synthetic data with ground-truth optical flow. OCLR [83] slightly outperforms our method on DAVIS17, where the optical flow can often be estimated accurately, that greatly benefits the segmentation by allowing it to align with the real boundaries, resulting in better mloU performance. However, our method excels at detecting and clustering multiple objects, even when the exact boundaries are only roughly located, as evidenced by the higher FG-ARL. The quality of optical flow deteriorates with increased complexity in YTVIS vidcos, causing OCLR [83] to fall significantly behind in all metrics. The qualitative comparison in Fig. demonstrates that our method can successfully segment multiple objects a variety of videos. Additional qualitative results can be found in the Supplementary Material.
4.4 Ablation Study
On the Effectiveness of Architecture: We conducted an experiment examine the impact of each proposed component on the performance. Firstly, we replaced the spatial binding module, 1 with the original formulation in Slot Attention [52]. Secondly, we removed the temporal binding module, 1/1 hind, which prevents information sharing between frames at different time steps. Finally, we eliminated the slot merging module, m/merge-
We can make the following observations from the results in Table 3: (i) as shown by the results of Model-A, simple extension of DINOSAUR [67] to videos, by training with DINOv2 features and matching mask indices across frames, results in poor performance; (ii) only adding slot merging (Model-B) does significantly improve performance, indicating that over-clustering is the primary issue; (iii) significant mloU gains can be achieved with temporal binding (Model-C) or
Table 3: Components. The effect of changing our spatial binding module (1/s bind) to the original slot attention, removing the temporal binding module (virbina) or the slot merging module (Wmerge).
Table 4: Number Slots. The effect of varying the number of slots with/without slot merging.
#Slots
Merging
mIoU f FG-ARI
X
43.29
27.73
Model
V Vmergee
Uisiind
Ussiid
mloU t FG-ARI 1
6
/
44.90
28.52
X
x
X
37.75
27.05
X
39.90
22.55
ACC
X
38.23
29.09
8
/
45.32
29.11
DDD
X
/
44.95
28.42
x
44.94
27.38
X
36.04
20.20
12
E
V
/
45.32
29.11
/
43.19
27.94

---

Table 5: Visual Encoder. The effect of differ- types of self-supervised pretraining methods and architectures for feature extraction.
miou
40
30
0.50
Model
mloU
FG-ART
0.90
Supervised ViT-B/16
37.58
19.94
16
18
20
22
24
26
28
Memory Consumption (GB)
DINO VIT-B/8
39.53
24.70
DINO VIT-B/16
41.91
24.01
Figure 7: Token Drop Ratio. The effect of
DINOv2 VIT-B/14
45.32
29.11
varying token drop ratio performance (mloU) and memory consumption.
spatial binding (Model-D), which shows the importance of spatially grouping pixels within each frame as well as temporally relating slots in the video. Finally, the best performance is obtained with Model-E in the last row by combining all components.
Number of Slots: We perform experiment by varying the number of slots with and without slot merging in Table 4. Consistent with the findings in previous work [67], the performance is highly dependent on the number of slots, especially when slot merging not applied. For example, without slot merging, increasing the number of slots leads to inferior results in terms of both metrics, due to the over-segmentation issue in some videos. While using slot merging, we can better utilize larger number of slots as can be seen from the significantly improved results of 8 or 12 slots with slot merging. This is also reflected FG-ARI, pointing to better performance in terms of clustering. We set the number of slots to 8 and use slot merging in our experiments. We provide a visual comparison of the resulting segmentation of an image with and without slot merging in Fig. 4. Our method can successfully merge parts of the same object although each part is initially assigned to different slot.
Token Drop Ratio: We ablated the ratio of tokens drop in Fig. 7. As our motivation for the token drop is two-fold (Section 3.2.1), we show the effect of varying the token drop ratio on both performance and memory usage. Specifically, we plot mloU the vertical axis versus the memory consumption on the horizontal axis. We report the memory footprint as the peak in the GPU memory usage throughout single pass with constant batch size. Fig. confirms the trade-off between memory usage and performance. As the token drop ratio decreases. mloU increases due to less information loss from tokens dropped. However, this comes the cost of significantly more memory usage. We cannot report results by using all tokens, i.e. token drop ratio of O, due to memory constraints. Interestingly, beyond a certain threshold, i.e. at 0.5, the token drop starts to act as regularization, resulting in worse performance despite increasing the number of tokens kept. We use 0.5 in our experiments, which reaches the best mloU with a reasonable memory requirement. In summary, the token drop provides not only efficiency but also better performance due to the regularization effect.
Visual Encoder: We conducted experiments to investigate the effect different visual encoders in Table 5. We adjust the resize values to maintain a consistent token count of 864 for different architec- tures. These findings underscore the critical role of pretraining methods. Please see Supplementary for qualitative comparison. Notably, the DINOv2 model [62] outperforms its predecessor, DINO [7], in terms clustering efficiency. On the other hand, models pretrained using supervised methods results in the weakest performance, with substantial gap in the FG-ARI metric when compared to self-supervised pretraining methods. Furthermore, no specific patch size offers clear advantage within the same pretraining type. For instance, DINO pretraining with ViT-B/8 exhibits superior performance the FG-ARI metric but lags behind its 16-patch-sized counterpart in the mloU metric.
Discussion
We presented the first fully unsupervised object-centric learning method for scaling multi-object segmentation in-the-wild videos. Our method advances the state-of-the-ar significantly commonly used simulated data but more importantly, is the first fully unsupervised method to report state-of-the-art performance unconstrained videos of the Youtube-VIS dataset.
This work takes a first step towards the goal of decomposing the world into semantic entities from in-the-wild wild videos without any supervision. The performance can always be improved with better

---

features from self- sule-vviee pretraining as shown in the comparison of different DINO versions. While we obtain significant performance boosts from self-supervised pretraining, also comes with some limitations. For example, our method can locate objects roughly, but it fails to obtain pixel-accurate boundaries due to features extracted at patch level. Furthermore, nearby objects might fall into the same patch and cause them to be assigned to the same slot. Our slot merging strategy, although has been proved effective as a simple fix for the issue of sver-segmentttion, remains not differentiable. Future work needs to go beyond that assumption and develop methods that can adapt to a varying number of objects in a differentiable manner.
Acknowledgements
Weidi Xie would like to acknowledge the National Key R&D Program of China (No. 2022ZD0161400).
References
[1] Zhipeng Bao, Pavel Tokmakov, Allan Jabri, Yu-Xiong Wang. Adrien Gaidon, and Martial Hebert. Discov- ering objects that can move. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 2, 3
[2] Zhipeng Bao, Pavel Tokmakov, Yu- Yu-Xiong Wang, Adrien Gaidon, and Martial Hebert. Object discovery from motion- guided tokens. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR).
2023. 2, 6.7
[3] Ondrej Biza, Sjoerd van Steenkiste, Mehdi S. M. Sajjadi, Gamaleldin F. Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames. In Proc. of the International Conf. on Machine learning (ICML), 2023. 4, 5, 16
[4] Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In Proc. of the European Conf. on Computer Vision (ECCV), 2010. 3
[5] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXivorr, 2019. 2 [6] Sergi Caelles, Kevis- Kokitsi Maninis, Jordi Pont- Tuset, Laura Leal-Taixe, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2017.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021. 2, 9, 18
[8] Chang Chen, Fei Deng, and Sungjin Ahn. Roots: Object-centric representation and rendering of 3d scenes.
In Journal of Machine Learning Research (JMLR), 2021. 2
[9] Subhabrata Choudhury, Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. A simple and powerful global optimization for unsupervised video object segmentation. In Proc. of the British Machine Vision Conf. (BMVC), 2022. 2022., 3
[10] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detection with convolutional
neural networks. In Proc. of the Conf. on Artificial Intelligence (AAAI), 2019. 2
[11] Achal Dave, Pavel Tokmakov, and Deva Ramanan. Towards segmenting anything that moves. In Proc. of the IEEE International Conf. on Computer Vision (ICCV) Workshops, 2019.
[12] Fei Deng, Zhuo Zhi, Donghun Lee, and Sungjin Ahn. Generative scene graph networks. In Proc. of the International Conf. on Learning Representations (ICLR), 2021. 2
[13] Andrea Dittadi, Samuele Papa, Michele De Vita, Bernhard Schölkopf, Ole Winther, and Francesco Lo- catello. Generalization and robustness implications in object-centric learning. In Proc. of the International Conf. on Machine learning (ICML), 2022.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In Proc. of the International Conf. on Learning Representations (ICLR). 2021. 2, 4
[15] Yilun Du Shuang Li, Yash Sharma, Josh Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2 [16] Suyog Dutt Jain, Bo Xiong, and Kristen Grauman. Fusionseg: Learning to combine motion and appearance for fully automatic segmentation of generic objects in videos. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2017.
[17] Gamaleldin Elsayed, Aravindh Mahendran, Sjoerd van Steenkiste, Klaus Greff, Michael C Mozer, and Thomas Kipf. Savi++: Towards end-to-end object-centric learning from real-world videos. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 1, 7.7
10

---

[18] Martin Engelcke, Adam R Kosiorek, Oiwi Parker Jones, and Ingmar Posner. Genesis: Generative scene inference and sampling with object-centric latent representations. In Proc. of the International Conf. on Learning Reprexentations (ICLR), 2020.
[19]
SM Eslami, Nicolas Heess, Theophane Weber, Yuval Tassa, David Szepesvari, Geoffrey E Hinton, et al. Attend, infer, repeat: Fast seene understanding with generative models. In Advances Neural Information
Processing Systems (NeurlPS). 2016. 2
[20] Mark Everingham. Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. In International Journal of Computer Vision (IJCV), 2010.
[21] Alon Faktor and Michal Trani. Video segmentation by non-local consensus voting. In Proc. of the British Machine Vision Conf. (BMVC), 2014. 3
[22] Deng-Ping Fan. Wenguan Wang, Ming-Ming Cheng, and Jianbing Shen, Shifting more attention to video salient object detection. In Proc. IEFE Conf. Computer Vision and Puttern Recognition (CVPR), 2017.
[23] Katerina Fragkiadaki, Geng Zhang, and Jianbo Shi. Video segmentation by tracing discontinuities in a trajectory embedding. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2012. [24] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Journal of Machine Learning Research (JMIR), 2010. 15
[25] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator. In Proc. IEEE Conf. on Computer Vision and Pottern Recognition (CVPR), 2022. 2, 6
[26] Klaus Greff, Raphaël Lopez Kaufman, Rishabh Kabra, Nick Watters, Christopher Burgess, Daniel Zoran, Loic Matthey, Matthew Botvinick, and Alexander Lerchner. Multi-object representation learning with iterative variational inference. In Proc. of the International Conf. on Machine learning (ICML), 2019. 1,2
[27] Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Jürgen Schmidhuber. Tagger: Deep unsupervised perceptual grouping. In Advances in Neural Information Processing Systems (NeurIPS), 2016. 2
[28] Klaus Greff. Sjoerd Van Steenkiste, and Jürgen Schmidhuber. Neural expectation maximization. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 2
[29] Klaus Greff, Sjoerd Van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural
networks. In arXiv.org, 2020. 1
[30] Kaiming Hc, Xinlei Chen, Saining Xic, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 3
[31] Hénaff, Olivier J and Koppula, Skanda and Shelhamer, Evan and Zoran, Daniel and Jaegle, Andrew and Zisserman, Andrew and Carreira, João and Arandjelovié, Relja. Object discovery and representation networks. In Proc. of the European Conf. on Computer Vision (ECCV), 2022.
[32] Paul Henderson and Christoph H Lampert. Unsupervised object-centric vidco generation and decomposition in 3d. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 2
[33] Jindong Jiang, Sepehr Janghorbani, Gerard De Melo, and Sungjin Ahn. Scalor: Generative world models with scalable object representations. In Proc. of the International Conf. on Learning Representations
(ICLR), 2020.
[34] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proc. IELE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017.
[35] Rishabh Kabra, Daniel Zoran, Goker Erdogan, Loic Matthey, Antonia Creswell, Matt Botvinick, Alexander Lerehner, and Chris Burgess. Simone: View-invariant, temporally-aee abbttrrce object representations via unsupervised video decomposition. In Advances in Neural Information Processing Systems (NeurIPS),
2021. 2
[36] Laurynas Karazija, Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Un- supervised Multi-objeet Segmentation by Predicting Probable Motion Patterns. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 2, 3,
[37] Laurynas Karazija. Iro Laina. and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsuper- vised multi-object segmentation. In arXiv.org, 2021.
[38] Margret Keuper, Bjoemn Andres, and Thomas Brox. Motion trajectory segmentation via minimum cost multicuts. In Proc. of the IEFE International Conf. on Computer Vision (ICCV), 2015. 3
[39] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. of the
International Conf. on Learning Representations (ICLR), 2015. 2015. 16
[40] Thomas Kipf, Gamaleldin F. Elsayed, Aravindh Mahendran, Sara Stone, Austinand Sabour, Georg Heigold, Rico Jonschkowski, Alexey Dosovitskiy, and Klaus Greff. Conditional object-centric learning from video. In Proc. of the International Conf. on Learning Representations (ICLR), 2022. 1, 3.
[41] Thomas Kipf, Elise Van der Pol, and Max Welling. Contrastive learning of structured world models. In Proc. of the International Conf. Learning Representations (ICLR), 2020. 2
11

---

[42]
Yeong Jun Koh and Chang-Su Su Kim. Primary object segmentation in videos based on region augmentation and reduction. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR). 2017. 3
[43] Adam Kosiorek, Hyunjik Kim, Yee Whye Teh, and Ingmar Posner. Sequential attend, infer, repeat: Generative modelling of moving objects. In Advances in Neural Information Processing Systems (NeurIPS).
2018. 2
[44] Zihang Lai, Erika Lu, and Weidi Xie. Mast: A memory-augmeented self-supervised tracker. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 3
[45] Zihang Lai and Weidi Xie. Self-supervised learning for video correspondence flow. In Proc. of the British Machine Vision Conf. (BMVC), 2019. 3
[46] Fuxin Li. Taeyoung Kim, Ahmad Humayun. David Tsai, and James M Rehg. Video segmentation by tracking many figure-groud segments. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2013.
[47] Siyang Li, Bryan Seybold, Alexey Vorobyov, Alireza Fathi, Qin Huang, and C-C Jay Kuo. Instance embedding transfer to unsupervised vidco object segmentation. Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018.
[48] Huaijia Lin, Ruizheng Wu, Shu Liu, Jiangbo Lu, and Jiaya Jia. Video instance segmentation with propose-reduce paradigm. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021. 17 [49] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. of the European Conf. on Computer Vision (ECCV), 2014. 1
[50] Zhixuan Lin, Yi-Fu Wu, Skand Vishwanath Peri, Weihao Sun, Gautam Singh, Fei Deng. Jindong Jiang. and Sungjin Ahn. Space: Unsupervised object-oriented scene representation via spatial attention and decomposition. In Proc. of the International Conf. Learning Representations (ICLR), 2020. 2
[51] Runtao Liu, Zhirong Wu, Stella Yu, and Stephen Lin. The emergence of objectness: Learning zero-shot segmentation from vidcos. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 3
[52] Francesco Locatello. Dirk Weissenborn. Thomas Unterthiner, Aravindh Mahendran, Georg Heigold. Jakob Uszkorcit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems (NeurIPS), 2020. 1, 2, 8
[53] Xiankai Lu, Wenguan Wang. Chao Ma, Jianbing Shen, Ling Shao, and Fatih Porikli. See more, know more: Unsupervised vidco object segmentation with co-attention siamese networks. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[54] Jonathon Luiten, Paul Voigtlaender, and Bastian Leibe. Premvos: Proposal-generatee refinement and merging for video object segmentation. In Proc. of the Asian Conf. on Computer Vision (ACCV), 2018. [55] Jonathon Luiten, Idil Esen Zulfikar, and Bastian Leibe. Unovost: Unsupervised offiine video object segmentation and tracking. In Proc. of the IEEE Winter Conference on Applications of Computer Vision (WACV), 2020. 17
[56] K.-K. Maninis, S. Caelles, Y. Chen, J. Pont-Tuset, L. Taixe, D. Cremers, and L Van Gool. Video object segmentation without temporal information. In IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI). 2018.
[57] Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
[58] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes compositional generative neural feature fields. In Proc. IEEE Conf. on Computer Vision and Pattern Recognation (CVPR), 2021. 2
[59] Peter Ochs and Thomas Brox. Object segmentation in video: a hierarchical variational approach for turning point trajectories into dense regions. In Proc. of the IEEE International Conf. on Computer Vision (ICCV),
2011. 3 [60] Peter Ochs, Jitendra Malik, and Thomas Brox. Segmentation of moving objects by long term vidco analysis.
In IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), 2013. 1
[61] Scoung Wug Oh, Joon-Young Lee, Ning Xu, and Scon Joo Kim. Video object segmentation using space- time memory networks. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019.
[62] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafranice, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po- Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.
Dinov2: Learning robust visual features without supervision. In arXiv.org. 2023. 4, 7. 9, 15, 16. 18
[63] Anestis Papazoglou and Vittorio Ferrari. Fast object segmentation in unconstrained video. In Proc. of the
IEEE International Conf. on Computer Vision (ICCV), 2013.01. 3
[64] Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg. Jake Vanderplas, Alexandre Passos, and David Cournapeau. leari: Machine learning in Python. In Journal of Machine Learning Research (JMLR), 2011. 15
12

---

[65]
Federico Perazzi. Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Homung. A benchmark dataset and evaluation methodology for video object segmentation. In Proc. IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2016. 1, 6,7
[66] Georgy Ponimatkin, Nermin Samet, Yang Xiao, Yuming Du, Renaud Marlet, and Vincent Lepetit. A simple and powerful global optimization for unsupervised video object segmentation. In Proc. of the IEEE
Winter Conference on Applications of Computer Vision (WACV). 2022.
[67] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. In Proc. of the International Conf. on Learning Representations (ICLR), 2023, 2,
6, 7, 8, 9, 19
[68]
Oriane Siméoni. Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc. Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no label. In Proc. the British Machine Vision Conf. (BMVC), 2021. 2
[69]
Gautam Singh, Fei Deng, and Sungjin Ahn. Illiterate dall-c learns to compose. In Proc. of the International Conf. on Learning Representations (ICLR), 2022. 2,7
[70] Gautam Singh, Yi-Fu Wu, and Sungjin Ahn. Simple unsupervised object-centric learning for complex and naturalistic videos. In Advances in Neural Information Processing Systems (NeurIPS), 2022, 3, 7
[71] Karl Stelzner, Robert Peharz, and Kristian Kersting. Faster attend-infer-repeat with tractable probabílistic models. In Proc. of the International Conf. on Learning Representations (ICLR). 2019.
[72] Pavel Tokmakov, Cordelia Schmid, and Karteek Alahari. Learning segment moving objects. In
International Journal of Computer Vision (IJCV), 2019. 3
[73] Yi-Hsuan Tsai, Ming-Hsuan Yang, and Michael J Black. Video segmentation via object flow. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR). 2016.
[74] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017. 15
[75] Rishi Veerapaneni, John D Co-Reyes, Michael Chang, Michael Janner, Chelsea Finn, Jiajun Wu, Joshua Tenenbaum, and Sergey Levine. Entity abstraction in visual model-basee reinforcement learning. In Proc. Conf. on Robot Learning (CoRL), 2020. 2
[76] Huy V Vo, Patrick Pérez, and Jean Ponce. Toward unsupervised. multi-object discovery in large-scale
image collections. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 2
[77] Van Huy Vo, Elena Sizikova, Cordelia Schmid, Patrick Pérez, and Jean Ponce. Large-scale unsupervised object discovery. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
[78] Paul Voigtlaender, Yuning Chai, Florian Schroff, Hartwig Adam, Bastian Leibe, and Liang- -hiich Chen. Feelvos: Fast end-to-end embedding learning for video object segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 3
[79] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy, Tracking emerges by colorizing videos. In Proc. of the European Conf. Computer Vision (ECCV), 2018. 3
[80] Xudong Wang. Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2023.
[81] Yangtao Wang, Xi Shen, Shell Xu Hu, Yuan Yuan, James L Crowley, and Dominique Vaufreydaz. Self- supervised transformers for unsupervised object discovery using normalized cut. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022. 2
[82] Nicholas Watters, Loie Matthey, Christopher P Burgess, and Alexander Lerchner. Spatial broadcast decoder: A simple architecture for learning disentangled representations in vaes. In arXiv.rrg, 2019. 6 [83] Junyu Xie, Weidi Xie, and Andrew Zisserman. Segmenting moving objects via an object-centric layered representation. In Advances in Neural Information P Processing Systems (NeurIPS), 2022. 7. 8, 19, 21
[84] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 2022. 2
[85] Ning Xu, Linjie Yang. Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. In Proc. of the Furopean Conf. on
Computer Vision (ECCV), 2018. 1
[86] Charig Yang. Hala Lamdouar, Erika Lu, Andrew Zisserman, and Weidi Xie. Self-supervised video object segmentation by motion grouping. In Proc. of the IEEE International Conf. on Computer Vision (ICCV),
2021. 3
[87] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1,2, 6
[88] Zhao Yang. Qiang Wang. Luca Bertinetto, Weiming Hu, Song Bai, and Philip HS Torr. Anchor diffusion for unsupervised video object segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
13

---

[89] Peiyu Yu, Sirui Xie. Xiaojian Ma. Yixin Zhu, Ying Nian Wu, and Song-Chur Zhu. Unsupervised foreground extraction via deep region competition. In Advances Neural Information Processing Systems (NeurlPS), 2021.
[90] Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo Rezende. Parts: Unsupervised segmenta- tion with slots, attention and independence maximization, In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021.
14

---

Appendices
In this supplementary material, we discuss the broader impact (Section A), report experiment details including dataset, model, and training details (Section B), provide more details of Invariant Slot Attention, as mentioned in the main paper (Section C), perform extra ablation studies on input resolution and generalization performance of our model (Section D), compare the performance to supervised models (Section E), provide an analysis of failure cases (Section F), and finally present extra visualizations (Section G). In the last section, we provide a visual comparison between models in our ablation studies as well as previous work.
A Broader Impact
We proposed object-oriented approach that can be applied videos of real-world environments. We trained our model on the Youtube- VVI 2019 dataset that includes videos of humans. The proposed work can be used to locate and segment multiple instances of wide variety of objects including all kinds of animals and humans from videos. While progress in this area can be used to improve not only human life but also, for example, wildlife, acknowledge that it could also inadvertently assist in the creation of computer vision applications that could potentially harm society.
B Experiment Details
B.1 Dataset Details
We eliminated the black borders for all videos in YTVISI9 dataset. Since we propose a supeevupe model, we merge the available splits on datasets for training. For all datasets except YTVIS, we use the available validation splits for evaluation. The annotations for the validation split missing on the YTVIS, therefore, we use a subset of 300 videos from the train split for evaluation. We will share the indices of selected videos for future comparisons together with the code. For evaluation, we upsample the segmentation masks to match the resolution of the original input frames by using bilinear interpolation.
B.2
Model Details
Feature Extractor (ÓDINO): We use the ViT-B/14 architecture with DINOv2 pretraining [62] as our default feature extractor. Our feature vector is the output of the last block without the CLS token. We add positional embeddings to the patches and then drop the tokens.
We project the feature tokens from ΦDINO to slot dimension Dator = 128, with a 2-layer-MLP, followed by layer normalization. Then the slots and the projected tokens are passed to the Invariant Slot Attention (ISA) (as detailed in Section C) as input. After slot attention, slots are updated with a GRU cell. Following the sequential update, slots are passed MLP with a hidden size of 4 x D All projection layers (p, q, k. v, g) have the same size as slots, i.e. Datee We repeat the binding operation 3 times. We multiply the scale parameter S, by prevent relative grid G from containing large numbers.
Spatial Binding((s
residual
to
We the following initializations for the learnable parameters: Gats, a coordinate grid in the range 1]; slots z, Xavier initialization [24]; slot scale S, and slot position Sp, a normal distribution.
Temporal Binding (t/r-bima): We use transformer encoder with 3 layers and 8 heads [74] for temporal binding. The hidden dimension of encoder layers is set to 4 x D We initialize the temporal positional embedding with a normal distribution. We masked the slots of not available frames, Lee frames with indices that are either less than 0 or exceed the frame number, in transformer layers.
Slot Merging (Wmerge): We use the implementation of Agglomerative Clustering the sklearn library [64] with complete linkage. For each cluster, we compute the mean slot and the sum of attention matrices, iee matrix A (13), for the associated slots. We determine the scale S, and position Sp parameters for the merged attention values to calculate the relative grid G of the new
15

---

slots. Then. G is projected onto sttor using linear layer h and added to the broadcasted slots before decoding.
Decoder Mapper (#mmpper); The mapper maaapprr consists of 5 linear layers with ReLU activations and hidden size of 1024. The final layer maps the activations to the dimension of VTT-B tokens with an extra alpha value i.e. 768 + 1.
B.3 Training Details
In all our experiments, unless otherwise specified, we employ DINOv2 [62] with the ViT-B/14 architecture. We set the number of consecutive frame range n to 2 and drop half of the tokens before the slot attention step. We train our models on 2x V100 GPUs using the Adam [39] optimizer with a batch size of 48. We clip the gradient norms at to stabilize the training. We match mask indices of consecutive frames by applying Hungarian Matching on slot similarity to provide temporal consistency. To prevent immature slots in slot merging, we apply merging with a probability that is logarithmically increasing through epochs.
MOVi-E: We train our model from scratch for total of 60 epochs, which is equivalent to approx- imately 300K iterations. We use a maximum learning rate of 4 10 and an exponential decay schedule with linear warmup steps constituting 5% of the overall training period. The model is trained using 18 slots and the input frames are adjusted to size of 336 x 336, leading to 576 feature tokens for each frame. The slot merge coefficient in M'merge configured to 0.12.
YTVISI9: Similar to MOVi-E, we train the model from scratch for 180 epochs, corresponding to approximately 300K iterations with a peak learning rate of 4 10 4 and decay it with an exponential schedule. Linear warmup steps are introduced for 5% of the training timeline. The model training involves slots, and the input frames are resized dimensions of 336 x 504, resulting in 864 feature tokens for each frame. The slot merge coefficient in d'merge is set to be 0.12.
Due to the small size of DAVIS17. we fine-tune the model pretrained the YTVIS19 dataset explained above. We finetune on DAVIS17 for 300 epochs, corresponding to approximately 40K iterations with reduced learning rate of 1 x 10-4, We use the same learning rate scheduling strategy as in YTVIS19. During the fine-tuning process, we achieve the best result without slot merging, likely due to the fewer number of objects, typically object the center, DAVIS17 compared to YTVIS19.
DAVIS17:
C Invariant Slot Attention
In this section, we provide the details of invariant slot attention (ISA), initially proposed by Biza et al. [3]. We use ISA in our Spatial Binding module V'sdind with shared initialization as explained in the main paper. Given the shared initialization 2, = {zz), St Sip, Gah,,)) ) )= our goal is to
j=1'
update the K slots: {x/] jj=1-
} For clarification, in the following, we focus on the computation of
single-step slot attention time step T:
A' softmax (M)) R. K
MP
((G.) (
(x)) ( E ERN
(G
(13)
VDai
where p. g. k. and g are linear projections while the relative grid of each slot is defined as:
G S
G
R
N22
(14)
S
The slot attention matrix A from (13) is used to compute the scale S, and the positions S,, of slots following Biza et al. [3]:
S sum AO O (Gabs,r - 5 2
sum (A' O Gabs.r) I
E 2
s/:=
(15)
sum (A))
sum (A))
After this step, following the original slot attention, input features aggregated to slots using the weighted mean with another linear projection 2:
U U: = WT ( (f.) g (Grl.r) )) E KxD
W A
E N
(16)
sum (At)
16

---

Then, U from (16) is used to update slots {z) 1= { with GRU followed by an additional MLP as residual connection as shown in (17). This operation is repeated times.
z MLP ((norm(z))
z: GRU (z, U) E IR k Par
(17)
D Additional Ablation Studies
In this section, we provide additional experiments to show the effect of resolution on the segmentation quality. We also report the results by varying the evaluation split on the YTVIS to confirm the generalization capability our model.
Effect of Resolution: We conducted experiments to investigate the effect of the input frame resolution, L.e. the number of input tokens, in Fig. 8. Specifically, we experimented with resolutions of 168 x 252, 224 x 336, and our default resolution 336 x 506, corresponding to 216, 384, and 864 input feature tokens, respectively. These experiments show that the imput resolution is crucial for segmentation performance.
As stated before, cannot the validation split of YTVIS19 due to manual annotations that are not publicly available. For evaluation, we only choose subset of 300 videos. Here, perform an experiment to examine the effect of varying the set of evaluation videos on performance.
Varying The Evaluation Split:
We repeat the experiment in Table 3 by choosing mutually exclusive subsets of 300 videos, times. We report mean and standard deviation (u of experiments for each model in Table 6. These results coherent with the reported result on the fixed subset, showing that segmentation performance peaks when all of our components are combined, i.e. model E. Similarly, removing our components b/idbind (model D) and bind (model C) Removing both (model B), results the worst performance, even falling behind its counterpart without slot merging (model A). Overall, the results of varying the evaluation set agree with the reported performance on the selected subset in the main paper. This shows that the performance of model generalizes different evaluation subsets.
time leads performance drop, shown in Table 3.
Table Varying evaluation splits. The effect of changing evaluation sets for models in the component ablation.
mloU
42
Model
mIoU
FG-ARI 1
40
37.72
0.82
29.06 2.24
200
300
400
500
600
700
800
37.20
1.53
30.15 1.90
Input Tokens
42.76
+ 2.09
30.03 + 1.67
D
42.98
2.01
29.69
2.02
Figure 8: Input resolution. The effect of input resolution, iee the number of tokens, on perfor- mance (mloU).
43.28
2.57
31.33 + 1.51
E Additional Comparisons
Table Comparison to supervised models. These results show the video multi-object evaluation results on the validation split of DAVIS17 between our model and supervised models.
Model
Unsupervised
mloU t
UnOVOST [55]
x
66.4
Propose-Reduce [48]
X
67.0
Ours
30,2
compare the performance our model to available supervised models, exploiting pre-trained object detectors large data in supervised manner, for unsupervised video object segmentation on DAVIS17 in Table 7. The results underscore a notable gap between supervised and unsupervised approaches.
17

---

F
Failure Analysis
Although our model can detect in-the-wild objects in different scales, segmentation boundaries are not perfectly aligned with the object due to patch-wise segmentation, as has been pointed out in identify three types of commonly occurring failure cases: (i) The over-clustering issue that remains unresolved in some cases even with slot merging. (ii) The tendency to cluster nearby instances of the same class into a single slot. (iii) Failure to detect small objects. particularly when they are situated near large objects. We provide visual examples of these cases Fig.
the Discussion (Section 5). In addition to these observations,
First Frame
Middle Frame
Last Frame
First Frame
Middle Frame
Last Frame
RGB
GT
Ours
(a) Failure cases due to over-clustering.
GT
Qurs
(b) Failure cases due grouping nearby instances of the same class into one cluster.
RGB
GT
Ours
(c) Failure cases due to missing relatively small objects.
Figure 9: Failure cases of our model with potential reasons, grouped into three.
G
Additional Visualizations
We provide additional qualitative results in Fig. 10. Our model can recognize not only the most salient object in the middle but also small, subtle objects in the background. Furthermore, it can handle varying number of objects in the scene with slot merging.
Feature Extractor: In Fig. 11, we provide visualizations different feature extractors, correspond- ing to the quantitative evaluation in Table 5, including DINOv2 [62]. DINO [7], and Supervised. self-supervised models performs better than the supervised one, also qualitatively. In particular, DINOv2 stands out for its exceptional capability to learn object representations across a diverse range
18

---

First Frame
Middle Frame
Last Frame
First Frame
Middle Frame
Last Frame
RGB
GT
Ours
RGB
G
OuR
RGB
GT
Ours
Figure 10: Qualitative results of multi-object video segmentation on YTVIS19
of categories. It also effectively identifies and segments intricate details that missed by other feature extractors such as tree branches on the left and the bag on the table on the right.
Components: In Fig. 12, we visually compare the segmentation results of the models corresponding the component ablation study the main paper (Table ). First of all, model which corresponds to the temporally consistent DINOSAUR [67]. struggles to cluster the instances
whole and also fails to track objects, for instance, the human on the right, due to discrepancies in mask index across three frames. With the help of slot merging, model B effectively addresses the over-clustering issue, as observed the mask of the calf on the left. Integrating our binding modules vir-bind and e/s-bind, resulting in model C and D, respectively, leads to a marked improvement in both segmentation and tracking quality. On the other hand, both models C and have shortcomings in detecting the human in certain frames of the right video. Finally, combining them, our full model, i.e. model E, excels at segmenting and tracking not only labeled objects but also other objects, such as the car visible in the first frame of the second video.
We provide a visual comparison between OCLR [83] and our model Fig. 13 after matching the ground-truth and predictions. OCLR fails to detect static objects (top-left, middle-left, bottom-left) and deformable objects (middle-right) due to failures of optical flow in these cases. Moreover, it considers moving regions as different objects such the water waves (top-right). On other hand, SOLV can accurately detect all objects as whole.
Comparison:
19

---

First Frame
Middle Frame
Frame
Frame
Middle Frame
Last Frame
RGB
GT
Sup.
DINO
DINOV2
Figure 11: Qualitative comparison of different pretraining methods for visual encoder.
First Frame
Middle Frame
Last Frame
First Frame
Middle Frame
Last Frame
RGB
GT
A
D
Figure Qualitative results of different models in the component ablation study (Table 3).
20

---

First Frame
Middle Frame
Frame
Frame
Middle Frame
Last Frame
RGB
GT
OCLR
Qurs
RGB
GT
OCLR
Ours
RGB
GT
OCLR
Ours
Figure 13: Qualitative results of multi-object video segmentation YTVIS19 after Hungarian Matching applied. Segmentation results of OCLR [83] are provided third row.
21


Reviews:
[{"review_id": "QYSxnZUDMm", "reviewer": "Reviewer_xM1B", "summary": "This paper proposes a Self-supervised Object-centric Learning framework for unsupervised multi-object video segmentation. To achieve this, this paper proposes to derive object-centric representations in a self-supervised manner to facilitate video segmentation tasks. The proposed approach adopts axial spatial-temporal slots attention to group visual regions with similarity properties, while exploiting the training techniques of masked feature reconstruction and slot merging. Experimental results on synthetic and real video datasets demonstrate superior performance against previous works.", "questions": "1. This proposed method utilizes a pre-defined number of slots to allow multi-object segmentation (e.g., 8 slots). Can the trained model be applicable to an image with more than 8 objects?\n2. In Table 3, why did adding spatial and temporal binding (Model-C and D) deteriorate the FG-ARI score compared with Model-B?\n3. Also in Table 3, without temporal binding (Model-D) just slightly drops the performance of the full version (Model-E). Does it imply that the temporal information would not be necessarily considered if the frame-by-frame segmentation is well performed?", "limitations": "The authors have listed the limitations of this proposed method and the potential future research directions to alleviate these raised issues.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Learning object-centric representations in an unsupervised fashion is vital yet challenging for several downstream applications, such as video object segmentation or robot manipulation. The task aiming to be addressed in this paper is significant.\n2. The experiment results and analysis both in the main paper and supplementary material are comprehensive.\n3. The overall paper is easy to follow.", "weaknesses": "My primary concern lies in the novelty and significance of the proposed framework. The novelty and contributions of the proposed approach are unclearly described in this paper. In the current paper draft, though it employs and modifies a series of existing techniques, it is not evident how the proposed approach is novel or provides significant advancements over existing methods. For example, COMUS [A] also learns self-supervised object-centric representations from DINO, aiming to achieve unsupervised object segmentation. The spatial binding is slightly modified from invariant slot attention [3], and the temporal binding is simply realized by the self-attention mechanism. In the visual decoder, the slot merging process is based on Agglomerative Clustering (AC) algorithm, while the decoder architecture follows DINOSAUR [62]. In summary, the proposed method seems to be a combination of existing approaches to perform the task of multi-object video segmentation, it is currently unclear how their combination in this framework constitutes a significant improvement or novelty over existing methods.\n \n[A] Zadaianchuk et al., Unsupervised Semantic Segmentation with Self-Supervised Object-Centric Representations. ICLR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough review and insightful comments. We have offered comprehensive clarifications and hope these will address your concerns.\n\n## W1. Concern on novelty and significance\nPlease refer to Q1 of the global response for our contributions. We would like to highlight that our model is distinct from COMUS [1]. COMUS clusters features of object proposals at the dataset level for image-level semantic segmentation and does not employ any slot-attention.\n\n## Q1. On the number of slots vs. the number objects \nPlease refer to Q3 of the global response.\n\n## Q2. FG-ARI gets worse while adding spatial-temporal binding\nPlease refer to Q2 of the global response.\n\n**References**\n\n[1] Zadaianchuk et al., Unsupervised Semantic Segmentation with Self-Supervised Object-Centric Representations. ICLR23"}, {"review_id": "cei0vb5ihw", "reviewer": "Reviewer_Sj2S", "summary": "This paper proposes a new self-supervised method for multi-object segmentation in videos called SOLV (Self-supervised Object-centric Learning for Videos). It adopts axial spatial-temporal slot attention to group pixels into slots within frames and then relates these slots across frames to track objects. It uses the masked autoencoder training objective to reconstruct visual features from latent slots. A slot merging strategy based on agglomerative clustering is proposed to address over-segmentation by dynamically merging similar slots. Experiments show state-of-the-art results on the MOVi-E synthetic dataset and Youtube-VIS 2019 real videos without requiring additional modalities like optical flow.", "questions": "Please see the weakness part.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The problem setup and overall approach are clearly explained.\nThe performance looks great. The masked feature reconstruction seems interesting. \nThe spatial-temporal binding and the slot merging make sense to me.", "weaknesses": "A comparison with self-supervised baselines on the real data (DAVIS17 / YTVIS19) needs to be added, as there is a huge performance gap between synthetic data and real data (80 vs 30). Such a huge gap makes it questionable whether the results on synthetic data is really representative. Also, it would be great to also add recent supervised methods to help readers understand the gap between self-supervised / supervised methods.\n\nIt would be great also to add an error analysis to showcase the failure mode and model behaviors.\n\nIt seems that the model needs a superior pretrained backbone (DINOv2) to achieve good performance. DINOv2 is pretrained on a large, curated dataset LVD-142M, which gives the authors an advantage over the baselines. A more fair comparison should be made in Table 2 when compared with DINOSAUR, which is equipped with DINO. I encourage the authors use more text to describe the necessity of using a well-pre-trained encoder instead of simply saying using a frozen encoder in the model.\n\nGiven those concerns, I feel like the current results do not show enough evidence for acceptance, I'd be happy to increase my rating if the authors could address my concerns by showing more results (especially for real data with the same visual encoder). \n\nMisc:\nI suggest the authors revise Fig.2 and its caption to make it more self-contained. Now the illustration is unclear and has many symbols that are not explained in the caption, making it difficult to understand without referring to the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive feedback and valuable comments, we have provided detailed clarification, hope these can resolve your concern, thus raise the score accordingly.\n\n## W1. Comparisons on real data\nWe agree with the reviewer that performance on synthetic data is questionable, this is exactly the reason to benchmark on real videos. As far as we know, our work is the first model that tries to segment multiple objects on **real-world data**, without relying on any other modality, for example, optical flow, depth etc. \n\nIn Table 2, we have compared our model to the recent SOTA unsupervised video segmentation model, namely, OCLR, that is trained by using optical flow, and SOTA image-based unsupervised segmentation, namely, DINOSAUR (with a powerful DINOv2 backbone), in Table 3. \n\nSupervised models for unsupervised video object segmentation generally exploits pretrained object detectors, as requested by the reviewer, for reference, existing supervised SOTAs of UVOS in DAVIS17 validation set are Propose-Reduce [1] and UnOVOST [2] with mIoUs of 67.0 and 66.4, respectively. We will add these numbers into the table in our final paper.\n\n## W2. Error analysis\nThanks for the suggestion. In fact, we have actually provided common failure cases in Supplementary (line 136 and Figure 13), we will move those to the main text of the final paper with more explanations.\n\nGenerally speaking the failure mode can be cast into four categories:\n\n**(i) lack of sharp boundaries:** This is because we encode and process the image on the patch-level (14 x 14 patches), therefore cannot recover some of the pixel-level details such as boundaries while upsampling the patch-level segmentation results into the original resolution.\n\n**(ii) overclustering:** This refers to the case where a single object is represented by multiple slots, this often occurs when the object has distinct visual features that are hard to be grouped together.\n\n**(iii) underclustering nearby instances:** When multiple instances of the same semantic class are represented by a single slot, it occurs due to their similar features and positional encodings. This similarity, stemming from their spatial proximity, results in their reconstruction using just one slot.\n\n**(iiii) missing small objects:** When encountering small objects, the model often doesn't allocate a separate slot for them because they minimally impact the reconstruction loss.\n\nQualitative examples of failure cases can be found in Supplementary, Figure 13.\n\n## W3. The model needs a superior pre-trained backbone\nThank you for the comments. In order to compare with DINOSAUR fairly, we have re-trained it with a DINO-v2 backbone, as explained in line 263. Comparing Model-A (DINOSAUR) and Model-E (ours) in Table 3, it can be observed that ours substantially outperform the DINOSAUR model, using the same backbone, i.e., 37.8 vs. 45.3 mIOU. In addition, we also experimented with the DINO backbone in Table 5, and it can be seen, our model even outperforms DINOSAUR with DINO-v2 backbone, i.e.,  37.8  vs. 41.9 mIOU. We have provided more discussion about the effect of different backbones (specifically DINO vs DINOv2) and provide qualitative results in Supplementary. More examples with DINO and DINO-v2 are shown in Fig 10, 11 in FD. In some cases, DINO backbone performs better than DINO-v2, mostly due to more distinct features of different regions leading to overclustering  (Fig 12, 13 in FD).\n\nThe DINO series has been widely used for unsupervised object segmentation in the literature. As suggested by the reviewer, we will add DINOSAUR with DINOv2 backbone (model-A) to Table 2 as well, and add more discussion on the necessity of using a well-trained self-supervised backbone in line 293 paragraph.\n\n## W4. Misc\nThank you for the suggestion, we will revise the caption of the main figure and explain each symbol and operation briefly.\n\n**References**\n\n[1] Lin et al., Video Instance Segmentation with a Propose-Reduce Paradigm, ICCV21\n\n[2] Luiten et al., Unsupervised Offline Video Object Segmentation and Tracking, WACV20\n"}, {"review_id": "S3bt8D87hk", "reviewer": "Reviewer_xWs8", "summary": "This paper proposes an unsupervised segmentation method in videos. The backbone network is pre-trained with the self-supervised method. Then, the model spatially binds objects to slots on each frame and then relates these slots across frames. The framework is trained to reconstruct the middle frame in a high-level semantic feature space. ", "questions": "1. What's the advantage of the proposed model compared to the recent Segment Anything model?\n2. In slot merging, whether the clustering method will affect the performance?\n3. What if the method is applied to more complex scenes which have more than 12 objects?", "limitations": "see weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is able to segment a real work video without supervision by utilizing a powerful pre-trained feartures.\n2. The performance of the proposed method is better than comparison methods.\n3. The method is evaluted on both sythethic and real-world dataset. ", "weaknesses": "1. Although the method can roughly segment the objects in videos, the method is not capable of detecting the boundary of objects.\n2. The method can only be applied to videos since it relies on temporal information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's feedback and perceptive remarks. We have provided responses and hope these will resolve any issues you've raised.\n## W1. The method is not capable of detecting the boundary of objects\nWe agree with the reviewer. Our method currently only segments objects at patch-level (due to the ViT backbone), and this causes a loss of pixel-level details at the boundaries. However, given more computation resources available, scaling the image resolution, reducing the patch size, or using feature pyramid may resolve this limitation, we leave this as future work.\n\n## W2. The model relies on temporal information\nWe agree with the reviewer. We believe that using temporal information for self-supervised video object segmentation is the right way to go, as we humans are experiencing dynamic scenes everyday, rather than static frames.\n\n## Q1. Differences to SAM\nThe fundamental difference lies in the source of supervision signals. Specifically, SAM is a supervised model, trained with 11M images and 1B+ masks, which is extremely costly to produce, while in contrast, our proposed model is trained in a self-supervised manner, by that we mean, the entire training procedure **does not** require any manual annotations.\n\n## Q2. Different clustering algorithms\nWe cannot use other clustering algorithms that require a predefined number of clusters. In agglomerative clustering, the number of clusters are dynamically set based on a threshold. For more details, please refer to the Q4 of the global response.\n\n## Q3. Slot number\nPlease refer to Q3 of the global response.\n"}, {"review_id": "Zu7AeRXG8Y", "reviewer": "Reviewer_gzBP", "summary": "The paper introduces an approach to segment multiple objects for video sequence in both real and synthetic data without utilizing any additional signals besides RGB frames. It has 3 components including a visual encoder, a spatial-temporal binding model used for grouping pixels into slots across different time, and a visual decoder to get the segmentation mask. The method also uses a slot merging strategy to address the over-segmentation issue caused by a fixed number of slots. The paper conducts experiments and it outperforms previous methods on synthetic and real data. The ablation studies are also presented to prove each component's effectiveness.", "questions": "For the video segmentation dataset like DAVIS17, how could you choose which segment mask as the object mask, not the background, used to calculate the metric? \n\nHow to make sure that the merging mechanism can successfully merge the masks into a whole object? How to determine which parts should be merged into one object?", "limitations": "It could be better to visualize several cases of each slot's segmentation mask results for slot=8 which is claimed to be the best in Table 4.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper can segment complex scenes in Youtube videos and achieve satisfying qualitative results. \nIt didn’t rely on extra signals in videos which is claimed not stable in video segmentation. \nThe drop approach can save memory and calculation time successfully. \nThis paper achieves great performance on MOVi-E.", "weaknesses": "This paper didn't test on single object segmentation dataset like DAVIS 2017 and compare methods like CIS, Y Yang 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the feedback and valuable comments you have shared. In response, we have furnished explanations, aiming to address your concerns effectively, hoping the scores can be raised accordingly.\n\n## W1. Single object segmentation\nWe did test DAVIS2017 in our paper, the results are reported in Table 2, it is a dataset of multi-object video segmentation. In terms of single video object segmentation, existing datasets are usually defined for motion segmentation, that assigns all pixels with similar motion into one group, regardless of semantic categories. \n\nWhile in our case, we aim to segment objects based on semantics, for example, in the given example from DAVIS17 (Fig 14 (a) in FD), the person and the bicycle are segmented into two different groups, consistent with the ground-truth (Fig 14 (b) in FD) and mIoU is calculated as 60.0. However, when ground-truth annotations are turned into single-object evaluation (Fig 14 (c) in FD), mIoU score drops to 41.2 (for the whole video). \n\nNevertheless, we provide the results of our models on DAVIS17, which is designed for multi object segmentation:\n- 6 slots, no merging: 39.06 mIoU\n- 8 slots, merging: 47.50 mIoU\n- 4 slots, merging: 49.01 mIoU\n- CIS [1]: 53.1 mIoU\n- DyStaB [2]: 58.9 mIoU\n\n## Q1. Mask matching for evaluation\nWe follow the common practice in this community, and use Hungarian Matching for evaluation, as also used in DINOSAUR, SAVi, OCLR. We will clarify this evaluation metric in our final paper.\n\n## Q2. Slot merging mechanism\nPlease refer to Q4 of the global response.\n\n## L1. Qualitative examples\nWe have provided additional qualitative results of our best model for masks of each slot in Figure 9 of Supplementary. Note that, although the model is trained with 8 slots, some cases result in fewer masks due to dynamically set number of slots in slot merging.\n\n\n**References**\n\n[1] Yang et al., Unsupervised Moving Object Detection via Contextuaş Information Separation, CVPR19\n[2] Yang et al., DyStaB: Unsupervised Object Segmentation via Dynamic-Static Bootstrapping, CVPR21\n"}, {"review_id": "uLqzGY7d7C", "reviewer": "Reviewer_zEVX", "summary": "This paper introduced a fully unsupervised method, SOLV, for segmenting multiple objects in real-world sequences. In this paper, the author employed the token drop strategy to reduce computation and enhance regularization. Additionally, Spatial-temporal Binding was proposed to aggregate the features of objects within and across frames, resulting in a remarkable improvement in mIoU. To address over-clustering, the author employed Agglomerative Clustering to merge slots. It is noteworthy that SOLV not only significantly advances the state-of-the-art on commonly used simulated data but also stands out as the first fully unsupervised method to demonstrate state-of-the-art performance on unconstrained videos from the Youtube-VIS dataset.", "questions": "1. In the ablation experiments, a notable observation arises regarding the performance of Model-C and Model-D compared to their predecessor, Model-B, as they exhibit a decrease in FG-ARI. However, Model-E, which combines the spatial bind and temporal bind components from both Model-C and Model-D, ultimately achieved superior results in terms of FG-ARI. What are the possible reasons behind this phenomenon？\n\n2. Whether the shape of $\\mathcal V_t$ at line 120 is indeed $\\mathbb R^{(2n+1)\\times N \\times (3P^2)}$, and the dimension of $\\mathcal V_t'$ in Equation (2) should be $\\mathbb R^{(2n+1)\\times N \\times (3P^2)}$.\n", "limitations": "1. As discussed at the end of the paper, it is acknowledged that Agglomerative Clustering is non-differentiable during the training process. Therefore, it might be worthwhile to explore alternative clustering algorithms as potential alternatives.\n2. Based on the visualized results, it is evident that SOLV excels in the detection and clustering of foreground objects. However, there is room for improvement in terms of the accuracy of segmentation boundaries.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A fully unsupervised object-centric learning method for scaling multi-object segmentation to in-the-wild videos was first proposed and achieved significant improvements on traditional synthetic datasets. Moreover, it has delivered impressive results on real-world video datasets.\n2. Many details in the method, such as token drop ratio, number of slots, and variations of the visual encoder, have undergone ablation experiments, enhancing the robustness and reproducibility of the work.\n3. The slot merging strategy significantly reduces the sensitivity of the model's performance to the number of slots.\n4. Both temporal binding and spatial binding techniques bring significant gains in mIoU.", "weaknesses": "1. A more elaborate explanation is required to clarify how the Agglomerative Clustering algorithm is employed to dynamically determine the optimal number of slots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our sincere appreciation to the reviewer for their valuable insights and constructive feedback. We have responded to their questions and concerns, and we hope our explanations thoughtfully encompass the raised points.\n\n## W1. Explanations on slot merging\nPlease refer to Q4 of the global response.\n\n## Q1. Reasons behind the model behaviours in the ablation\nPlease refer to Q2 of the global response.\n\n## Q2. Typo in the text\nWe appreciate the reviewer for their extremely careful reading and inspection. We will add the missing patch size in line 120.\n"}]

Author rebuttal:
We appreciate all reviewers for their valuable comments and feedback. We hope the following response can fully resolve the raised concerns. For referred images, please see the **Figure Document (FD)** attached below.

## Q1. Contribution Summary
We would like to start the rebuttal by elaborating our contributions in this paper:

**(i) On the considered task:** We propose the first unsupervised video multi-object segmentation model that can work on both real-world and synthetic videos, without relying on any other modality, for example, optical flow, depth etc.

**(ii) On architecture design:** we propose a conceptually simple architecture that consists of 
* **DINO backbone:** *We are the first to address multi-object segmentation in real-world videos by performing reconstruction in the feature space of DINO.* Furthermore, we perform masked feature reconstruction which provides efficiency in time and memory (as pointed out by reviewers Sj2S, gzBP)
* **S-Bind & T-Bind:** *We extract spatially invariant slots (S-Bind) and relate them to each other through time (T-Bind).* Our spatial binding module is a modified version of ISA, as we mentioned in the paper. T-Bind attends slots with the same index across time for consistent slot representations. The two binding modules provide complementary information to slots
* **Slot Merging:** *Given distinctive slot representations learned by our model, we merge slots to solve over-clustering issue.* Over-clustering is a well-known issue in unsupervised object-centric segmentation due to predefined number of slots. Our merging module adopts a simple agglomerative clustering to tackle this problem, as will be detailed in response to Q4
* **Overall Pipeline:** *We propose a distinct architecture compared to previous work*. Specifically, existing object-centric methods for videos follow a similar approach by simply propagating the updated slot information to the next frame through the video. We adopt a totally different approach by extracting slots in frame level, relating them temporally and training it with feature reconstruction

**(iii) Comparison with other approaches:** Compared to other methods, ours outperforms in synthetic data (Table 1) and handles real-world videos. Ablation studies validate our binding modules' efficacy (Table 3) and emphasise the memory efficiency of our masked feature reconstruction (Fig 7). Our slot-merging addresses the over-clustering issue prevalent in object-centric models (Table 4)

## Q2. Behaviours of Models
Reviewers ask about the performance variance between models in our model ablation table (Table 3).
In our ablation study, two key observations emerge:
* FG-ARI, which we report to follow the common practice, is not an accurate indicator of segmentation quality in videos, as mIoU is the widely accepted metric for such tasks
* Both model-C and model-D tend to over cluster objects when compared to model-B

We explain the reasons below with examples in detail:
* FG-ARI considers only in-frame segmentation. FG-ARI does not consider the consistency of indices across frames since it is calculated per frame. On the other hand, mIoU evaluates temporal consistency
* FG-ARI becomes either 1 or 0 when there is only one object. FG-ARI calculation becomes problematic when there is a single object to evaluate. Assume that there is only one labelled object and 4 pixels for it: 
`ARI([0, 0, 0, 0], [0, 1, 1, 1]) = 0`. Therefore, if there is only one GT object with a small mistake in the mask, FG-ARI considers it completely wrong. This works for the benefit of model-B roughly segmenting the objects (Fig 1, 2 in FD)
* Each binding module increases the slot specialisation. As reflected in quantitative results, combining the two binding modules increases the total consistency, both for merging and tracking. Slot specialisation is reinforced by both temporal information and invariant visual representations. To show this behaviour qualitatively, we visualise the similarity matrices comparing slot representation inframe and between consecutive frames for model-C, model-D, and model-E in Fig 3, 4, 5 in FD, respectively
* Due to slot specialisation, over-clustering occurs more in model-C and model-D. After inspecting specific cases where model-D and model-C have higher mIoU but lower FG-ARI than model-B, we found that increased slot specialisation leads to overclustering. Specialisation improves tracking, but FG-ARI does not get affected by errors in tracking. Please see Fig 6 in FD to compare segmentations of model-B and model-D for a sample video.
* Overall, due to (i) issues with FG-ARI (ii) tendency to over cluster, model-C and model-D have a lower FG-ARI than model-B. On the other hand, full model (model-E) reaches the FG-ARI of model-B also with a large improvement in mIoU (7.09).

## Q3. Number of Slots vs. Number of Objects
The model tends to undercluster the scene if the predefined number of slots is less than the optimal number. Specifically, our model has a tendency to merge nearby objects into a single slot (Fig 7 (a), (b) in FD) and misses some objects in the background (Fig 7 (c) in FD) when the optimal number of slots is higher than the predefined one. 

## Q4. On Slot Merging with Agglomerative Clustering
Empirically, we found that parts of an object might be assigned to separate slots, though these slots still have similar representations (please refer to Fig 3 of the paper). Agglomerative Clustering (AC) works by merging data points based on similarity until a group similarity threshold is reached, therefore, we group similar slots into a dynamically defined number of clusters with AC. It does not need to set a predefined number of clusters as in k-means, and instead, it can dynamically determine the number of groups based on an empirically found similarity threshold. We provide some clustering examples with different threshold values, using the slots of the model trained without slot merging in Fig 8, 9 in FD.

Meta-review: This paper proposes a Self-supervised Object-centric Learning framework for unsupervised multi-object video segmentation. To achieve this, this paper proposes to derive object-centric representations in a self-supervised manner to facilitate video segmentation tasks. The proposed approach adopts axial spatial-temporal slots attention to group visual regions with similarity properties, while exploiting the training techniques of masked feature reconstruction and slot merging. During the review stages, issues like clarification/motivation, novelty/technical contributions, and experiment/justification were raised. The authors also provided detailed responses to address the raised issues, and the reviewers all lean toward the acceptance rating. Therefore, I think this paper is above the threshold for publication.
Decision: Accept (poster)