SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Datasets: Our proposed method is evaluated on one synthetic and two real-world video datasets. For the synthetic dataset, we select MOVi [25], a widely-used benchmark for evaluating object-centric methods, particularly for multi-object segmentation in videos. To ensure a rigorous evaluation, we use the challenging MOVi-E dataset with up to 20 moving and static objects and random camera motion, as suggested by Bao et al. [2]. For the real-world datasets, we use the validation split of DAVIS17 [65]. In addition, we evaluate our method on a subset of the Youtube-VIS 2019 (YTVIS19) [87] train set, because there is no official validation or test set provided with ground-truth masks. Specifically, the first evaluation set consists of 30 videos out of 90 videos on DAVIS17, and the second evaluation set consists of 300 videos out of 2,883 high-resolution videos on YTVIS19. We perform all the ablations on the YTVIS19 dataset.
Metrics: For our synthetic dataset evaluation, we use the foreground adjusted rand index (FG-ARI) to measure the quality of clustering into multiple foreground objects. To maintain consistency with prior studies, we calculate the per-frame FG-ARI and report the mean across all frames, as done in
recent works such as Karazija et al. [36], Bao et al. [2], and Seitzer et al. [67]. This approach also allows us to compare to state-of-the-art image-based segmentation methods, such as DINOSAUR [67]. For real-world datasets, we use the mean Intersection-over-Union (mIoU) metric, which is widely accepted in segmentation, by considering only foreground objects. To ensure temporal consistency of assignments between frames, we apply Hungarian Matching between the predicted and ground-truth masks of foreground objects in the video, following the standard practice [65].
4.2 Results on Synthetic Data: MOVi-E
The results on the MOVi-E dataset are presented in Table 1, with methods divided based on whether they use an additional modality. For example, the sequential extensions of slot attention, such as SAVi [40] and SAVi++ [17], reconstruct different modalities like flow and sparse depth, respectively, their performance falls behind other approaches despite additional supervision. On the other hand, STEVE [69] improves results with a transformer-based decoder for reconstruction. PPMP [36] significantly improves performance by predicting probable motion patterns from an image with flow supervision during training. MoTok [2] outperforms other methods, but relies on motion segmentation masks to guide the attention maps of slots during training, with a significant drop in performance without motion segmentation. The impressive performance of DINOSAUR [67] highlights the importance of self-supervised training for segmentation. Our method stands out by using the spatial-temporal slot binding, and autoencoder training in the feature space, resulting in significantly better results than all previous work (see Fig. 5).
Table 1: Quantitative Results on MOVi-E. This table shows results in comparison to the previous work in terms of FG-ARI on MOVi-E.
Model	+Modality	FG-ARI ↑
SAVi [40]	Flow	39.2
SAVi++ [17]	Sparse Depth	41.3
PPMP [36]	Flow	63.1
MoTok [2]	Motion Seg.	66.7
MoTok [2]	52.4
STEVE [70]	-	54.1
DINOSAUR [67]	65.1
Ours	80.8
Figure 5: Qualitative Results on MOVi-E.
4.3 Results on Real Data: DAVIS17 and Youtube-VIS 2019
Due to a lack of multi-object segmentation methods evaluated on real-world data, we present a baseline approach that utilizes spectral clustering on the DINOv2 features [62]. To define the number of clusters, we conduct an oracle test by using the ground-truth number of objects. After obtaining masks independently for each frame, we ensure temporal consistency by applying Hungarian Matching between frames based on the similarity of the mean feature of each mask. However, as shown in Table 2, directly clustering the DINOv2 features produces poor results despite the availability of privileged information on the number of objects.
Table 2: Quantitative Results on Real-World Data. These results show the video multi-object evaluation results on the validation split of DAVIS17 and a subset of the YTVIS19 train split.
Model	Supervision	DAVIS17	YTVIS19
mIoU ↑	FG-ARI ↑	mIoU ↑	FG-ARI ↑
DINOv2 [62] + Clustering	#Objects	14.34	24.19	28.17	16.19
OCLR [83]	Flow	34.60	14.67	32.49	15.88
Ours	-	30.16	32.15	45.32	29.11
Figure 6: Qualitative Results of Multi-Object Video Segmentation on YTVIS19. We provide the first, middle, and last frames along with their corresponding segmentation after the Hungarian Matching step. Our model is proficient in accurately identifying objects in their entirety, irrespective of their deformability or lack of motion. This capability is highlighted in contrast to OCLR [83], whose results are displayed in the third row for comparison.
We also compare our method to the state-of-the-art approach OCLR [83], which is trained using a supervised objective on synthetic data with ground-truth optical flow. OCLR [83] slightly outperforms our method on DAVIS17, where the optical flow can often be estimated accurately, that greatly benefits the segmentation by allowing it to align with the real boundaries, resulting in better mIoU performance. However, our method excels at detecting and clustering multiple objects, even when the exact boundaries are only roughly located, as evidenced by the higher FG-ARI. The quality of optical flow deteriorates with increased complexity in YTVIS videos, causing OCLR [83] to fall significantly behind in all metrics. The qualitative comparison in Fig. 6 demonstrates that our method can successfully segment multiple objects in a variety of videos. Additional qualitative results can be found in the Supplementary Material.
On the Effectiveness of Architecture: We conducted an experiment to examine the impact of each proposed component on the performance. Firstly, we replaced the spatial binding module,  $\psi_{\mathrm{s - bind}}$  with the original formulation in Slot Attention [52]. Secondly, we removed the temporal binding module,  $\psi_{\mathrm{t - bind}}$  , which prevents information sharing between frames at different time steps. Finally, we eliminated the slot merging module,  $\psi_{\mathrm{merge}}$
We can make the following observations from the results in Table 3: (i) as shown by the results of Model-A, a simple extension of DINOSAUR [67] to videos, by training with DINOv2 features and matching mask indices across frames, results in poor performance; (ii) only adding slot merging (Model-B) does not significantly improve performance, indicating that over-clustering is not the primary issue; (iii) significant mIoU gains can be achieved with temporal binding (Model-C) or
Table 3: Components. The effect of changing our spatial binding module ( $\psi_{\mathrm{s - bind}}$ ) to the original slot attention, removing the temporal binding module ( $\psi_{\mathrm{t - bind}}$ ) or the slot merging module ( $\psi_{\mathrm{merge}}$ ).
Model	ψmerge	ψs-bind	ψt-bind	mIoU ↑	FG-ARI ↑
A	X	X	X	37.75	27.05
B	✓	X	X	38.23	29.09
C	✓	X	✓	44.95	28.42
D	✓	✓	X	44.94	27.38
E	✓	✓	✓	45.32	29.11
Table 4: Number of Slots. The effect of varying the number of slots with/without slot merging.
#Slots	Merging	mIoU ↑	FG-ARI ↑
6	X	43.29	27.73
✓	44.90	28.52
8	X	39.90	22.55
✓	45.32	29.11
12	X	36.04	20.20
✓	43.19	27.94
Table 5: Visual Encoder. The effect of different types of self-supervised pretraining methods and architectures for feature extraction.
Model	mIoU ↑	FG-ARI ↑
Supervised ViT-B/16	37.58	19.94
DINO ViT-B/8	39.53	24.70
DINO ViT-B/16	41.91	24.01
DINOv2 ViT-B/14	45.32	29.11
Figure 7: Token Drop Ratio. The effect of varying token drop ratio on performance (mIoU) and memory consumption.
spatial binding (Model-D), which shows the importance of spatially grouping pixels within each frame as well as temporally relating slots in the video. Finally, the best performance is obtained with Model-E in the last row by combining all components.
Number of Slots: We perform an experiment by varying the number of slots with and without slot merging in Table 4. Consistent with the findings in previous work [67], the performance is highly dependent on the number of slots, especially when slot merging is not applied. For example, without slot merging, increasing the number of slots leads to inferior results in terms of both metrics, due to the over-segmentation issue in some videos. While using slot merging, we can better utilize a larger number of slots as can be seen from the significantly improved results of 8 or 12 slots with slot merging. This is also reflected in FG-ARI, pointing to a better performance in terms of clustering. We set the number of slots to 8 and use slot merging in our experiments. We provide a visual comparison of the resulting segmentation of an image with and without slot merging in Fig. 4. Our method can successfully merge parts of the same object although each part is initially assigned to a different slot.
Token Drop Ratio: We ablated the ratio of tokens to drop in Fig. 7. As our motivation for the token drop is two-fold (Section 3.2.1), we show the effect of varying the token drop ratio on both performance and memory usage. Specifically, we plot mIoU on the vertical axis versus the memory consumption on the horizontal axis. We report the memory footprint as the peak in the GPU memory usage throughout a single pass with constant batch size. Fig. 7 confirms the trade-off between memory usage and performance. As the token drop ratio decreases, mIoU increases due to less information loss from tokens dropped. However, this comes at the cost of significantly more memory usage. We cannot report results by using all tokens, i.e. token drop ratio of 0, due to memory constraints. Interestingly, beyond a certain threshold, i.e. at 0.5, the token drop starts to act as regularization, resulting in worse performance despite increasing the number of tokens kept. We use 0.5 in our experiments, which reaches the best mIoU with a reasonable memory requirement. In summary, the token drop provides not only efficiency but also better performance due to the regularization effect.
Visual Encoder: We conducted experiments to investigate the effect of different visual encoders in Table 5. We adjust the resize values to maintain a consistent token count of 864 for different architectures. These findings underscore the critical role of pretraining methods. Please see Supplementary for a qualitative comparison. Notably, the DINOv2 model [62] outperforms its predecessor, DINO [7], in terms of clustering efficiency. On the other hand, models pretrained using supervised methods results in the weakest performance, with a substantial gap in the FG-ARI metric when compared to self-supervised pretraining methods. Furthermore, no specific patch size offers a clear advantage within the same pretraining type. For instance, DINO pretraining with ViT-B/8 exhibits superior performance in the FG-ARI metric but lags behind its 16-patch-sized counterpart in the mIoU metric.
We presented the first fully unsupervised object-centric learning method for scaling multi-object segmentation to in-the-wild videos. Our method advances the state-of-the-art significantly on commonly used simulated data but more importantly, it is the first fully unsupervised method to report state-of-the-art performance on unconstrained videos of the Youtube-VIS dataset.
This work takes a first step towards the goal of decomposing the world into semantic entities from in-the-wild videos without any supervision. The performance can always be improved with better
features from self-supervised pretraining as shown in the comparison of different DINO versions. While we obtain significant performance boosts from self-supervised pretraining, it also comes with some limitations. For example, our method can locate objects roughly, but it fails to obtain pixel-accurate boundaries due to features extracted at patch level. Furthermore, nearby objects might fall into the same patch and cause them to be assigned to the same slot. Our slot merging strategy, although has been proved effective as a simple fix for the issue of over-segmentation, remains not differentiable. Future work needs to go beyond that assumption and develop methods that can adapt to a varying number of objects in a differentiable manner.
Weidi Xie would like to acknowledge the National Key R&D Program of China (No. 2022ZD0161400).
[89] Peiyu Yu, Sirui Xie, Xiaojian Ma, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. Unsupervised foreground extraction via deep region competition. In Advances in Neural Information Processing Systems (NeurIPS), 2021. 2
[90] Daniel Zoran, Rishabh Kabra, Alexander Lerchner, and Danilo J Rezende. Parts: Unsupervised segmentation with slots, attention and independence maximization. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2021. 2
In this supplementary material, we discuss the broader impact (Section A), report experiment details including dataset, model, and training details (Section B), provide more details of Invariant Slot Attention, as mentioned in the main paper (Section C), perform extra ablation studies on input resolution and generalization performance of our model (Section D), compare the performance to supervised models (Section E), provide an analysis of failure cases (Section F), and finally present extra visualizations (Section G). In the last section, we provide a visual comparison between models in our ablation studies as well as previous work.
We proposed an object-oriented approach that can be applied to videos of real-world environments. We trained our model on the Youtube-VIS 2019 dataset that includes videos of humans. The proposed work can be used to locate and segment multiple instances of a wide variety of objects including all kinds of animals and humans from videos. While progress in this area can be used to improve not only human life but also, for example, wildlife, we acknowledge that it could also inadvertently assist in the creation of computer vision applications that could potentially harm society.
We eliminated the black borders for all videos in YTVIS19 dataset. Since we propose a self-supervised model, we merge the available splits on datasets for training. For all datasets except YTVIS, we use the available validation splits for evaluation. The annotations for the validation split are missing on the YTVIS, therefore, we use a subset of 300 videos from the train split for evaluation. We will share the indices of selected videos for future comparisons together with the code. For evaluation, we upsample the segmentation masks to match the resolution of the original input frames by using bilinear interpolation.
B.2 Model Details
Feature Extractor  $(\phi_{\mathbf{DINO}})$ : We use the ViT-B/14 architecture with DINOv2 pretraining [62] as our default feature extractor. Our feature vector is the output of the last block without the CLS token. We add positional embeddings to the patches and then drop the tokens.
Spatial Binding  $(\psi_{\mathrm{s - bind}})$ : We project the feature tokens from  $\phi_{\mathrm{DINO}}$  to slot dimension  $D_{\mathrm{slot}} = 128$  with a 2-layer-MLP, followed by layer normalization. Then the slots and the projected tokens are passed to the Invariant Slot Attention (ISA) (as detailed in Section C) as input. After slot attention, slots are updated with a GRU cell. Following the sequential update, slots are passed to a residual MLP with a hidden size of  $4\times D_{\mathrm{slot}}$ . All projection layers  $(p,q,k,v,g)$  have the same size as slots, i.e.  $D_{\mathrm{slot}}$ . We repeat the binding operation 3 times. We multiply the scale parameter  $\mathbf{S}_s$  by  $\delta = 5$  to prevent relative grid  $\mathbf{G}_{\mathrm{rel}}$  from containing large numbers.
We use the following initializations for the learnable parameters:  $\mathbf{G}_{\mathrm{abs}}$ , a coordinate grid in the range  $[-1, 1]$ ; slots  $\mathbf{z}$ , Xavier initialization [24]; slot scale  $\mathbf{S}_s$  and slot position  $\mathbf{S}_p$ , a normal distribution.
Temporal Binding  $(\psi_{\mathrm{t - bind}})$ : We use a transformer encoder with 3 layers and 8 heads [74] for temporal binding. The hidden dimension of encoder layers is set to  $4\times D_{\mathrm{slot}}$ . We initialize the temporal positional embedding with a normal distribution. We masked the slots of not available frames, i.e. frames with indices that are either less than 0 or exceed the frame number, in transformer layers.
Slot Merging  $(\psi_{\mathrm{merge}})$ : We use the implementation of Agglomerative Clustering in the sklearn library [64] with complete linkage. For each cluster, we compute the mean slot and the sum of attention matrices, i.e. matrix  $\mathbf{A}$  in (13), for the associated slots. We determine the scale  $\mathbf{S}_s$  and position  $\mathbf{S}_p$  parameters for the merged attention values to calculate the relative grid  $\mathbf{G}_{\mathrm{rel}}$  of the new
slot. Then,  $\mathbf{G}_{\mathrm{rel}}$  is projected onto  $D_{\mathrm{slot}}$  using a linear layer  $h$  and added to the broadcasted slots before decoding.
Decoder Mapper  $(\psi_{\mathrm{mapper}})$ : The mapper  $\psi_{\mathrm{mapper}}$  consists of 5 linear layers with ReLU activations and a hidden size of 1024. The final layer maps the activations to the dimension of ViT-B tokens with an extra alpha value i.e.  $768 + 1$ .
In all our experiments, unless otherwise specified, we employ DINOv2 [62] with the ViT-B/14 architecture. We set the number of consecutive frame range  $n$  to 2 and drop half of the tokens before the slot attention step. We train our models on  $2 \times \mathrm{V}100$  GPUs using the Adam [39] optimizer with a batch size of 48. We clip the gradient norms at 1 to stabilize the training. We match mask indices of consecutive frames by applying Hungarian Matching on slot similarity to provide temporal consistency. To prevent immature slots in slot merging, we apply merging with a probability that is logarithmically increasing through epochs.
MOVi-E: We train our model from scratch for a total of 60 epochs, which is equivalent to approximately 300K iterations. We use a maximum learning rate of  $4 \times 10^{-4}$  and an exponential decay schedule with linear warmup steps constituting  $5\%$  of the overall training period. The model is trained using 18 slots and the input frames are adjusted to a size of  $336 \times 336$ , leading to 576 feature tokens for each frame. The slot merge coefficient in  $\psi_{\mathrm{merge}}$  is configured to 0.12.
YTVIS19: Similar to MOVi-E, we train the model from scratch for 180 epochs, corresponding to approximately 300K iterations with a peak learning rate of  $4 \times 10^{-4}$  and decay it with an exponential schedule. Linear warmup steps are introduced for  $5\%$  of the training timeline. The model training involves 8 slots, and the input frames are resized to dimensions of  $336 \times 504$ , resulting in 864 feature tokens for each frame. The slot merge coefficient in  $\psi_{\text{merge}}$  is set to be 0.12.
DAVIS17: Due to the small size of DAVIS17, we fine-tune the model pretrained on the YTVIS19 dataset explained above. We finetune on DAVIS17 for 300 epochs, corresponding to approximately 40K iterations with a reduced learning rate of  $1 \times 10^{-4}$ . We use the same learning rate scheduling strategy as in YTVIS19. During the fine-tuning process, we achieve the best result without slot merging, likely due to the fewer number of objects, typically one object at the center, on DAVIS17 compared to YTVIS19.
C Invariant Slot Attention
In this section, we provide the details of invariant slot attention (ISA), initially proposed by Biza et al. [3]. We use ISA in our Spatial Binding module  $\psi_{\mathrm{s - bind}}$  with shared initialization as explained in the main paper. Given the shared initialization  $\mathcal{Z}_{\tau} = \left\{\left(\mathbf{z}^{j},\mathbf{S}_{s}^{j},\mathbf{S}_{p}^{j},\mathbf{G}_{\mathrm{abs},\tau}\right)\right\}_{j = 1}^{K}$ , our goal is to update the  $K$  slots:  $\{\mathbf{z}^j\}_{j = 1}^K$ . For clarification, in the following, we focus on the computation of single-step slot attention for time step  $\tau$ :
$$
\mathbf {A} ^ {j} := \operatorname {s o f t m a x} _ {K} \left(\mathbf {M} ^ {j}\right) \in \mathbb {R} ^ {N ^ {\prime}}, \quad \mathbf {M} ^ {j} := \frac {1}{\sqrt {D _ {\text {s l o t}}}} p \left(k \left(\mathbf {f} _ {\tau}\right) + g \left(\mathbf {G} _ {\text {r e l}, \tau} ^ {j}\right)\right) q \left(\mathbf {z} ^ {j}\right) ^ {T} \in \mathbb {R} ^ {N ^ {\prime}} \tag {13}
$$
where  $p, g, k$ , and  $q$  are linear projections while the relative grid of each slot is defined as:
$$
\mathbf {G} _ {\text {r e l}, \tau} ^ {j} := \frac {\mathbf {G} _ {\text {a b s} , \tau} - \mathbf {S} _ {p} ^ {j}}{\mathbf {S} _ {s} ^ {j}} \in \mathbb {R} ^ {N ^ {\prime} \times 2} \tag {14}
$$
The slot attention matrix  $\mathbf{A}$  from (13) is used to compute the scale  $\mathbf{S}_s$  and the positions  $\mathbf{S}_p$  of slots following Biza et al. [3]:
$$
\mathbf {S} _ {s} ^ {j} := \sqrt {\frac {\operatorname {s u m} \left(\mathbf {A} \odot \left(\mathbf {G} _ {\mathrm {a b s} , \tau} - \mathbf {S} _ {p} ^ {j}\right) ^ {2}\right)}{\operatorname {s u m} \left(\mathbf {A} ^ {j}\right)}} \in \mathbb {R} ^ {2}, \quad \mathbf {S} _ {p} ^ {j} := \frac {\operatorname {s u m} \left(\mathbf {A} ^ {j} \odot \mathbf {G} _ {\mathrm {a b s} , \tau}\right)}{\operatorname {s u m} \left(\mathbf {A} ^ {j}\right)} \in \mathbb {R} ^ {2} \tag {15}
$$
After this step, following the original slot attention, input features are aggregated to slots using the weighted mean with another linear projection  $v$ :
$$
\mathbf {U} := \mathbf {W} ^ {T} p \left(v \left(\mathbf {f} _ {\tau}\right) + g \left(\mathbf {G} _ {\text {r e l}, \tau} ^ {j}\right)\right) \in \mathbb {R} ^ {K \times D _ {\text {s l o t}}}, \quad \mathbf {W} ^ {j} := \frac {\mathbf {A} ^ {j}}{\operatorname {s u m} \left(\mathbf {A} ^ {j}\right)} \in \mathbb {R} ^ {N ^ {\prime}} \tag {16}
$$
Then,  $\mathbf{U}$  from (16) is used to update slots  $\{\mathbf{z}^j\}_{j=1}^K$  with GRU followed by an additional MLP as residual connection as shown in (17). This operation is repeated 3 times.
$$
\mathbf {z} := \mathbf {z} + \operatorname {M L P} \left(\left(\operatorname {n o r m} (\mathbf {z})\right), \quad \mathbf {z} := \operatorname {G R U} \left(\mathbf {z}, \mathbf {U}\right) \in \mathbb {R} ^ {K \times D _ {\text {s l o t}}}\right) \tag {17}
$$
D Additional Ablation Studies
In this section, we provide additional experiments to show the effect of resolution on the segmentation quality. We also report the results by varying the evaluation split on the YTVIS to confirm the generalization capability of our model.
Effect of Resolution: We conducted experiments to investigate the effect of the input frame resolution, i.e. the number of input tokens, in Fig. 8. Specifically, we experimented with resolutions of  $168 \times 252$ ,  $224 \times 336$ , and our default resolution  $336 \times 506$ , corresponding to 216, 384, and 864 input feature tokens, respectively. These experiments show that the input resolution is crucial for segmentation performance.
Varying The Evaluation Split: As stated before, we cannot use the validation split of YTVIS19 due to manual annotations that are not publicly available. For evaluation, we only choose a subset of 300 videos. Here, we perform an experiment to examine the effect of varying the set of evaluation videos on performance.