SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Görkay Aydemir<sup>1</sup> Weidi Xie<sup>3,4</sup> Fatma Güney<sup>1,2</sup>
$^{1}$  Department of Computer Engineering, Koç University  $^{2}$  KUIS AI Center
$^{3}$  CMIC, Shanghai Jiao Tong University  $^{4}$  Shanghai AI Laboratory
{gaydemir23, fguney}@ku.edu.tr weidi@sjtu.edu.cn
Unsupervised multi-object segmentation has shown impressive results on images by utilizing powerful semantics learned from self-supervised pretraining. An additional modality such as depth or motion is often used to facilitate the segmentation in video sequences. However, the performance improvements observed in synthetic sequences, which rely on the robustness of an additional cue, do not translate to more challenging real-world scenarios. In this paper, we propose the first fully unsupervised method for segmenting multiple objects in real-world sequences. Our object-centric learning framework spatially binds objects to slots on each frame and then relates these slots across frames. From these temporally-aware slots, the training objective is to reconstruct the middle frame in a high-level semantic feature space. We propose a masking strategy by dropping a significant portion of tokens in the feature space for efficiency and regularization. Additionally, we address over-clustering by merging slots based on similarity. Our method can successfully segment multiple instances of complex and high-variety classes in YouTube videos.<sup>1</sup>
Given a video sequence of a complex scene, our goal is to train a vision system that can discover, track, and segment objects, in a way that abstracts the visual information from millions of pixels into semantic components, i.e., objects. This is commonly referred to as object-centric visual representation learning in the literature. By learning such abstractions of the visual scene, the resulting object-centric representation acts as fundamental building blocks that can be processed independently and recombined, thus improving the model's generalization and supporting high-level cognitive vision tasks such as reasoning, control, etc. [13, 29].
The field of object-centric representation learning in computer vision has made significant progress over the years, starting from synthetic images [34, 37], and has since shifted towards in-the-wild image [20, 49] and real-world videos [65, 85, 87, 46, 60]. In general, existing approaches typically follow an autoencoder training paradigm [52, 26], i.e., reconstructing the input signals with certain bottlenecks, with the hope to group the regional pixels into semantically meaningful objects based on the priors of architecture or data. In particular, for images, low-level features like color, and semantic features from pretrained deep networks, are often used to indicate the assignment of pixels to objects, while for videos, additional modalities/signals are normally incorporated, such as optical flow [40], depth map [17], with segmentation masks directly available from the discontinuities. Despite being promising, using additional signals in videos naturally incurs computation overhead and error accumulation, for example, optical flow may struggle with static or deformable objects and large displacements between frames, while depth may not be readily available in generic videos and its estimation can suffer in low-light or low-contrast environments.
1Project page: https://kuis-ai.github.io/solv
Figure 1: We introduce SOLV, an object-centric framework for instance segmentation in real-world videos. This figure depicts instance segmentation results of first, middle, and last frames of videos on Youtube-VIS 2019 [87]. Without any supervision both in training and inference, we manage to segment object instances accurately.
In this paper, we introduce SOLV, a self-supervised model capable of discovering multiple objects in real-world video sequences without using additional modalities [2, 36, 40, 17] or any kind of weak supervision such as first frame initialization [40, 17]. To achieve this, we adopt axial spatial-temporal slot attentions, that first groups spatial regions within frame, then followed by enriching the slot representations using additional cues from interactions with neighboring frames. We employ masked autoencoder (MAE)-type training, with the objective of reconstructing dense visual features from masked inputs. This approach has two benefits: first, it acts as an information bottleneck, forcing the model to learn the high-level semantic structures, given only partial observations; second, it alleviates memory constraints, which helps computational efficiency. Additionally, due to the complexity of visual scenes, a fixed number of slots often leads to an over-segmentation issue. We address this issue by merging similar slots with a simple clustering algorithm. Experimentally, our method significantly advances the state-of-the-art without using information from additional modalities on a commonly used synthetic video dataset, MOVi-E [25] and a subset of challenging Youtube-VIS 2019 [87] with in-the-wild videos (Fig. 1).
To summarize, we make the following contributions: (i) We propose a self-supervised multi-object segmentation model on real-world videos, that uses axial spatial-temporal slots attention, effectively grouping visual regions with similar property, without relying on any additional signal; (ii) We present an object-centric learning approach based on masked feature reconstruction, and slot merging; (iii) Our model achieves state-of-the-art results on the MOVi-E and Youtube-VIS 2019 datasets, and competitive performance on DAVIS2017.
Object-centric Learning: The field of unsupervised learning of object-centric representations from images and videos has gained considerable attention in recent years. Several approaches have been proposed to address this problem with contrastive learning [31, 41, 84] or more recently with reconstruction objectives. An effective reconstruction-based approach first divides the input into a set of region identifier variables in the latent space, i.e. slots, that bind to distinctive parts corresponding to objects. Slot attention has been applied to both images [26, 5, 10, 12, 18, 50, 89, 19, 27, 15, 52, 69] and videos [35, 71, 28, 40, 33, 75, 90, 43]. However, these methods are typically evaluated on synthetic data and struggle to generalize to real-world scenarios due to increasing complexity. To address this challenge, previous works explore additional information based on the 3D structure [8, 58, 32] or reconstruct different modalities such as flow [40] or depth [17]. Despite these efforts, accurately identifying objects in complex visual scenes without explicit guidance remains an open challenge. The existing work relies on guided initialization from a motion segmentation mask [1, 2] or initial object locations [40, 17]. To overcome this limitation, DINOSAUR [67] performs reconstruction in the feature space by leveraging the inductive biases learned by recent self-supervised models [7]. We also follow this strategy which has proven highly effective in learning object-centric representations in real-world data without any guided initialization or explicit supervision.
Object Localization from DINO Features: The capabilities of Vision Transformers (ViT) [14] have been comprehensively investigated, leading to remarkable findings when combined with self-supervised features from DINO [7]. By grouping these features with a traditional graph partitioning method [57, 68, 81], impressive results can be achieved compared to earlier approaches [76, 77].
Recent work, CutLER [80], extends this approach to segment multiple objects with series of normalized cuts. The performance of these methods without any additional training shows the power of self-supervised DINO features for segmentation and motivates us to build on it in this work.
Video Object Segmentation: Video object segmentation (VOS) [86, 4, 22, 23, 42, 38, 44, 45, 56, 59, 61, 72, 78, 79, 88] aims to identify the most salient object in a video without relying on annotations during evaluation in unsupervised setting [73, 21, 47, 53] and only annotation of the first frame in semi-supervised setting [6]. Even if the inference is unsupervised, ground-truth annotations can be used during training in VOS [11, 16, 63, 54, 6]. Relying on labelled data during training might introduce a bias towards the labelled set of classes that is available during training. In this paper, we follow a completely unsupervised approach without using any annotations during training or testing.
Motion information is commonly used in unsupervised VOS to match object regions across time [36, 9, 51, 66]. Motion cues particularly come in handy when separating multiple instances of objects in object-centric approaches. Motion grouping [86] learns an object-centric representation to segment moving objects by grouping patterns in flow. Recent work resorts to sequential models while incorporating additional information [70, 40, 17, 1]. In this work, we learn temporal slot representations from multiple frames but we do not use any explicit motion information. This way, we can avoid the degradation in performance when flow cannot be estimated reliably.
In this section, we start by introducing the considered problem scenario, then we detail the proposed object-centric architecture, that is trained with self-supervised learning.
3.1 Problem Scenario
Given an RGB video clip as input,  $i.e., \mathcal{V}_t = \{\mathbf{v}_{t-n}, \ldots, \mathbf{v}_t, \ldots, \mathbf{v}_{t+n}\} \in \mathbb{R}^{(2n+1) \times H \times W \times 3}$ , our goal is to train an object-centric model that processes the clip, and outputs all object instances in the form of segmentation masks, i.e., discover and track the instances in the video, via self-supervised learning, we can formulate the problem as:
$$
\mathbf {m} _ {t} = \Phi (\mathcal {V} _ {t}; \Theta) = \Phi_ {\text {v i s - d e c}} \circ \Phi_ {\text {s t - b i n d}} \circ \Phi_ {\text {v i s - e n c}} (\mathcal {V} _ {t}) \tag {1}
$$
where  $\mathbf{m}_t \in \mathbb{R}^{K_t \times H \times W}$  refers to the output segmentation mask for the middle frame with  $K_t$  discovered objects. After segmenting each frame, we perform Hungarian matching to track the objects across frames in the video.  $\Phi(\cdot; \Theta)$  refers to the proposed segmentation model that will be detailed in the following section. Specifically, it consists of three core components (see Fig. 2), namely, visual encoder that extracts frame-wise visual features (Section 3.2.1), spatial-temporal axial binding that first groups pixels into slots within frames, then followed by joining the slots across temporal frames (Section 3.2.2), and visual decoder that decodes the spatial-temporal slots to reconstruct the dense visual features, with the segmentation masks of objects as by-products (Section 3.2.3).
Generally speaking, our proposed architecture is a variant of Transformer, that can be trained with simple masked autoencoder, i.e., reconstructing the full signals given partial input observation. However, unlike standard MAE [30] that recovers the image in pixel space, we adopt an information bottleneck design, that first assigns spatial-temporal features into slots, then reconstructs the dense visual features from latent slots. As a result, each slot is attached to one semantically meaningful object, and the segmentation masks can be obtained as by-products from reconstruction, i.e., without relying on manual annotations.
3.2.1 Visual Encoder
Given the RGB video clip, we divide each image into regular non-overlapping patches. Following the notation introduced in Section 3.1, i.e.,  $\mathcal{V}_t = \{\mathbf{v}_{t - n},\dots ,\mathbf{v}_t,\dots \mathbf{v}_{t + n}\} \in \mathbb{R}^{(2n + 1)\times N\times (3P^2)}$ , where  $N = HW / P^2$  is the number of tokens extracted from each frame with patches of size  $P$ . The visual encoder consists of token drop and feature extraction, as detailed below.
Figure 2: Overview. In this study, we introduce SOLV, an autoencoder-based model designed for object-centric learning in videos. Our model consists of three components: (i) Visual Encoder for extracting features for each frame using  $\phi_{\mathrm{DINO}}$ ; (ii) Spatial-temporal Binding module for generating temporally-aware object-centric representations by binding them spatially and temporally using  $\psi_{\mathrm{s - bind}}$  and  $\psi_{\mathrm{t - bind}}$ , respectively; (iii) Visual Decoder for estimating segmentation masks and feature reconstructions for the central frame with  $\psi_{\mathrm{dec}}$ , after merging similar slots using  $\psi_{\mathrm{merge}}$
Token Drop: As input to the encoder, we only sample a subset of patches. Our sampling strategy is straightforward: we randomly drop the input patches with some ratio for each frame,
$$
\mathcal {V} _ {t} ^ {\prime} = \left\{\mathbf {v} _ {t - n} ^ {\prime}, \dots , \mathbf {v} _ {t + n} ^ {\prime} \right\} = \left\{\operatorname {d r o p} \left(\mathbf {v} _ {t - n}\right), \dots , \operatorname {d r o p} \left(\mathbf {v} _ {t + n}\right) \right\} \in \mathbb {R} ^ {(2 n + 1) \times N ^ {\prime} \times (3 P ^ {2})}, \quad N ^ {\prime} <   N \tag {2}
$$
where  $N^{\prime}$  denotes the number of tokens after random sampling.
Feature Extraction: We use a frozen Vision Transformer (ViT) [14] with parameters initialized from DINOv2 [62], a self-supervised model that has been pretrained on a large number of images:
$$
\mathcal {F} = \left\{\mathbf {f} _ {t - n}, \dots , \mathbf {f} _ {t + n} \right\} = \left\{\phi_ {\mathrm {D I N O}} \left(\mathbf {v} _ {t - n} ^ {\prime}\right), \dots , \phi_ {\mathrm {D I N O}} \left(\mathbf {v} _ {t + n} ^ {\prime}\right) \right\} \in \mathbb {R} ^ {(2 n + 1) \times N ^ {\prime} \times D} \tag {3}
$$
where  $D$  refers to the dimension of output features from the last block of DINOv2, right before the final layer normalization.
Discussion: Our design of token drop serves two purposes: Firstly, masked autoencoding has been widely used in natural language processing and computer vision, acting as a proxy task for self-supervised learning, our token drop can effectively encourage the model to acquire high-quality visual representation; Secondly, for video processing, the extra temporal axis brings a few orders of magnitude of more data, processing sparsely sampled visual tokens can substantially reduce memory budget, enabling computation-efficient learning, as will be validated in our experiments.
3.2.2 Spatial-temporal Binding
After extracting visual features for each frame, we first spatially group the image regions into slots, with each specifying a semantic object, i.e., discover objects within single image; then, we establish the temporal bindings between slots with a Transformer, i.e., associate objects within video clips.
$$
\Phi_ {\mathrm {s t} - \text {b i n d}} (\mathcal {F}) = \psi_ {\mathrm {t} - \text {b i n d}} \left(\psi_ {\mathrm {s} - \text {b i n d}} \left(\mathbf {f} _ {t - n}\right), \dots , \psi_ {\mathrm {s} - \text {b i n d}} \left(\mathbf {f} _ {t + n}\right)\right) \in \mathbb {R} ^ {K \times D _ {\text {s l o t}}} \tag {4}
$$
Spatial Binding  $(\psi_{\mathrm{s - bind}})$ : The process of spatial binding is applied to each frame independently. We adopt the invariant slot attention proposed by Biza et al. [3], with one difference, that is, we use a shared initialization  $\mathcal{Z}_{\tau}$  at each time step  $\tau \in \{t - n,\dots ,t + n\}$ . Specifically, given features after token drop at a time step  $\tau$  as input, we learn a set of initialization vectors to translate and scale the input position encoding for each slot separately with  $K$  slot vectors  $\mathbf{z}^j\in \mathbb{R}^{D_{\mathrm{slot}}}$ ,  $K$  scale vectors
$\mathbf{S}_s^j \in \mathbb{R}^2$ ,  $K$  position vectors  $\mathbf{S}_p^j \in \mathbb{R}^2$ , and one absolute positional embedding grid  $\mathbf{G}_{\mathrm{abs}} \in \mathbb{R}^{N \times 2}$ . We mask out the patches corresponding to the tokens dropped in feature encoding (Section 3.2.1) and obtain absolute positional embedding for each frame  $\tau$  as  $\mathbf{G}_{\mathrm{abs},\tau} = \mathrm{drop}\left(\mathbf{G}_{\mathrm{abs}}\right) \in \mathbb{R}^{N' \times 2}$ . Please refer to the original paper [3] or the Supplementary Material for details of invariant slot attention on a single frame. This results in the following set of learnable vectors for each frame:
$$
\mathcal {Z} _ {\tau} = \left\{\left(\mathbf {z} ^ {j}, \mathbf {S} _ {s} ^ {j}, \mathbf {S} _ {p} ^ {j}, \mathbf {G} _ {\mathrm {a b s}, \tau}\right) \right\} _ {j = 1} ^ {K} \tag {5}
$$
Note that these learnable parameters are shared for all frames and updated with respect to the dense visual features of the corresponding frame. In other words, slots of different frames start from the same representation but differ after binding operation due to interactions with frame features. The output of  $\psi_{\mathrm{s - bind}}$  is the updated slots corresponding to independent object representations at frame  $\tau$ :
$$
\left\{\mathbf {z} _ {\tau} ^ {j} \right\} _ {j = 1} ^ {K} = \psi_ {\mathrm {s - b i n d}} \left(\mathbf {f} _ {\tau}\right) \in \mathbb {R} ^ {K \times D _ {\mathrm {s l o t}}}, \quad \tau \in \{t - n, \dots , t + n \} \tag {6}
$$
In essence, given that consecutive frames typically share a similar visual context, the use of learned slots inherently promotes temporal binding, whereby slots with the same index can potentially bind to the same object region across frames. Our experiments demonstrate that incorporating invariant slot attention aids in learning slot representations that are based on the instance itself, rather than variable properties like size and location.
Temporal Binding  $(\psi_{\mathbf{t}\cdot \mathbf{bind}})$ : Up until this point, the model is only capable of discovering objects by leveraging information from individual frames. This section focuses on how to incorporate temporal context to enhance the slot representation. Specifically, given the output slots from the spatial binding module, denoted as  $\left\{\{\mathbf{z}_{t-n}^j\}_{j=1}^K,\dots,\{\mathbf{z}_{t+n}^j\}_{j=1}^K\right\} \in \mathbb{R}^{(2n+1) \times K \times D_{\mathrm{slot}}}$ , we apply a transformer encoder to the output slots with same index across different frames. In other words, the self-attention mechanism only computes a  $(2n+1) \times (2n+1)$  affinity matrix across  $2n+1$  time steps. This approach helps to concentrate on the specific region of the image through time. Intuitively, each slot learns about its future and past in the temporal window by attending to each other, which helps the model to create a more robust representation by considering representations of the same object at different times.
To distinguish between time stamps, we incorporate learnable temporal positional encoding onto the slots, i.e., all slots from a single frame receive the same encoding. As the result of temporal transformer, we obtain the updated slots  $\mathbf{c}$  at the target time step  $t$ :
$$
\mathbf {c} = \Phi_ {\mathrm {s t - b i n d}} (\mathcal {F}) \in \mathbb {R} ^ {K \times D _ {\mathrm {s l o t}}} \tag {7}
$$
3.2.3 Visual Decoder
The spatial-temporal binding process yields a set of slot vectors  $\mathbf{c} \in \mathbb{R}^{K \times D_{\mathrm{slot}}}$  at target frame  $t$  that are aware of temporal context. However, in natural videos, the number of objects within a frame can vary significantly, leading to over-clustering when a fixed number of slots is used, i.e., as we can always initialize slots in an over-complete manner.
To overcome this challenge, we propose a simple solution for slot merging through Agglomerative Clustering. Additionally, we outline the procedure for slot decoding to reconstruct video features, which resembles a masked auto-encoder in feature space.
$$
\Phi_ {\text {v i s - d e c}} (\mathbf {c}) = \psi_ {\text {d e c}} \circ \psi_ {\text {m e r g e}} (\mathbf {c}) \tag {8}
$$
Slot Merging  $(\psi_{\mathrm{merge}})$ : As expected, the problem of object segmentation is often a poorly defined problem statement in the self-supervised scenario, as there can be multiple interpretations of visual regions. For instance, in an image of a person, it is reasonable to group all person pixels together or group the person's face, arms, body, and legs separately. However, it is empirically desirable for pixel embeddings from the same object to be closer than those of different objects. To address this challenge, we propose to merge slots using Agglomerative Clustering (AC), which does not require a predefined number of clusters. As shown in Fig. 3, we first compute the affinity matrix between all slots based on cosine similarity, then use this affinity matrix to cluster slots into groups, and compute the mean slot for each resulting cluster:
$$
\mathbf {c} ^ {\prime} = \psi_ {\text {m e r g e}} (\mathbf {c}) \in \mathbb {R} ^ {K _ {t} \times D _ {\text {s l o t}}}, \quad K _ {t} \leq K \tag {9}
$$
Figure 3: Assignments of slots to the input frame (left) and the pairwise similarity matrix of slots (right) where lighter colors indicate higher similarity. Slots corresponding to the parts of the same object (0 and 1) are highly similar to each other while being different from background slots (2, 3, 4, and 5).
Figure 4: Results without (left) and with (right) slot merging.
By merging semantically similar slots that correspond to the same object, we can dynamically determine the optimal number of slots. Our slot merging strategy, as shown in Fig. 4, effectively combines regions of the same object using solely learned visual features. Note that slot merging is not a post-processing step, but rather an integral part of our training process.
Decoder  $(\psi_{\mathrm{dec}})$ : We decode the merged slots  $\mathbf{c}'$  with decoder  $\psi_{\mathrm{dec}}$  to obtain the corresponding segmentation mask  $\mathbf{m}$  and the full reconstruction  $\mathbf{y}$ :
$$
\mathbf {y}, \mathbf {m} = \psi_ {\mathrm {d e c}} \left(\mathbf {c} ^ {\prime}\right), \quad \mathbf {y} \in \mathbb {R} ^ {N \times D}, \mathbf {m} \in \mathbb {R} ^ {K _ {t} \times N} \tag {10}
$$
We reshape and upsample the masks  $\mathbf{m}$  to the original input size to obtain the final segmentation. Similar to the MLP decoder design introduced in DINOSAUR [67], we use a spatial broadcast decoder [82] to reconstruct the full feature map, i.e.,  $\hat{\mathbf{y}}^j\in \mathbb{R}^{N\times D}$  of each slot  $j$  with their alpha weights  $\alpha^j\in \mathbb{R}^N$ . These alpha weights are converted to segmentation masks with a softmax. As per standard procedure, we add learned positional encodings to identify spatial locations during decoding. Each slot  $\mathbf{c}^{r^j}$ , broadcasted to the shape of the input feature map, is decoded by a series of linear layers,  $\psi_{\mathrm{mapper}}$ , with shared weights across all slots. The reconstruction is ultimately achieved through a weighted sum of the decoded slots:
$$
\mathbf {y} = \sum_ {j = 1} ^ {K _ {t}} \hat {\mathbf {y}} ^ {j} \odot \mathbf {m} ^ {j}, \quad \mathbf {m} ^ {j} = \operatorname {s o f t m a x} \left(\boldsymbol {\alpha} ^ {j}\right), \quad \boldsymbol {\alpha} ^ {j}, \hat {\mathbf {y}} ^ {j} = \psi_ {\text {m a p p e r}} \left(\operatorname {b r o a d c a s t} \left(\mathbf {c} ^ {\prime j}\right)\right) \tag {11}
$$
At training time, we optimise the model by minimising the difference between the feature map computed from original unmasked tokens of the frame at time  $t$  and the reconstructed tokens  $\mathbf{y}$ :
$$
\mathcal {L} = \left\| \phi_ {\mathrm {D I N O}} \left(\mathbf {v} _ {t}\right) - \mathbf {y} \right\| ^ {2} \tag {12}
$$