SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrii Zadaianchuk $^{1,2*}$
Maximilian Seitzer $^{1*}$
Georg Martius<sup>1</sup>
<sup>1</sup> Max Planck Institute for Intelligent Systems, Tübingen, Germany
$^{2}$  Department of Computer Science, ETH Zurich andrii.zadaianchuk@tuebingen.mpg.de
Unsupervised video-based object-centric learning is a promising avenue to learn structured representations from large, unlabeled video collections, but previous approaches have only managed to scale to real-world datasets in restricted domains. Recently, it was shown that the reconstruction of pre-trained self-supervised features leads to object-centric representations on unconstrained real-world image datasets. Building on this approach, we propose a novel way to use such pretrained features in the form of a temporal feature similarity loss. This loss encodes semantic and temporal correlations between image patches and is a natural way to introduce a motion bias for object discovery. We demonstrate that this loss leads to state-of-the-art performance on the challenging synthetic MOVi datasets. When used in combination with the feature reconstruction loss, our model is the first object-centric video model that scales to unconstrained video datasets such as YouTube-VIS. https://martius-lab.github.io/videosaur/
Autonomous systems should have the ability to understand the natural world in terms of independent entities. Towards this goal, unsupervised object-centric learning methods [1-3] learn to structure scenes into object representations solely from raw perceptual data. By leveraging large-scale datasets, these methods have the potential to obtain a robust object-based understanding of the natural world. Of particular interest in recent years have been video-based methods [4-7], not least because the temporal information in video presents a useful bias for object discovery [8]. However, these approaches are so far restricted to data of limited complexity, successfully discovering objects from natural videos only on closed-world datasets in restricted domains.
In this paper, we present the method Video Slot Attention Using temporal feature similarity, VideoSAUR, that scales video object-centric learning to unconstrained real-world datasets covering diverse domains. To achieve this, we build upon recent advances in image-based object-centric learning. In particular, Seitzer et al. [9] showed that reconstructing pre-trained features obtained from self-supervised methods like DINO [10] or MAE [11] leads to state-of-the-art object discovery on complex real-world images. We demonstrate that combining this feature reconstruction objective with a video object-centric model [5] also leads to promising results on real-world YouTube videos.
We then identify a weakness in the training objective of current unsupervised video object-centric architectures [4, 7]: the prevalent reconstruction loss does not exploit the temporal correlations existing in video data for object grouping. To address this issue, we propose a novel self-supervised loss based on feature similarities that explicitly incorporates temporal information (see Fig. 1). The loss works by predicting distributions over similarities between features of the current and future
Figure 1: We propose a self-supervised temporal similarity loss for training object-centric video models. For each patch at time  $t$ , the model has to predict a distribution  $\hat{P}_{t,t+k}$  indicating where all semantically-similar patches have moved to  $k$  steps into the future. The target distribution  $P_{t,t+k}$  is computed with a softmax on the affinity matrix  $A_{t,t+k}$  containing the cosine distance between all patch features  $h_t, h_{t+k}$ . The loss incentivizes the model to group areas with consistent motion and semantics into slots.
frames. These distributions encode information about the motion of individual image patches. To efficiently predict those motions through the slot bottleneck, the model is incentivized to group patches with similar motion into the same slot, leading to better object groupings as patches belonging to an object tend to move consistently. In our experiments, we find that such a temporal similarity loss leads to state-of-the-art performance on challenging synthetic video datasets [12], and significantly boosts performance on real-world videos when used in conjunction with the feature reconstruction loss.
In video processing, model efficiency is of particular importance. Thus, we design an efficient object-centric video architecture by adapting the SlotMixer decoder [13] recently proposed for 3D object modeling for video decoding. Compared to previous decoder designs [3], the SlotMixer decoder scales gracefully with the number of slots, but has a weaker inductive bias for object grouping. We show that this weaker bias manifests in optimization difficulties in conjunction with conventional reconstruction losses, but trains robustly with our proposed temporal similarity loss. We attribute this to the self-supervised nature of the similarity loss: compared to reconstruction, it requires predicting information that is not directly contained in the input; the harder task seems to compensate for the weaker bias of the SlotMixer decoder.
To summarize, our contributions are as follows: (1) we propose a novel self-supervised loss for object-centric learning based on temporal feature similarities, (2) we combine this loss with an efficient video architecture based on the SlotMixer decoder where it synergistically reduces optimization difficulties, (3) we show that our model improves the state-of-the-art on the synthetic MOVi datasets by a large margin, and (4) we demonstrate that our model is able to learn video object-centric representations on the YouTube-VIS dataset [14], while staying fully unsupervised. This paper takes a large step towards unconstrained real-world object-centric learning on videos.
Video Object-Centric Learning There is a rich body of work on discovering objects from video, with two broad categories of approaches: tracking bounding boxes [4, 15-17] or segmentation masks [2, 5-7, 18-25]. Architecturally, most recent image-based models for object-centric learning [3, 9, 26] are based on an auto-encoder framework with a latent slot attention grouping module [3] that extracts a set of slot representations. For processing video data, a common approach [5-7, 21, 24] is then to connect slots recurrently over input frames; the slots from the previous frame act as initialization for extracting the slots of the current frame. We also make use of this basic framework.
Scaling Object-Centric Learning Most recent work has attempted to increase the complexity of datasets where objects can successfully be discovered, such as the synthetic ClevrTex [27] and MOVi
datasets [12]. On natural data, object discovery has so far been limited to restricted domains with a limited variety of objects, such as YouTube-Aquarium and -Cars [7], or autonomous driving datasets like WaymoOpen or KITTI [28]. On more open-ended datasets, previous approaches have struggled [29].
To achieve scaling, some works attempt to improve the grouping module, for example by introducing equivariances to slot pose transformations [30], smoothing attention maps [31], formulating grouping as graph cuts [32] or a stick-breaking process [33], or by overcoming optimization difficulties by introducing implicit differentiation [34, 35]. In contrast, we do not change the grouping module, but use the vanilla slot attention cell [3].
Another prominent approach is to introduce better training signals than the default choice of image reconstruction. For example, one line of work instead models the image as a distribution of discrete codes conditional on the slots, either autoregressively by a Transformer decoder [7, 26], or via diffusion [36, 37]. While this strategy shows promising results on synthetic data, it so far has failed to scale to unconstrained real-world data [9].
An alternative is to step away from fully-unsupervised representation learning by introducing weak supervision. For instance, SAVi [5] predicts optical flow, and SAVi++ [6] additionally predicts depth maps as a signal for object grouping. Other works add an auxiliary loss that regularizes slot attention's masks towards the masks of moving objects [8, 38]. Our model also has a loss that focuses on motion information, but uses an unsupervised formulation. OSRT [13] shows promising results on synthetic 3D datasets, but is restricted by the availability of posed multi-camera imagery. While all those approaches improve on the level of data complexity, it has not been demonstrated that they can scale to unconstrained real-world data.
The most promising avenue so far in terms of scaling to the real-world is to reconstruct features from modern self-supervised pre-training methods [10, 11, 39, 40]. Using this approach, DINOSAUR [9] showed that by optimizing in this highly semantic space, it is possible to discover objects on complex real-world image datasets like COCO or PASCAL VOC. In this work, we similarly use such self-supervised features, but for learning on video instead of images. Moreover, we improve upon reconstruction of features by introducing a novel loss based on similarities between features.
Concurrent Work Parallel to this work, two more slot attention-based methods were proposed that learn object-centric representations on real-world videos: SMTC [41] and SOLV [42]. SMTC learns to extracts objects from videos by enforcing semantic and instance consistency over time using a student-teacher approach. SOLV extracts per-frame slots using invariant slot attention [30], applies a temporal consistency module and merges slots using agglomerative clustering; the model is also trained using DINOSAUR-style feature reconstruction, but on masked out intermediate frames.
In this section, we describe the main new components of VideoSAUR — our proposed object-centric video model — and its training: a pre-trained self-supervised ViT encoder extracting frame features (Sec. 3.1), a temporal similarity loss that adds a motion bias to object discovery (Sec. 3.2), and the SlotMixer decoder to achieve efficient video processing (Sec. 3.3). See Fig. 2 for an overview.
3.1 Slot Attention for Videos with Dense Self-Supervised Representations
VideoSAUR is based on the modular video object-centric architecture recently proposed by SAVi [5] and also used by STEVE [7]. Our model has three primary components: (1) a pre-trained self-supervised ViT feature encoder, (2) a recurrent grouping module for temporal slot updates, and (3) the SlotMixer decoder (detailed below in Sec. 3.3).
We start by processing video frames  $\boldsymbol{x}_t$ , with time steps  $t \in \{1, \dots, T\}$ , into patch features  $\boldsymbol{h}_t$ :
$$
\boldsymbol {h} _ {t} = f _ {\phi} (\boldsymbol {x} _ {t}), \quad \boldsymbol {h} _ {t} \in \mathbb {R} ^ {L \times D} \tag {1}
$$
where  $f_{\phi}$  is a self-supervised Vision Transformer encoder (ViT) [43] with pre-trained parameters  $\phi$ , and  $\boldsymbol{x}_t$  is the input at time step  $t$ . The ViT encoder processes the image by splitting it to  $L$  non-overlapping patches of fixed size (e.g.  $16 \times 16$  pixels), adding positional encoding, and transforming them into  $L$  feature vectors  $\boldsymbol{h}_t$  (see App. C.2 for more details on ViTs). Note that the  $i$ 'th feature retains an association to the  $i$ 'th image patch; the features thus can be spatially arranged. Next,
Figure 2: Overview of VideoSAUR. Object slots  $s_t$  are extracted from patch features  $\pmb{h}_t$  of a self-supervised ViT using time-recurrent slot attention, conditional on slots from the previous time step  $t - 1$ . The model is trained by reconstructing the patch features  $\pmb{h}_t$  of the current frame  $x_t$ , and by predicting the similarity distribution over patches of a future frame  $x_{t + k}$  (see also Fig. 1). The predictions  $y_t^{\mathrm{rec}}$  and  $y_t^{\mathrm{sim}}$  are decoded efficiently using SlotMixer decoder.
we transform the features from the encoder with a slot attention module [3] to obtain a latent set  $\mathbf{s}_t = \{\mathbf{s}_t^i\}_{i=1}^K$ ,  $s_t^i \in \mathbb{R}^M$  with  $K$  slot representations:
$$
\boldsymbol {s} _ {t} = \operatorname {S A} _ {\theta} \left(\boldsymbol {h} _ {t}, \boldsymbol {s} _ {t - 1}\right). \tag {2}
$$
Slot attention is recurrently initialized with the slots of the previous time step  $t - 1$ , with initial slots  $s_0$  sampled independently from a Gaussian distribution with learned location and scale. Slot attention works by grouping input features into slots by iterating competitive attention steps; we refer to Locatello et al. [3] for more details. To train the model, we use a SlotMixer decoder  $g_{\psi}$  (see Sec. 3.3) to transform the slots  $s_t$  to outputs  $y_t = g_{\psi}(s_t)$ . Those outputs are used as model predictions for the reconstruction and similarity losses introduced next.
3.2 Self-Supervised Object Discovery by Predicting Temporal Similarities
We now motivate our novel loss function based on predicting temporal feature similarities. Video affords the opportunity to discover objects from motion: pixels that consistently move together should be considered as one object, sometimes called the "common fate" principle [44]. However, the widely used reconstruction objective — whether of pixels [5], discrete codes [7] or features [9] — does not exploit this bias, as to reconstruct the input frame, the changes between frames do not have to be taken into account.
Taking inspiration from prior work using optical flow as a prediction target [5], we design a self-supervised objective that requires predicting patch motion: for each patch, the model needs to predict where all semantically-similar patches have moved to  $k$  steps into the future. By comparing self-supervised features describing the patches, we integrate both semantic and motion information; this is in contrast to optical flow prediction, which only relies on motion. Specifically, we construct an affinity matrix  $A_{t,t+k}$  with the cosine similarities between all patch features from the present frame  $h_t$  and all features from some future frame  $h_{t+k}$ :
$$
\boldsymbol {A} _ {t, t + k} = \frac {\boldsymbol {h} _ {t}}{\| \boldsymbol {h} _ {t} \|} \cdot \left(\frac {\boldsymbol {h} _ {t + k}}{\| \boldsymbol {h} _ {t + k} \|}\right) ^ {\top}, \quad \boldsymbol {A} _ {t, t + k} \in [ - 1, 1 ] ^ {L \times L}. \tag {3}
$$
As self-supervised features are highly semantic, the obtained feature similarities are high for patches that share the same semantic interpretation. Due to the ViT's positional encoding, the similarities also take spatial closeness of patches into account. Figure 3 shows several example affinity matrices.
Because there are ambiguities in our similarity-based derivation of feature movements, we frame the prediction task as modeling a probability distribution over target patches — instead of forcing the prediction of an exact target location, like with optical flow prediction. Thus, we define the
Figure 3: Affinity matrix  $A_{t,t+k}$  and transition probabilities  $P_{t,t+k}$  values between patches (marked by purple and green) of the frame  $x_t$  and patches of the future frame  $x_{t+k}$  in MOVi-C (left) and YT-VIS (right). Red indicates maximum affinity/probability. Also see Fig. B.4 for more examples, and our website for an interactive visualization of temporal feature similarities.
probability that patch  $i$  moves to patch  $j$  by normalizing the rows of the affinity matrix with the softmax, while masking negative similarity values (superscripts refer to the elements of the matrix):
$$
\boldsymbol {P} ^ {i j} = \left\{ \begin{array}{l l} \frac {\exp \left(\boldsymbol {A} ^ {i j} / \tau\right)}{\sum_ {k \in \{j \mid \boldsymbol {A} ^ {i j} \geq 0 \}}} & \text {i f} \boldsymbol {A} ^ {i j} \geq 0, \\ 0 & \text {i f} \boldsymbol {A} ^ {i j} <   0, \end{array} \right. \tag {4}
$$
where  $\tau$  is the softmax temperature. The resulting distribution can be interpreted as the transition probabilities of a random walk along a graph with image patches as nodes [45]. Then, we define the similarity loss as the cross entropy between decoder outputs and transition probabilities:
$$
\mathcal {L} _ {\theta , \psi} ^ {\text {s i m}} = \sum_ {l = 1} ^ {L} \operatorname {C E} \left(\boldsymbol {P} _ {t, t + k} ^ {l}; \boldsymbol {y} _ {t} ^ {l}\right). \tag {5}
$$
Figure 1 illustrates the loss computation for an example pair of input frames.
Why is this Loss Useful for Object Discovery? Predicting which parts of the videos move consistently is most efficient with an object decomposition that captures moving objects. This is similar to previous losses predicting optical flow [5]. But in contrast, our loss (Eq. 5) also yields a useful signal for grouping when parts of the frame are not moving: as feature similarities capture semantic aspects, the task also requires predicting which patches are semantically similar, helping the grouping into objects e.g. by distinguishing fore- and background (see Fig. 3). Optical flow for grouping also has limits when camera motion is introduced; in our experiments, we find that our loss is more robust in such situations. Methods based on optical flow or motion masks can also struggle with inaccurate flow/motion mask labels — unlike our method, which does not require such labels. This is of particular importance for in-the-wild video, where motion estimation is challenging.
Role of Hyperparameters. The loss has two hyperparameters: the time shift into the future  $k$  and the softmax temperature  $\tau$ . The optimal time shift depends on the expected time scales of movements in the modeled videos and should be chosen accordingly. The temperature  $\tau$  controls the concentration of the distribution onto the maximum. Thus, it effectively modulates between two different tasks: accurately estimating the patch motion (low  $\tau$ ), and predicting the similarity of each patch to all other patches (high  $\tau$ ). In particular in scenes with little movement, the latter may be important to maintain a meaningful prediction task. In our experiments, we find that the best performance is obtained with a balance between the two, showing that both modes are important.
Final Loss. While the temporal similarity loss yields state-of-the-art performance on synthetic datasets, as shown below, we found that on real-world data, performance can be further improved by adding the feature reconstruction objective as introduced in Seitzer et al. [9]. We hypothesize this is because the semantic nature of feature reconstruction adds another useful bias for object discovery. Thus, the final loss is given by:
$$
\mathcal {L} _ {\theta , \psi} = \sum_ {t = 1} ^ {T - k} \mathcal {L} _ {\theta , \psi} ^ {\operatorname {s i m}} \left(\boldsymbol {P} _ {t, t + k}, \boldsymbol {y} _ {t} ^ {\operatorname {s i m}}\right) + \alpha \mathcal {L} _ {\theta , \psi} ^ {\operatorname {r e c}} \left(\boldsymbol {h} _ {t}, \boldsymbol {y} _ {t} ^ {\operatorname {r e c}}\right), \tag {6}
$$
Figure 4: Example predictions of VideoSAUR compared to recent video object-centric methods.
where  $\pmb{y}_t = [\pmb{y}_t^{\mathrm{sim}}\in \mathbb{R}^{L\times L},\pmb{y}_t^{\mathrm{rec}}\in \mathbb{R}^{L\times D}]$  is the output of the SlotMixer decoder  $g_{\psi}$  and  $\alpha$  is a weighting factor used to make the scales of the two losses similar (we use a fixed value of  $\alpha = 0.1$  for all experiments on real-world datasets). Like in Seitzer et al. [9], we do not train the ViT encoder  $f_{\phi}$ .
3.3 Efficient Video Object-Centric Learning with the SlotMixer Decoder
In video models, resource efficiency is of particular concern: recurrent frame processing increases the load on compute and memory. The standard mixture-based decoder design [3] decodes each output  $K$ -times, where  $K$  is the number of slots, and thus scales linearly with  $K$  both in compute and memory. This can become prohibitive even for a moderate number of slots. The recently introduced SlotMixer decoder [13] for 3D object-centric learning instead has, for all practical purposes, constant overhead in the number of slots, by only decoding once per output. Thus, we propose to use a SlotMixer decoder  $g_{\psi}$  for predicting the probabilities  $P_{t,t + k}$  from the slots  $s_t$ . To adapt the decoder from 3D to 2D outputs, we change the conditioning on 3D query rays to  $L$  learned positional embeddings, corresponding to  $L$  patch outputs  $y_{t}^{l}$ . See App. C.1 for more details on the SlotMixer module.
As a consequence of the increased efficiency of SlotMixer, there also is increased flexibility of how slots can be combined to form the outputs. Because of this, this decoder has a weaker inductive bias towards object-based groupings compared to the standard mixture-based decoder. With the standard reconstruction loss, we observed that this manifests in training runs in which no object groupings are discovered. But in combination with our temporal similarity loss, these instabilities disappear (see App. B.4). We attribute this to the self-supervised nature of the similarity loss $^2$ ; having to predict information that is not directly contained in the input increases the difficulty of the task, reducing the viability of non-object based groupings.
We have conducted a number of experiments to answer the following questions: (1) Can object-centric representations be learned from a large number of diverse real-world videos? (2) How does VideoSAUR perform in comparison to other methods on well-established realistic synthetic datasets? (3) What are the effects of our proposed temporal feature similarity loss and its parameters? (4) Can we transfer the learned object-grouping to unseen datasets? (5) How efficient is the SlotMixer decoder in contrast to the mixture-based decoder?
Datasets To investigate the characteristics of our proposed method, we utilize three synthetic datasets and three real-world datasets. For synthetic datasets, we selected the MOVi-C, MOVi-D
$^{2}$ Novel-view synthesis, the original task for which SlotMixer was proposed, is similarly a self-supervised prediction task. This may have contributed to the success of SlotMixer in that setting.
Table 1: Comparison with state-of-the-art methods on the MOVi-C, MOVi-E, and YT-VIS datasets. We report foreground adjusted rand index (FG-ARI) and mean best overlap (mBO) over 5 random seeds. Both metrics are computed for the whole video (24 frames for MOVi, 6 frames for YT-VIS).
MOVi-C	MOVi-E	YT-VIS
FG-ARI	mBO	FG-ARI	mBO	FG-ARI	mBO
Block Pattern	24.2	11.1	36.0	16.5	24	14.9
SAVi [5]	22.2 ± 2.1	13.6 ± 1.6	42.8 ± 0.9	16.0 ± 0.3	11.1 ± 5.6	12.7 ± 2.3
STEVE [7]	36.1 ± 2.3	26.5 ± 1.1	50.6 ± 1.7	26.6 ± 0.9	20.0 ± 1.5	20.9 ± 0.5
VideoSAUR	64.8 ± 1.2	38.9 ± 0.6	73.9 ± 1.1	35.6 ± 0.5	39.5 ± 0.6	29.1 ± 0.4
and MOVi-E datasets [12] that consist of numerous moving objects on complex backgrounds. Additionally, we evaluate the performance of our method on the challenging YouTube Video Instance Segmentation (YT-VIS) 2021 dataset [14] as an unconstrained real-world dataset. Furthermore, we examine how well our model performs when transferred from YT-VIS 2021 to YT-VIS 2019 [46] and DAVIS [47]. Finally, we use the COCO dataset [48] to study our proposed similarity loss function with image-based object-centric learning.