SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Beyond the initial examination of how our VideoSAUR performs on the relatively brief 6-frame video segments from YouTube-VIS 2021, we extend our evaluation to also assess its effectiveness on more substantial, longer video segments. In Table B.1, we show the performance for 12-frame and full YT-VIS video segments (see Fig. B.1 for the distribution of video lengths). Although the performance of VideoSAUR on extended video segments predictably decreases in terms of FG-ARI, the observed difference between VideoSAUR and the baseline models is significant. This suggests that VideoSAUR maintains its efficacy in tracking the primary objects in videos across longer time intervals.
Additionally, we investigate if VideoSAUR benefits from using DINOv2 features [58] that are obtained by training DINO on the larger dataset and fine-tuning the representation on larger resolution  $(518\times 518)$ . We show that VideoSAUR performance benefits from using such features as a backbone, especially in terms of mask quality  $(+6\mathrm{mBO}$  points). Using those features with the original resolution VideoSAUR reaching  $29.7\mathrm{mBO}$  on full-length YT-VIS 2021 videos. In addition to this quantitative evaluation, we visualize VideoSAUR predictions on long YT-VIS videos (longer than 30-frames) in Figure E.5 and Figure E.6.
<sup>4</sup>We use part of the train split, as validation labels are not released. Exact indexes are available at validation split of the TensorFlow version of the YT-VIS 2019 dataset: https://www.tensorflow.org/datasets/catalog/youtube.vis#youtube_visonly Frames_with_labels_train_split
Table B.1: Performance of VideoSAUR on the YT-VIS 2021 dataset, varying the length of the video segment (mean ± standard dev., 5 seeds for VideoSAUR with DINO features and STEVE. VideoSAUR with DINOv2 features are one seed only due to computational limitations).
6 frames	12 frames	Full Videos
FG-ARI	mBO	FG-ARI	mBO	FG-ARI	mBO
Block Pattern	24.0	14.9	20.3	14.2	15.1	13.1
STEVE [7]	20.0 ± 1.5	20.9 ± 0.5	18.0 ± 1.4	21.5 ± 0.5	15.0 ± 0.7	19.1 ± 0.4
VideoSAUR with DINO B/16	39.5 ± 0.6	29.1 ± 0.4	35.8 ± 0.3	29.4 ± 0.3	28.9 ± 0.4	26.3 ± 0.2
VideoSAUR with DINOv2 B/14 [58]	39.7	35.6	38.7	34.5	31.2	29.7
Figure B.1: Histogram of video lengths on the YT-VIS 2021 validation dataset.
B.2 Optical Flow as Self-Supervised Target
The choice of the self-supervised target plays an important role in creating suitable inductive biases for object discovery and scene decomposition. As such, understanding the properties of the prediction target leading to an effective scene decomposition is crucial. The temporal feature similarity prediction proposed in this work combines two different inductive biases: high-level semantic information and motion information. To elucidate the significance of both types of bias for scene decomposition, we assess the performance of VideoSAUR with prediction targets that consist only of one of those biases.
In Table 2 and Table 3 in the main text, we compare predicting temporal similarities with predicting self-supervised features of the current frame. Such features only contain semantic information, but no information about motion. Depending on the dataset, including temporal information brings a small (YT-VIS) or large benefit (MOVi-C).
Subsequently, we study if motion cues alone (without the semantic information) are enough for successful scene decomposition. In particular, we compare self-supervised temporal similarity targets with (ground truth) optical flow targets (only motion information) on the MOVi-C and MOVi-E datasets. To this end, we train VideoSAUR by predicting optical flow targets, using a spatial broadcast decoder similar to SAVi [5] instead of the mixer decoder. All other components of VideoSAUR stay unchanged. The optical flow map is predicted at full image resolution  $(128 \times 128)$ .
The results are presented in Table B.2. We train VideoSAUR with GT optical flow for the best potential performance from optical flow alone. Yet, even on the MOVi-C datasets favoring optical flow (no camera motion, no static objects), VideoSAUR with temporal feature similarities significantly outperforme optical flow (+10 FG-ARI). This disparity is even greater on the MOVi-E dataset, highlighting VideoSAUR's resilience to camera movements and static objects. Together, these results demonstrate that our temporal feature similarity targets, despite not requiring signals such as optical flow (which would need estimation in real-world scenarios like the YouTube-VIS dataset), excel over mere optical flow targets. We attribute this to the enriched semantic bias inherent to the self-supervised feature similarities.
Table B.2: Comparing VideoSAUR predicting temporal similarities to predicting ground truth optical flow on the MOVi-C and MOVi-E datasets. We report Video FG-ARI of a version of VideoSAUR with optical flow (both backward and forward) as well as the original VideoSAUR with temporal features similarity.
VideoSAUR	MOVi-C	MOVi-E
w/ GT Optical Flow (backward)	48.1	28.9
w/ GT Optical Flow (forward)	48.9	30.1
w/ Temporal Similarities	60.7	73.9
Table B.3: Extended ablation of VideoSAUR components on MOVi-E. We compare VideoSAUR model with different choices of the decoder (Mixer vs MLP used by DINOSAUR) and loss types (temporal similarity loss vs feature reconstruction).
Decoder	Loss Type	FG-ARI	mBO
MLP	Feature Reconstruction	68.6	27.6
MLP	Temp. Feat. Sim. Prediction	74.5	28.8
Mixer	Feature Reconstruction	62.3	20.6
Mixer	Temp. Feat. Sim. Prediction	74.1	34.1
B.3 Comprehensive Ablation Study on the MOVi-E Dataset
In this section, we present the results of a comprehensive ablation study conducted on the MOVi-E dataset. The purpose of this study is to investigate the impact of two key factors: decoder choice (MLP vs. Mixer) and loss function selection. Our goal is to gain a deeper understanding of how these choices affect the performance of our method. The results are summarized in Table B.3.
The similarity loss is beneficial for both decoders, pushing the FG-ARI to approximately 74, as compared to 69 when using the feature reconstruction loss. Notably, the Mixer decoder significantly enhances the clarity of the object mask, with an improvement of  $+5$  mBO. When combined, the similarity loss and Mixer consistently outshine the MLP decoder equipped with the feature reconstruction loss. These insights provide valuable auxiliary information to our main paper ablations (see Table 2 and Table 3), painting a more detailed picture of VideoSAUR's components and their respective performances.
B.4 Stability of Mixer Decoder
As mentioned in Sec. 3.3 of the main text, we found that the mixer decoder sometimes exhibits training instabilities. For instance, Table B.4 shows that there is high variance over random seeds when training purely with feature reconstruction, i.e. some training runs fail to discover an object grouping. These instabilities manifest in slot masks that follow a Voronoi-like decomposition of the image. When adding the temporal similarity loss, the instabilities disappear.
We hypothesize this is because the mixer decoder has increased flexibility in how to model the image with slots (compared to the conventional mixture-based decoder), and thus more failure modes
Table B.4: Mixer decoder with smaller DINO features (S/16) on YT-VIS 2021 (mean ± standard dev., 3 seeds).
Loss type	Metric
Feat. Rec.	Temp. Sim.	FG-ARI	mBO
✓	14.9 ± 12.0	12.9 ± 5.9
✓	✓	37.0 ± 3.5	29.1 ± 0.6
Table B.5: Loss ablation study on COCO dataset. Metrics are image-based ARI and mBO (mean, 3 seeds).
Loss type	Metric
Feat. Rec.	Self-Sim.	FG-ARI	mBOi
✓	34.8	23.9
✓	28.5	25.6
✓	✓	38.0	25.9
(non-object groupings) the model can "fall into" during training. Increasing the difficulty of the task by adding the temporal similarity loss makes these failure modes less viable: by putting more pressure on the slot bottleneck to encode information, object-based slot groupings are more efficient representations than alternative groupings.
However, we found that the mixer decoder with feature reconstruction does not show instabilities in all settings. For example, training with DINO ViT Base/8 features on MOVi-C or DINO Base/16 features on YouTube-VIS 2021 (Fig. 2 and Fig. 3 in the main text) is relatively stable. We attribute this to the increased task difficulty when predicting ViT "Base" features instead of ViT "Small" features (as in Table B.4). Once more [5, 6, 9, 26, 29], these findings demonstrate the central lesson of unsupervised object discovery: to be successful, the model needs to have sufficient inductive biases, whether they stem from the dataset, the decoder, the grouping module, or the training task.
B.5 Image-Based Feature Similarity on COCO
In this section, we show that our proposed similarity loss can also be useful for image-based datasets, and thus is not restricted to just the video setting. Note that by setting the time-shift  $k$  to 0, the temporal similarities turn into a self-similarities, that is, the target similarities are computed by comparing features from the same image. The resulting similarity maps highlight semantically and spatially related patches and thus could be useful targets to discover objects.
To test this, we train the DINOSAUR method with ViT S/16 [9] with the self-similarity loss on the real-world COCO dataset (see Table B.5). Similar to the results from the time-shift analysis (see  $k = 0$  in Fig. 6b), we find that using the self-similarity loss alone does not seem to carry enough signal to train the model and leads to degraded performance. However, combining the self-similarity loss with feature reconstruction shows significant improvement over using only feature reconstruction. Even though the targets contain the same information overall, different transformations of the original targets (e.g. their relative similarity) create different biases — a combination of these different views into the targets appears to be beneficial for object discovery.
B.6 Sensitivity to Number of Slots During Evaluation
One of the noteworthy properties of slot attention-based models with randomly sampled initialization<sup>5</sup> is that the number of slots can be adjusted during inference. As we demonstrate in Fig. 6b, this is helpful for successfully transferring to datasets that have a different average number of objects per video. For this purpose, it is important to examine how stable the model's performance is if it is used with a different number of slots than during training. Thus, we evaluate our model (trained on YT-VIS 2021 with  $k = 7$  slots) using a varying number of slots (from 1 to 12) and present the results in Fig. B.2. We observe that while using fewer slots steadily deteriorates the performance, our method performs relatively well with a larger number of slots. This suggests that our method is relatively robust to the usage of a larger number of slots than needed. This property is useful for object discovery in images where the number of objects is unknown.
B.7 Effect of ViT Architecture on Final Performance
In addition to studying how the choice of self-supervised method and ViT outputs affect the performance of VideoSAUR (see Fig. 6c and Fig. 6d in the main paper), we also explore the effect of the scale of ViT architecture on the performance. To this end, we compare VideoSAUR with DINO features of 2 different ViT sizes (Small and Base) and two different patch sizes  $(16 \times 16$  and  $8 \times 8$  resolutions) on the MOVi-C dataset (see Table B.6). The results are presented in Fig. B.3. We find that smaller patch size is important for sharper masks (measured by the mBO metric), while both larger architecture and smaller patch size are important for a better split of the scene to object masks (measured by the FG-ARI metric).
C Architectural Details and Hyperparameters
Here we describe details about our model, its training and baselines that we use for comparison. We release our code at https://github.com/martius-lab/videosaur.
5This is in contrast to the fixed learned initialization used in unconditioned SAVi [5] and SAVi++[6].
C.1 SlotMixer Decoder
We describe the SlotMixer decoder, and how we adapted it for 2D decoding. We keep the original terminology from Sajjadi et al. [13] for consistency. SlotMixer performs three steps for decoding: the allocation step assigns slots to spatial positions, the mixing step creates a slot mix for each spatial position, and the render step decodes the slot mix to the final output. See Fig. C.1 for an overview and pseudocode implementing the decoder.
Allocation Step This step takes as input the slots  $s_t \in \mathbb{R}^{K \times M}$  and a learned positional embedding  $p \in \mathbb{R}^{L \times M}$  and outputs a feature vector  $f \in \mathbb{R}^{L \times M}$  using a cross-attention Transformer. In particular, this Transformer iterates several cross-attention operations (and applies residual two-layer MLPs) using the position embeddings as queries to attend into the set of slots, where the position embeddings are residually updated using values from the slots. Importantly, the position embeddings are processed independently of each other. We utilize pre-normalization and also apply a layer norm to the slots before feeding them into the Transformer. In contrast to Sajjadi et al. [13], we use a learned positional embedding initialized from a normal distribution instead of 3D encodings for the query rays, and do not apply a MLP to the positional embedding.
Mixing Step The mixing step is similar to a single-head attention step using the features  $\pmb{f}$  as queries and the slots  $s_t$  as keys, where the slots are averaged as the values to form the slot mix  $m \in \mathbb{R}^{L \times M}$  that is used for decoding to the final output:
$$
\mathbf {q} = \operatorname {n o r m} (\boldsymbol {f}) U _ {q} \quad U _ {q} \in \mathbb {R} ^ {M \times M},
$$
$$
\mathbf {k} = \operatorname {n o r m} (\boldsymbol {s}) U _ {k} \quad U _ {k} \in \mathbb {R} ^ {M \times M},
$$
$$
\boldsymbol {A} = \operatorname {s o f t m a x} \left(\mathbf {q k} ^ {\top} / \sqrt {M}\right) \quad \boldsymbol {A} \in \mathbb {R} ^ {L \times K},
$$
$$
\boldsymbol {m} = s \boldsymbol {A} \quad \boldsymbol {m} \in \mathbb {R} ^ {L \times M}.
$$
**Render Step** The render step takes the slot mix  $m \in \mathbb{R}^{L \times M}$ , adds the positional embedding  $p \in \mathbb{R}^{L \times M}$  and applies a MLP with ReLU activation independently to each position:
$$
\boldsymbol {y} = \operatorname {M L P} (\boldsymbol {m} + \boldsymbol {p}).
$$
Instead of adding the positional embedding, we also explored concatenating it to the slot mix; we did not find large differences from doing so.
C.2 ViT Encoders as Dense Features Extractors
The Vision Transformer (ViT) [43] architecture takes an input frame, denoted here as  $\pmb{x} \in \mathbb{R}^{244 \times 224 \times 3}$ , which is divided into a grid of non-overlapping contiguous patches of resolution  $N \times N$ . Each of these patches is then passed through a linear transformation to generate a set of patch feature embeddings,  $\pmb{h}^0 \in \mathbb{R}^{L \times D}$ .
The set of patch tokens is subsequently provided as input to a standard Transformer network. This Transformer network is comprised of a sequence of self-attention and feed-forward layers, alongside
VideoSAUR
STEVE
- Random pattern
Figure B.2: Changing the number of slots during evaluation on the YT-VIS 2021 dataset (mean  $\pm$  standard dev., 5 seeds).
Figure B.3: Effect of ViT architecture choice and patch size on VideoSAUR training on the MOVi-C (mean  $\pm$  standard dev., 3 seeds).
Table B.6: ViT networks configuration.
Model	Patch Size	Dim	Heads	Tokens	Params
Small	16	384	6	196	21M
Small	8	384	6	784	21M
Base	16	768	12	196	85M
Base	8	768	12	784	85M
Figure B.4: Additional visualization of affinity matrix  $\mathbf{A}$ , transition probabilities  $\mathbf{P}$  and decoder predictions of transition probabilities  $\hat{\mathbf{P}}$  between patches (marked by purple and green) of the frame  $\mathbf{x}_t$  and patches of the next frame  $\mathbf{x}_{t+1}$  for YouTube-VIS 2021 validation videos. Red indicates maximum affinity/probability.
residual connections for each layer. These residual connections are important for letting each patch representation  $h^i$  keep a correspondence to the original image patch representation  $h^0$  and thus making the patch representation a dense (with the resolution  $\sqrt{L} \times \sqrt{L}$ ) representation of the image. In the self-attention layers, the token representations are updated through an attention mechanism that takes into account the representations of all tokens:
$$
\begin{array}{l} [ \mathbf {q} ^ {i}, \mathbf {k} ^ {i}, \mathbf {v} ^ {i} ] = \mathbf {h} ^ {i - 1} \mathbf {U} _ {q k v} \quad \mathbf {U} _ {q k v} \in \mathbb {R} ^ {D \times 3 D} \\ \boldsymbol {A} = \operatorname {s o f t m a x} \left(\mathbf {q} ^ {i} \mathbf {k} ^ {i \top} / \sqrt {D}\right) \quad \boldsymbol {A} \in \mathbb {R} ^ {L \times L} \\ \mathbf {o} ^ {i} = \mathbf {A v} ^ {i} \\ \end{array}
$$
Here  $\mathbf{q}^i, \mathbf{k}^i, \mathbf{v}^i, \mathbf{o}^i$  are queries, keys, values and outputs of the self-attention layer  $i$ . In contrast to DINOSAUR [9], which uses the outputs  $\mathbf{o}^i$  as the target image representation, we use attention keys
Figure B.5: Failure case for long video prediction. Note that the original videos in YouTube-VIS are resampled from the original 30 fps to 6 fps, thus the original length of the video is 250 frames. The slots are reassigned to the background, while small objects are not recognized.
Figure C.1: SlotMixer decoder. Left: SlotMixer performs three steps for decoding: the allocation transformer assigns slots  $s_t$  to spatial positions  $p$ , the slot mixing step creates a slot mix for each spatial position, and the render MLP decodes the slot mix to the final output  $y_t$ . Right: PyTorch-like pseudocode for the SlotMixer decoder.
pos_emb = Param(random_norm(n_pos, dim))  
def mixer(slots):  
    bs, n_slots, dim = slots.shape  
    pos_emb = pos_emb unsqueeze(0) \  
    .expand(bs, -1, -1)  
# Step 1: Allocation Transformer  
feats = xa_transf(q=pos_emb,  
                    kv=norm_kv(slots))  
# Step 2: Slot Mixing  
q = linear_q(norm_q(feats))  
k = linear_k(norm_k(slots))  
dots = einsum("bpd, bsd -> bps",  
                    q, k)  
dots *= q.shape[-1]**-0.5  
attn = softmax(dots, dim=-1)  
slot_mix = einsum("bps, bsd -> bpd",  
                    attn, slots)  
# Step 3: Rendering with MLP  
slot_mix += pos_emb  
outputs = mlp(slot_mix)  
return outputs
$\mathbf{k}^i$  from the last self-attention layer of ViT as the dense image representation that is provided to the temporal similarity loss (see Fig. 6c for a detailed comparison of these representations). However, we still use the outputs  $\mathbf{o}^i$  as the input to the slot attention grouping module.
C.3 Other Modules
We group the dense encoder features with a recurrent slot attention module similar to Singh et al. [7] and Kipf et al. [5]. First, we transform the original features with a two-layer MLP with an output dimension equal to the slot dimension. Second, we use a slot attention module initialized with randomly sampled slots to group the first frame features, while for subsequent frames, we initialize the slot attention module with the slots of the previous frame, additionally transformed with a predictor module. We use the GRU recurrent unit in the slot attention grouping, but not the residual MLP. Similar to SAVi [5] and STEVE [7], we use a one-layer transformer as the predictor module. In addition, we propose to decouple the number of Slot Attention iterations in the first frame and other frames of the video. This allows more iterations for the first frames (we use 3 iterations similar to image-based methods) and fewer iterations for the next frames where the initialization is much
Table C.1: Next-frame feature prediction with different decoders.
MOVi-C	MOVi-E	YT-VIS
FG-ARI	mBO	FG-ARI	mBO	FG-ARI	mBO
One decoder head	44.6	23.5	61.3	22.1	33.4	24.6
Two decoder heads	47.2	24.7	62.9	24.0	37.9	27.3
better (we use 2 iterations). For computational reasons, we were training on relatively short 4-frame segments of original videos, i.e.  $T = 4$ .
C.4 Next-Frame Feature Prediction Details
In this part, we cover implementation details for the next-frame feature prediction ablation presented in Table 2 and in Table 3. Reconstructing frame features from the current and next frame simultaneously with a single decoder is problematic because the decoder masks that are used for evaluation would be in reference to both the current and next frame. One way to overcome this problem is by using two decoder heads:  $d_{\mathrm{current}}$  for the current frame and  $d_{\mathrm{next}}$  for the next frame. Each head produces its own predictions and masks. In this case, masks from the  $d_{\mathrm{current}}$  head can be used for evaluation. While more powerful, this approach also requires more memory and is slower than standard setting with only one head. In our experiments, we confirm that the version with two different Mixer decoders performs better than simultaneous reconstruction with one decoder (see Table C.1). We use this better version for our comparisons even though it is heavier than our method which needs only one decoder.
C.5 Baselines
Block Pattern The block pattern baseline serves to show metrics for a trivial decomposition of the video into regular blocks. It is intended to show the difficulty of the dataset and how much object-centric methods improve upon such a trivial solution, as the metrics values could be difficult to interpret without further calibration. To this end, we are splitting the video to  $k$  spatial blocks consistently for all frames of the video, similar to how Seitzer et al. [9] are splitting images into regular blocks.
SAVi We reimplement SAVi [5] close to the official implementation<sup>6</sup>. In particular, we use the SAVi-L architecture for all experiments. This corresponds to the version using a ResNet-34 encoder as described by Kipf et al. [5], and a CNN broadcast mixture decoder with 4 layers. We apply the unconditional version of SAVi, using a fixed learned slot initialization instead. We train the model for 200000 steps on all datasets, with a batch size of 64, using image reconstruction as the training signal. For training, we use videos with 4 frames, with a single slot attention step per frame.
STEVE We reimplement STEVE [7] close to the official implementation<sup>7</sup>. We use the proposed configuration for the MOVi datasets and only change the number of slots to 11 for MOVi-C, 15 for MOVi-E and 7 for YT-VIS 2021. STEVE trains a dVAE [50] on the video frames to extract a discrete latent code that is used as the reconstruction target. STEVE uses a CNN encoder with 4 layers and a Transformer decoder with 8 layers. We train the model for 200000 steps for MOVi-C and YT-VIS datasets and for 100000 steps for MOVi-E datasets which resulted in optimal performance on this dataset, with a batch size of 24. Like Singh et al. [7], for training, we use videos with 3 frames, with two slot attention steps per frame.
C.6 Hyperparameters
The full list of the hyperparameters used to train VideoSAUR on MOVi-C, -D, -E, and YouTube-VIS 2021 datasets are presented in Table C.2. Most of the hyperparameters are similar to previous methods, while new ones (such as softmax temperature  $\alpha$  and time shift  $k$ ) are optimized using a grid search.
$^{6}$ https://github.com/google-research/slot-attention-video/
7https://github.com/singhgautam/stve