SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table C.2: Hyperparameters of VideoSAUR for the main results on MOVi-C, MOVi-E, and YouTube-VIS 2021 datasets.
Dataset	MOVi-C	MOVi-E	YouTube-VIS
Training Steps	100k	100k	100k
Batch Size	128	128	128
Training Segment Length	4	4	4
LR Warmup Steps	2500	2500	2500
Optimizer	Adam	Adam	Adam
Peak LR	0.0004	0.0004	0.0004
Exp. Decay	100k	100k	100k
ViT Architecture	ViT Base	ViT Base	ViT Base
Patch Size	8	8	16
Feature Dim. Dfeat	768	768	768
Gradient Norm Clipping	0.05	0.05	0.05
Image/Crop Size	224	224	224
Cropping Strategy	Full	Full	Rand. Cent. Crop
Augmentations	-	-	Rand. Hori. Flip
Image Tokens	784	784	196
Slot Attention	Slots	11	15	7
Iterations (first / other frames)	3/2	3/2	3/2
Slot Dim. Dslot	128	128	64
Predictor	Type	Transformer	Transformer	Transformer
Layers	1	1	1
Heads	4	4	4
Decoder	Type	Mixer	Mixer	Mixer
Allocator Transformer Layers	2	2	3
Allocator Transformer Heads	4	4	4
Renderer MLP Layers	4	4	3
Renderer MLP Hidden Dim.	1024	1024	1024
Loss	Softmax Temperature τ	0.075	0.075	0.25
Time-shift k	1	1	1
Feature reconstruction weight α	-	-	0.1
C.7 Compute Requirements
We used a cluster of A100 GPUs (with 40 and 80 Gb memory) for running the experiments. A single training run (100k steps) of VideoSAUR equipped with DINO B/16 with a batch size of 128 takes roughly 18 hours on one A100 GPU with 40 GB memory. We use 5 seeds for the final results and 3 seeds for other experiments. Overall, we estimate the total compute spend on the whole project including training of all baselines and the method development (including dead ends) to be around 800-1000 GPU days (this is a rough estimate obtained from our cluster usage).
In this section, we provide details about the datasets used in this work. See Table D.1 for an overview.
MOVi datasets For MOVi-C, MOVi-D and MOVi-E, we use the standard training and test splits provided in the respective releases of MOVi datasets: 9750 training videos and 250 validation videos. For a fair comparison, all the methods are trained on images with the same initial resolution  $128 \times 128$ . Consistent with previous work [5-7, 9], we utilize the validation split of the MOVi dataset for our evaluations.
YouTube-VIS datasets The YouTube-VIS datasets [14, 46] are benchmarks originally used for supervised video instance segmentation. The video instance segmentation task involves segmenting, tracking and classifying instances throughout a video. It is a challenging dataset because it contains
Table D.1: Overview of datasets used in this work. For the training process, we solely utilize images or videos derived from the relevant datasets, with no reliance on labels. To generate central crops, we initially resize the mask so that its shorter dimension is 224 pixels. Subsequently, we extract the most centrally located crop, maintaining a size of 224 by 224 pixels.
Dataset	Videos	Images	Description	Citation
MOVi-C, -D, -E	9750	-	Train split videos	Greff et al. [12]
MOVi-C, -D, -E validation	250	-	Val. split w. instance segm. labels	Greff et al. [12]
COCO 2017	-	118287	Train split	Lin et al. [48]
COCO 2017 validation	-	5000	Val split w. instance segm. labels	Lin et al. [48]
YouTube-VIS 2021	2785	-	Part of train split videos	Yang et al. [14]
YouTube-VIS 2021	210	-	Part of train split w. instance segm. labels	Yang et al. [14]
YouTube-VIS 2019	200	-	Part of train split w. instance segm. labels	Yang et al. [46]
DAVIS-2017-Unsupervised	30	-	Val. split w. instance segm. labels	Pont-Tuset et al. [47]
various different classes of objects and the complexities of real-world video dynamics. To the best of our knowledge, this is the first work attempting unsupervised instance segmentation on YouTube-VIS.
There are two different versions of this dataset: YouTube-VIS 2019 and YouTube-VIS 2021. The YouTube-VIS 2019 dataset is mainly derived from the Video Object Segmentation (VOS) dataset. It has a limited number of individual instances (an average of 1.7 per video for the training set), and the categories of instances in the same video are usually different. In contrast, the 2021 edition of YouTube-VIS incorporates a higher quantity of objects with more difficult trajectories (average 3.4 per video for the additional videos in the train set) and thus are more interesting for object-centric learning. We sample both training and validation frames with a rate of 6 frames per second (each 5th frame of the original videos with 30 fps).
As the original validation dataset is not publicly available, and the evaluation server does not compute the object-centric metrics we need, we split the original training set into two parts (210 videos for validation and the other videos for training). Overall, we use 2775 videos for training, and 210 videos additionally added in YT-VIS 2021 train for validation.
DAVIS dataset The DAVIS-2017-Unsupervised dataset [47] is used for video object segmentation and also contains videos with multiple objects per video (average is 1.97 for 30 validation videos). As those videos are not related to YouTube-VIS videos, this dataset is useful for the evaluation of transfer abilities for real-world object-centric learning algorithms.
COCO dataset To test the properties of our proposed similarity loss for image-based object-centric learning we use the COCO dataset [48]. Similar to Seitzer et al. [9], we use 118287 training images (without labels) to train and 5000 validation images (with labels) for performance evaluation.
E Additional Examples
We include additional example predictions of our model:
Figure E.1: comparing VideoSAUR to STEVE on Youtube-VIS 2021.
Figure E.2: comparing VideoSAUR to STEVE on MOVi-C.
Figure E.3: comparing VideoSAUR to STEVE on MOVi-E.
- Figure E.4: comparing feature reconstruction and temporal similarity loss on MOVi-C.
- Figure E.5: long-term video predictions on Youtube-VIS 2021 (VideoSAUR with DINO B/16 features).
- Figure E.6: long-term video predictions on Youtube-VIS 2021 (VideoSAUR with DINOv2 B/14 features).
Figure E.1: Additional examples on YouTube-VIS 2021.
Figure E.2: Additional examples on MOVi-C.
Figure E.3: Additional examples on MOVi-E.
Figure E.4: Difference between VideoSAUR train with feature reconstruction and temporal feature similarity losses on MOVi-C. We show videos with larger differences in performance between the two methods.
Figure E.5: Prediction of VideoSAUR with DINO B/16 features on longer videos from YouTube-VIS 2021. 31
Figure E.6: Prediction of VideoSAUR with DINOv2 B/14 features on longer videos from YouTube-VIS 2021. 32