SlowGuess/NeurIPS_2023_OCR huggingface dataset:
FrameMAE. We compare SiamMAE with FrameMAE (Table 2a), an extension of MAEs to video frames, i.e., joint encoder and joint decoder with symmetric masking ratio. FrameMAE performs significantly worse when the masking ratio is too high (90%) or too low (50%). With a 90% masking ratio, the task becomes challenging (higher loss) due to the insufficient number of patches available to learn temporal correspondence. When the masking ratio is 50%, the task becomes easier (lower loss) and the network can reconstruct the frames without relying on temporal information, due to the spatial redundancy of images. SiamMAE with an asymmetric masking ratio works best.
Encoder-decoder design. An important design decision of SiamMAE is the choice of encoder and decoder. We study the performance of various combinations of encoders and decoders with asymmetric masking in Table 2b. Joint encoders perform significantly worse compared to their siamese counterparts across all decoder designs. This can be attributed to the difference between the training and testing setups, as each frame is processed independently during the testing phase.
Siamese encoder with cross decoder performs worst among siamese encoders. We also observe that the training loss is higher and the reconstructed frames are spatially incoherent, as all patches from  $f_{2}$  are processed independently. Finally, the combination of a siamese encoder with a cross-self decoder outperforms all other pairings. The cross-attention operation is similar to the affinity matrix used
spatial	color	J&Fm	Jm	Fm
56.8	55.5	58.1
✓	58.1	56.6	59.6
✓	55.8	54.6	57.0
✓	✓	56.7	55.4	57.9
(a) Data augmentation. SiamMAE requires minimal data augmentation.
frame gap	J&Fm	Jm	Fm
4	55.1	53.5	56.7
8	56.4	54.9	57.8
16	58.0	56.7	59.4
32	57.7	56.3	59.1
4-48	58.1	56.6	59.6
(b) Frame sampling. Random frame gap works the best.
Table 3: Data augmentation. We ablate the importance of manual (spatial and color jitter) and natural data augmentation (frame sampling) for learning correspondence via our SiamMAE on DAVIS [95]. The table format follows Table 2.
Figure 4: Training schedule and patch size. Evaluation of SiamMAE performance for 3 downstream tasks for ViT-S/16 and ViT-S/8 models. Across all tasks, longer training and smaller patch sizes lead to improved performance.
in self-supervised correspondence learning and is also used for label propagation in our evaluation protocol. Hence, by processing the frames independently and decoding them via the cross-self decoder, the network is encouraged to learn good representations for dense visual correspondence.
Masking. Next, we discuss the effect of the masking scheme for the combination of a siamese encoder with a self-cross-decoder. Random symmetric masking performs poorly and is also worse than the corresponding FrameMAE configurations (Table 2a, 2c). We also study the grid-wise mask sampling strategy, which keeps every alternate patch. This is an easier task, as the masking pattern enables the network to exploit and learn spatio-temporal correlations. Although we see significant gains (41.5 to 48.2), performance is still significantly poor compared to SiamMAE. In Table 2d, we study the role of different asymmetric masking ratios. We notice a clear trend: increasing the masking ratio from  $50\%$  to  $95\%$  increases the performance  $(49.0\%$  to  $58.1\%)$ .
Data augmentation. In Table 3a we study the influence of different data augmentation strategies. Similar to the findings in the image [24] and video [27] domains, we find that SiamMAE does not require extensive data augmentation to achieve competitive performance. Random cropping with a scale range of  $[0.5,1]$  and horizontal flipping works best, and adding color jitter leads to performance degradation. Contrastive methods like DINO show impressive  $k$ -NN performance by using extensive data augmentation. In contrast SiamMAE achieves superior results by relying on natural data augmentation available in videos, discussed next.
Frame sampling. Video data is a rich source of data augmentation, e.g. variations in pose, lighting viewpoint, occlusions, etc. To effectively leverage this, we study the importance of frame sampling in Table 3b. The performance improves as we increase the frame sampling gap. Natural videos frequently exhibit gradual temporal changes; therefore, increasing the frame interval results in a more robust natural data augmentation, which in turn enhances performance. Our frame sampling strategy is simple and effective: randomly sample frames with a frame gap ranging from 4 to 48 frames.
Figure 5: Self-attention maps. Self-attention maps from a ViT-S/8 model. We examine the self-attention of the [CLS] token on the heads of the final layer. Unlike contrastive methods, there is no explicit loss function acting on the [CLS] token. These self-attention maps suggest that the model has learned the notion of object boundaries from object motion in videos. See project page for videos.
Training schedule. As noted earlier, our ablations are based on 400-epoch pre-training. Figure 4 studies the influence of the training length schedule for ViT-S/16 and ViT-S/8 models for the three downstream tasks considered in this work. Due to compute limitations, we report evaluations of a single model at different checkpoints. Across both patch sizes and across all tasks, the accuracy improves gradually with longer training.
Prediction target. In Table 5a (see § B) we study the importance of predicting the future. We consider two additional SiamMAE variations: one where we always predict the past frame (f1) and another where the order of frame prediction (f1 or f2) is randomized. All variations perform reasonably well, with our default setting (i.e., predicting the future) performing the best. We emphasize predicting future behavior due to its natural alignment with most real-world applications, which often necessitate the anticipation or prediction of agents' future behavior.
4.4 Attention Map Analysis
In Fig. 5, we visualize the self-attention map of the ViT-S/8 model. We use the [CLS] token as the query and visualize the attention of a single head from the last layer with 720p images from ImageNet. We find that the model attends to the object boundaries. For instance, it can clearly delineate iconic objects (such as the sheep in the first row, first column), multiple objects (like the three baseball players in the third row, sixth column), and even when the scene is cluttered (as seen with the bird in the second row, fourth column). While other self-supervised learning approaches [17, 101] have reported emergent object segmentation capabilities, we are unaware of any methods demonstrating an emergent ability to predict object boundaries. This emergent ability is unique and surprising since, unlike contrastive learning approaches, no loss function operates on the [CLS] token in SiamMAE (or in MAEs). We attribute the emergence of this ability to our asymmetric masking ratio, which encourages the model to learn about object boundaries from object motion in videos.
4.5 Failure Analysis
We evaluate the quality of learnt representations using label propagation and consequently inherit its limitations. Specifically, the inference algorithm lacks semantic understanding, leading to globally inconsistent labels (Fig 6). This limitation can be overcome by fine tuning the learnt representations with task specific architectural changes. Additionally, there are instances where the inference process
Figure 6: Failure analysis. A key disadvantage of using label propagation is the lack of global semantic understanding of objects. Assigning labels based solely on low-level features can lead to globally inconsistent labels, as illustrated by the following examples: (a) a segmentation mask that covers both hands; (b) a pose key-point determined using the person's hair, rather than their posture; (c) challenges in assigning labels to parts of the object that are occluded in the reference frame; and (d) the inability to assign labels to fine object details, such as the spokes of a tire.
might miss intricate object details, like the spokes of a tire. While this shortcoming can be mitigated by using a smaller patch size during training and inference, it comes at a higher compute cost.
In this work, we introduce SiamMAE, a simple method for representation learning from videos. Our approach is based on the intuition that the temporal dimension should be treated differently from the spatial dimension. We demonstrate that an asymmetric masking strategy, i.e., masking a high percentage of patches of the future frame while keeping the past frame unchanged, is an effective strategy for learning correspondence. By predicting a majority fraction of the future frame, we find that our SiamMAE is able to learn the notion of object boundaries (Fig 5). Moreover, unlike MAEs, features learned via our approach can be used in a zero-shot manner and outperform state-of-the-art self-supervised methods in various tasks, such as video object segmentation, pose keypoint propagation, and semantic part propagation. SiamMAE achieves these competitive results without the need for data augmentation, handcrafted tracking-based pretext tasks, or other techniques to prevent representational collapse. We hope our work will encourage further exploration of learning representations by predicting the future.
Future work. Our study focuses on learning correspondences by operating on pairs of video frames. This choice was driven by the empirical success of the approach and the limited computational resources available. Consequently, we believe that further investigation is needed to understand the role of predicting multiple future frames based on past frames, both for general visual representation learning and for correspondence learning specifically. An important future direction is to systematically examine the scalability of our approach in terms of both data and model size. Following previous work, we utilize internet videos for pre-training. However, it is essential to also investigate the impact of different types of video data, such as egocentric videos [102] versus "in-the-wild" internet videos. Lastly, our learned representations hold potential for applications involving embodied agents (i.e., robots), as the concept of correspondence could be useful in tasks such as object manipulation, navigation, and interaction within dynamic environments.
Acknowledgments. We thank Abhishek Kadian for helpful discussions. This research was in part supported by the Stanford Institute for Human-Centered Artificial Intelligence (HAI) and ONR MURI N00014-22-1-2740.
[99] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefer, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8129-8138, 2020. 6, 18
[100] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7.6,18
[101] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=ydopy-e6Dg.9
[102] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. arXiv preprint arXiv:2110.07058, 2021. 10
[103] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703. PMLR, 2020. 18
[104] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016. 18
[105] Priya Goyal, Piotr Dálár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: TrainingImagenet in 1 hour. arXiv preprint arXiv:1706.02677, 2017. 18
Training. Our training settings follow [24] and we build on the open-source implementation of MAEs (https://github.com/facebookresearch/mae) for all our experiments. We use the parameters specified in the original implementation unless specified otherwise in Table 4a. All our experiments are performed on 4 Nvidia Titan RTX GPUs for ViT-S/16 models, and on 8 Nvidia Titan RTX GPUs for ViT-S/8 models and ViT-B models.
Evaluation methodology. Our evaluation methodology follows prior work [14-16] and in Table 1 we report results previously reported in these studies. For recent self-supervised learning approaches like DINO, MAEs, MAE-ST and VideoMAE, we carry out a comprehensive grid search on the evaluation hyperparameters listed in Table 4b, and report the optimal results obtained. The evaluation parameters for SiamMAE can be found in Table 4b.
config	value
optimizer	AdamW [100]
optimizer momentum	β1, β2=0.9, 0.95 [103]
weight decay	0.05
learning rate	1.5e-4
learning rate schedule	cosine decay [104]
warmup epochs [105]	40
epochs	2000 (ablations 400)
repeated sampling [99]	2
augmentation	hflip, crop [0.5, 1]
batch size	2048 (S) 1024 (B)
frame sampling gap	[4, 48]
(a) Kinetics pre-training setting.
config	DAVIS	VIP	JHMDB
top-k	7	10	7
queue length	20	20	20
neighborhood size	20	8	20
Prediction target. In Table 5a we study the importance of predicting the future. We consider two additional SiamMAE variations: one where we always predict the past frame (f1) and another where the order of frame prediction (f1 or f2) is randomized. All variations perform reasonably well, with our default setting (i.e., predicting the future) performing the best. We emphasize predicting future behavior due to its natural alignment with most real-world applications, which often necessitate the anticipation or prediction of agents' future behavior.
Frame overlap analysis. To perform frame overlap analysis, we sampled video frames from the Kinetics-400 validation set with the specified frame gap and calculated two image similarity metrics: mean squared error (mse) and structural similarity index measure (ssim). We observed that either a very high overlap (low frame gap, high ssim, and low mse) or a low overlap (high frame gap, low ssim, and high mse) adversely affects performance. Sampling with a frame gap of 16 or within a range of [4, 48] yields the best results. Interestingly, the overlap metrics for a frame gap of 16 and [4, 48] are comparable, suggesting that a particular degree of overlap is important for best results.
(b) Evaluation setting.
Table 4: Training and evaluation hyperparameters.
pred. target	J&Fm	Jm	Fm
f1 (past)	57.5	56.0	59.0
random (f1, f2)	57.8	56.3	59.2
f2 (future)	58.1	56.6	59.6
(a) Prediction target. Predicting the future works the best.
frame gap	ssim	mse	J&Fm
4	0.6231	0.0230	55.1
8	0.5343	0.0360	56.4
16	0.4749	0.0480	58.0
32	0.4221	0.0597	57.7
4-48	0.4548	0.0528	58.1
(b) Frame sampling. Random frame gap works the best.
Table 5: Additional ablations. We ablate the importance of predicting the future and perform overlap analysis for different frame gaps. The table format follows Table 2.