SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Agrim Gupta<sup>1*</sup> Jiajun Wu<sup>1</sup> Jia Deng<sup>2</sup> Li Fei-Fei<sup>1</sup>
$^{1}$ Stanford University,  $^{2}$ Princeton University
Establishing correspondence between images or scenes is a significant challenge in computer vision, especially given occlusions, viewpoint changes, and varying object appearances. In this paper, we present Siamese Masked Autoencoders (SiamMAE), a simple extension of Masked Autoencoders (MAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. These frames are processed independently by an encoder network, and a decoder composed of a sequence of cross-attention layers is tasked with predicting the missing patches in the future frame. By masking a large fraction (95%) of patches in the future frame while leaving the past frame unchanged, SiamMAE encourages the network to focus on object motion and learn object-centric representations. Despite its conceptual simplicity, features learned via SiamMAE outperform state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks. SiamMAE achieves competitive results without relying on data augmentation, handcrafted tracking-based pretext tasks, or other techniques to prevent representational collapse.
"The distinction between the past, present, and future is only a stubbornly persistent illusion."
—Albert Einstein
Time is a special dimension in the context of visual learning, providing the structure within which sequential events are perceived, cause-effect relationships are learned, objects are tracked as they move through space, and future events are predicted. Central to all of these capabilities is the ability to establish visual correspondence over time. Our visual system is adept at establishing correspondence between scenes despite occlusions, viewpoint changes, and object transformations. This capability is unsupervised, critical to human visual perception, and remains a significant challenge in computer vision. Equipping machines with such a capability enables a wide range of applications such as object segmentation and tracking in videos, depth and optical flow estimation, and 3D reconstruction [1-8].
A powerful self-supervised learning paradigm is predictive learning, i.e., predicting any unobserved or hidden part of the signal from any observed or unhidden part of the signal [9]. Notably, this form of predictive learning has been used for learning correspondences [10-12] by predicting the colors of grayscale future frame by observing a (colorful) past reference frame. However, the performance of these methods has trailed behind contrastive self-supervised learning [13] approaches. State-of-the-art methods [14-17] for learning correspondence primarily employ some form of contrastive learning [13]. Intuitively, contrastive learning-based approaches are well-suited for the task of learning correspondence, as they utilize extensive data augmentation to learn features invariant to changes in pose, lighting, viewpoint, and other factors. However, a major criticism of contrastive approaches is their reliance on careful selection of augmentations to learn useful invariances [18], along with a suite of additional components [19, 20, 17, 21] to prevent representational collapse.
*Correspondence to agrim@stanford.edu
Figure 1: Siamese Masked Autoencoders. During pre-training we randomly sample a pair of video frames and randomly mask a huge fraction (95%) of patches of the future frame while leaving the past frame unchanged. The two frames are processed independently by a siamese encoder parametrized by a ViT [31]. The decoder consists of a sequence of cross-attention layers and predicts missing patches in the future frame. Videos available at this project page.
Recently, predictive learning methods like masked language modelling [22, 23] and masked visual modeling (MVM) [24-26] have demonstrated promising results in natural language processing and computer vision domains. MVM methods like Masked Autoencoders (MAE) learn good visual representations without relying on data augmentation by learning to reconstruct the missing patches from randomly masked input image patches. Extending MVM methods from images to videos for learning correspondence is however nontrivial for two reasons. First, features learned by MAEs are specialized for the pixel reconstruction task, which show excellent downstream performance on finetuning, but do not transfer well in zero-shot settings. Second, existing extensions of MAEs in the video domain [27, 28] also symmetrically mask a huge fraction of patches across all frames. Unlike images, which are (approximately) isotropic [29], the temporal dimension is special [30], and not all spatio-temporal orientations are equally likely. Hence, symmetrically treating spatial and temporal information might be sub-optimal. Indeed, MAEs trained on videos do not outperform MAEs trained on ImageNet on video instance tracking benchmarks (Table 1).
To address these limitations, we present Siamese Masked Autoencoders (SiamMAE): a simple extension of MAEs for learning visual correspondence from videos. In our approach, two frames are randomly selected from a video clip, with the future frame having a significant portion  $(95\%)$  of its patches randomly masked, while the past frame is left intact. These frames are processed independently by an encoder network, and a decoder composed of a sequence of cross-attention layers is tasked with predicting the missing patches in the future frame. Our asymmetric masking approach encourages the network to model object motion, or in other words, to understand what went where [32]. Simple extensions of MAEs to frames with symmetric masking wastes model capacity on modeling low-level image details. However, by providing the entire past frame as input, our network is primarily focused on propagating the patches from the past frame to their corresponding locations in the future frame. The cross-attention layers in our decoder serve a function akin to the affinity matrix often employed in self-supervised correspondence learning approaches. Empirically, we find that the combination of asymmetric masking, a siamese encoder, and our decoder can effectively learn features suitable for tasks requiring fine-grained and object-level correspondence.
Despite the conceptual simplicity of our method, it outperforms state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation. Moreover, our ViT-S/16 models significantly outperform larger models trained on ImageNet  $(+8.5\%)$ $\mathcal{J}\& \mathcal{F}_m$  for ViT-B) and Kinetics-400  $(+7.4\%)$ $\mathcal{J}\& \mathcal{F}_m$  for ViT-L) via MVM in video object segmentation tasks. We also observe significant performance gains across all tasks with models trained with smaller patch sizes (ViT-S/8, ViT-B/8). SiamMAE achieves competitive results without relying on data augmentation [16, 17], handcrafted tracking-based pretext tasks [15, 14], multi-crop training [17], additional techniques to prevent representational collapse [10, 11, 17, 16] or enhance performance [11]. We believe that our detailed analysis, straightforward approach, and state-of-the-art performance can serve as a robust baseline for self-supervised correspondence learning.
Temporal correspondence. The visual world is smooth and continuous [33, 34], providing a rich source of information for biological and machine vision systems. In biological vision, infants learn about objects and their properties by establishing temporal correspondence, taking advantage of the inherent smoothness in the visual input [35]. Similarly, in machine vision, learning fine-grained correspondence from video frames is an important problem that has been studied for decades in the form of optical flow and motion estimation [36-42, 5, 6, 43, 44, 7]. However, despite their impressive performance, these methods rely on costly human-annotated or synthetic data with pixel-level ground truth annotations [45, 46]. A more semantically meaningful task involves determining object-level correspondence i.e., visual object tracking [47-52]. One popular approach is tracking-by-matching methods that utilize deep features learned via supervised [53, 54] or self-supervised learning [10-12, 14-16] on videos. State-of-the-art methods [14-17] for self-supervised feature learning for correspondence primarily employ some form of contrastive learning [13]. Predictive learning has also been used for learning correspondences [10, 11] by predicting the target colors for gray-scale input frame by observing a colorful reference frame. However, the performance of these methods has trailed behind contrastive approaches. In this work, we show that predictive learning based methods can be used for learning fine-grained and object-level correspondence.
Self-supervised visual representation learning. Self-supervised learning is a way to learn generalizable visual representations often using different pretext tasks for pre-training [55-59]. The presence of temporal information in videos has been leveraged in numerous ways for representation learning, including future prediction [60-62, 62-64, 10, 65], temporal ordering [66-70], object motion [71, 72, 58, 14], and temporal coherence [73, 74]. Recently, the community has made great progress in self-supervised representation learning via contrastive learning [75, 13]. Contrastive methods in the image domain encourage models to learn representations by modeling image similarity and dissimilarity [76, 19, 17, 20] or only similarity [77, 21]. Furthermore, contrastive learning has also been successfully applied to videos [78-82, 82, 83]. However, a limitation of contrastive approaches is their dependence on careful selection of augmentations to learn useful invariances [18], and as well as the need for a suite of additional components [19, 20, 17, 21] to prevent representational collapse.
Masked autoencoders. Masked autoencoders are a type of denoising autoencoder [84] that learn representations by reconstructing the original input from corrupted (i.e., masked) inputs. The introduction of masked language modeling in BERT [85] has had a transformative impact on the natural language processing field, particularly when scaled to large datasets and model sizes [86, 87]. Masked autoencoders have also been successfully adapted to learn representations from images [24-26] and videos [28, 27]. Our work studies a simple extension of MAEs [24] to videos. However, unlike prior methods [28, 27] that symmetrically mask all frames, we propose an asymmetric masking scheme, leaving the past frame unchanged and masking a higher percentage of the future frame.
Siamese networks. Siamese networks [88] are weight-sharing neural networks used to compare entities. They have been widely used in a variety of application domains [88-90] including tracking [53]. Siamese networks have been extensively used in modern contrastive learning approaches [76, 19, 17, 20], as their design allows an easy way to learn invariant visual representations from data. Inspired by the success of masked autoencoders, researchers have also explored combining contrastive learning with siamese networks and masked visual modeling [91, 92]. However, we are not aware of any previous studies that have investigated siamese masked autoencoders using asymmetric masking for representation learning from videos.
Our goal is to develop a self-supervised method for learning correspondence. To that end, we study a simple extension of MAE [24] to video data (Fig. 1). In this section, we describe the key components of our Siamese Masked Autoencoder.
Patchify. Given a video clip with  $L$  frames, we first randomly sample 2 frames  $f_{1}$  and  $f_{2}$ . The distance between these two frames is determined by selecting a random value from the predetermined range of potential frame gaps. Following the original ViT [31], we "patchify" each frame by converting it into a sequence of non-overlapping  $N \times N$  patches. Finally, position embeddings [94]
Figure 2: Visualizations on the Kinetics-400 [93] validation set (masking ratio  $90\%$ ). For each video sequence, we sample a clip of 8 frames with a frame gap of 4 and show the original video (top), SiamMAE output (middle), and masked future frames (bottom). Reconstructed are shown with  $f_{1}$  as the first frame of the video clip and  $f_{2}$  as the remaining frames, using a SiamMAE pre-trained ViT-S/8 encoder with a masking ratio of  $95\%$ .
are added to the linear projections [31] of the patches, and a [CLS] token is appended. We do not use any temporal position embeddings.
Masking. Natural signals like images and videos are highly redundant, exhibiting spatial and spatiotemporal redundancies, respectively [33, 34]. To create a challenging predictive self-supervised learning task, MAEs randomly mask a high percentage (75%) of image patches [24] and extensions to videos [28, 27] use an even higher masking ratio (90%). In both images and videos, the masking strategy is symmetric, i.e., all frames have a similar masking ratio. This deliberate design choice prevents the network from leveraging and learning temporal correspondence, leading to sub-optimal performance on correspondence learning benchmarks.
We posit that asymmetric masking can create a challenging self-supervised learning task while encouraging the network to learn temporal correlations. Specifically, we do not mask any patches in  $f_{1}$  (0%) and mask a very high ratio (95%) of patches in  $f_{2}$ . By providing the entire past frame as input, the network only needs to propagate the patches from the past frames to their appropriate locations in the future frame. This, in turn, encourages the network to model object motion and focus on object boundaries (Fig. 5). To further increase the difficulty of the task, we sample the two frames with a large temporal gap. Although predicting further into the future is inherently ambiguous and may yield multiple plausible outcomes, providing a small number of patches as input for the second frame results in a challenging yet tractable self-supervised learning task.
Encoder. We explore two different encoder configurations for processing input frames.
A joint encoder is a natural extension of image MAEs to a pair of frames. The unmasked patches from the two frames are concatenated and then processed by a standard ViT encoder.
A siamese encoder [88] are weight-sharing neural networks used for comparing entities and are an essential component of modern contrastive representation learning methods [21]. Siamese networks have been used for correspondence learning [53, 11, 10] and often require some information bottleneck to prevent the network from learning trivial solutions. For example, Lai and Xie [11] propose to use color channel dropout to force the network to avoid relying on colors for matching correspondences.
Object Propagation
Semantic Part Propagation
Figure 3: Qualitative results on three downstream tasks: video object segmentation (DAVIS-2017 [95]), human pose propagation (JHMDB [96]) and semantic part propagation (VIP [97]).
We use siamese encoders to process the two frames independently and our asymmetric masking serves as an information bottleneck.
Decoder. The output from the encoder is projected using a linear layer and [MASK] tokens with position embeddings are added to generate the full set of tokens corresponding to the input frame. We explore three different decoder configurations which operate on the full set of tokens.
A joint decoder applies vanilla Transformer blocks on the concatenation of full set of tokens from both frames. A key downside of this approach is a substantial increase in GPU memory requirement, especially when using smaller patch sizes.
A cross-self decoder is similar to the original encoder-decoder design of the Transformer [94] model. Each decoder block consists of a cross-attention layer and a self-attention layer. The tokens from  $f_{2}$  attend to the tokens from  $f_{1}$  via the cross-attention layer and then attend to each other via the self-attention layer. We note that the cross-attention layer is functionally similar to the affinity matrix often used in self-supervised correspondence learning approaches [11, 10].
A cross decoder consists of decoder blocks with only cross-attention layers, where tokens from  $f_{2}$  attend to the tokens from  $f_{1}$ .
Finally, the output sequence of the decoder is used to predict the normalized pixel values [24] in the masked patches.  $l2$  loss is applied between the prediction of the decoder and the ground truth.
In this section, we evaluate our method on three different tasks, compare its performance with prior state-of-the-art methods, and perform extensive ablation studies of different design choices. For qualitative results, see Fig. 2, Fig. 3, Fig. 5 and videos on our project website.
Backbone. We use the ViT-S/16 model for most of our experiments as it is similar to ResNet-50 in terms of the number of parameters (21M vs 23M) and allows for fair comparisons across different self-supervised learning and correspondence learning methods.
Pre-training. Models are pre-trained using Kinetics-400 [93] for self-supervised learning. SiamMAE takes as input pairs of randomly sampled frames  $(224 \times 224)$  with a frame gap ranging from 4 to 48 frames at a rate of 30 fps. We perform minimal data augmentation: random resized cropping
Method	Backbone	Dataset	DAVIS	VIP mIoU	JHMDB
J&Fm	Jm	Fm	PCK@0.1	PCK@0.2
Supervised [98]	ResNet-50	ImageNet	66.0	63.7	68.4	39.5	59.2	78.3
SimSiam [20]	ResNet-50	ImageNet	66.3	64.5	68.2	35.0	58.4	77.5
MoCo [19]	ResNet-50	ImageNet	65.4	63.2	67.6	36.1	60.4	79.3
TimeCycle [14]	ResNet-50	VLOG	40.7	41.9	39.4	28.9	57.7	78.5
UVC [12]	ResNet-50	Kinetics	56.3	54.5	58.1	34.2	56.0	76.6
VFS [16]	ResNet-50	Kinetics	68.9	66.5	71.3	43.2	60.9	80.7
MAE-ST [27]	ViT-L/16	Kinetics	54.6	55.5	53.6	33.2	44.4	72.5
MAE [24]	ViT-B/16	ImageNet	53.5	52.1	55.0	28.1	44.6	73.4
VideoMAE [28]	ViT-S/16	Kinetics	39.3	39.7	38.9	23.3	41.0	67.9
Dino [17]	ViT-S/16	ImageNet	61.8	60.2	63.4	36.2	45.6	75.0
SiamMAE (ours)	ViT-S/16	Kinetics	62.0	60.3	63.7	37.3	47.0	76.1
SiamMAE (ours)	ViT-B/16	Kinetics	62.8	60.9	64.6	38.4	47.2	76.4
Dino [17]	ViT-S/8	ImageNet	69.9	66.6	73.1	39.5	56.5	80.3
SiamMAE (ours)	ViT-S/8	Kinetics	71.4	68.4	74.5	45.9	61.9	83.8
SiamMAE (ours)	ViT-B/8	Kinetics	72.3	68.9	75.6	46.5	62.4	84.0
Table 1: Comparison with prior work on three downstream tasks: video object segmentation (DAVIS-2017 [95]), human pose propagation (JHMDB [96]) and semantic part propagation (VIP [97]).
and horizontal flipping. Training is done for 400 epochs for the ablation studies (Table 2, 3) and for 2000 epochs for the results in Table 1. We adopt repeated sampling factor [99, 27] of 2 and report "effective epochs", i.e., the number of times a training video is viewed during training. We use the AdamW optimizer [100] with a batch size of 2048. Additional training details are in § A.
Evaluation methodology. We evaluate the quality of learned representations for dense correspondence task using  $k$ -nearest neighbor inference on three downstream tasks: video object segmentation (DAVIS-2017 [95]), human pose propagation (JHMDB [96]) and semantic part propagation (VIP [97]). Following prior work [14-16], all tasks are formulated as video label propagation: given the ground-truth label for the initial frame, the goal is to predict the label for each pixel in future frames of a video. We also provide temporal context during inference by maintaining a queue of the last  $m$  frames, and we limit the set of source patches considered to a spatial neighborhood of the query patch. See § A for evaluation hyperparameters.
4.2 Comparison with Prior Work
Video Object Segmentation. We first evaluate our model on DAVIS 2017 [95], a benchmark for video object segmentation, for the task of semi-supervised multi-object segmentation. We follow the evaluation protocol of prior work and use images of a  $480\mathrm{p}$  resolution for evaluation. We find that SiamMAE significantly outperforms VideoMAE (39.3% to 62.0%), which we attribute to the use of tube masking scheme in VideoMAE which prevents the model from learning temporal correspondences. Like DINO [17], we also find that reducing the patch size leads to significant performance gains. Our ViT-S/8 (+9.4%) model outperforms all prior contrastive learning and self-supervised correspondence learning approaches. Finally, we note that although the larger MAE-ST models (ViT-L/16, 304M parameters) trained with random masking perform better than VideoMAE, their performance still lags SiamMAE by a considerable margin. Surprisingly, we find that MAEs trained on videos perform similarly to image MAEs. Unlike images, which are (approximately) isotropic [29], the temporal dimension is special [30], and not all spatio-temporal orientations are equally likely. Hence, symmetrically treating spatial and temporal information might be sub-optimal.
Video Part Segmentation. Next, we evaluate SiamMAE on the Video Instance Parsing (VIP) [97] benchmark, which involves propagating semantic masks for 20 different human parts. Compared to other datasets in our evaluation, VIP is especially challenging, as it involves much longer videos (up to 120 seconds). We follow the evaluation protocol of prior work [12], using  $560 \times 560$  images and a single context frame. On this challenging task, our ViT-S/8 model substantially outperforms DINO (39.5 to 45.9). SiamMAE benefits more from smaller patch sizes than DINO, achieving an  $+8.6$
encoder	decoder	mask ratio	J&Fm	Jm	Fm
joint	joint	0.50 (s)	51.8	50.7	52.9
joint	joint	0.75 (s)	55.4	54.3	56.6
joint	joint	0.90 (s)	51.9	50.8	52.9
siam	cross-self	0.95 (a)	58.1	56.6	59.6
(a) FrameMAE. Simple extension of MAEs to frames does not work.
encoder	decoder	J&Fm	Jm	Fm
joint	joint	49.7	48.0	51.5
joint	cross	44.6	43.6	45.7
joint	cross-self	41.1	39.6	42.7
siam	joint	56.7	55.4	58.1
siam	cross	52.2	51.2	53.1
siam	cross-self	58.1	56.6	59.6
(b) Encoder-decoder design. The combination of a siamese encoder and a cross-self decoder works the best.
mask ratio	pattern	J&Fm	Jm	Fm
0.50 (s)	random	41.5	40.2	42.7
0.50 (s)	grid	48.2	46.7	49.7
0.75 (s)	random	52.7	51.3	54.1
0.90 (s)	random	51.4	50.0	52.8
0.95 (a)	random	58.1	56.6	59.6
(c) Symmetric masking. Symmetric random masking degrades performance.
mask ratio	J&Fm	Jm	Fm
0.50 (a)	49.0	48.4	49.6
0.75 (a)	55.3	54.1	56.4
0.90 (a)	58.4	57.0	59.8
0.95 (a)	58.1	56.6	59.6
(d) Asymmetric masking. Extremely high asymmetric masking is essential.
Table 2: SiamMAE ablation experiments on DAVIS [95] with the default setting: siamese encoder, cross-self decoder, asymmetric (a) masking ratio  $(95\%)$ , and a frame sampling gap of  $[4 - 48]$ . Default settings are marked in blue and (s) denotes symmetric masking.
mIoU improvement over DINO's  $+3.3$  mIoU. Finally, SiamMAE outperforms all prior contrastive learning and self-supervised correspondence learning approaches.
Pose Tracking. We evaluate SiamMAE on the task of keypoint propagation, which involves propagating 15 keypoints and requires spatially precise correspondence. We follow the evaluation protocol of prior work [12], using  $320 \times 320$  images and a single context frame. SiamMAE outperforms all prior work and benefits more from smaller patch sizes than DINO (+14.9 to +10.9 PCK@0.1).
Finally, we test the scalability of SiamMAE by training and evaluating ViT-B models. Across all three tasks, ViT-B models outperformed ViT-S models for both patch sizes tested.
We ablate SiamMAE to understand the contribution of each design decision with the default settings: siamese encoder, cross-self decoder, asymmetric masking ratio (95%), frame sampling gap 4 - 48.