Siamese Masked Autoencoders
Authors: Agrim Gupta
Jiajun Wu
Jia Deng
Li Fei-Fei
Venue: NeurIPS 2023 oral (2023)
Keywords: Representation Learning
Visual Correspondence
Self-supervised learning
Videos
TL;DR: We propose Siamese Mased Autoencoders to learn visual correspondence from videos.

Abstract: Establishing correspondence between images or scenes is a significant challenge in computer vision, especially given occlusions, viewpoint changes, and varying object appearances. In this paper, we present Siamese Masked Autoencoders (SiamMAE), a simple extension of Masked Autoencoders (MAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them. These frames are processed independently by an encoder network, and a decoder composed of a sequence of cross-attention layers is tasked with predicting the missing patches in the future frame. By masking a large fraction (95%) of patches in the future frame while leaving the past frame unchanged, SiamMAE encourages the network to focus on object motion and learn object-centric representations. Despite its conceptual simplicity, features learned via SiamMAE outperform state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks. SiamMAE achieves competitive results without relying on data augmentation, handcrafted tracking-based pretext tasks, or other techniques to prevent representational collapse.

Siamese Masked Autoencoders
Agrim Gupta' 1+ Jiajun Wu' Jia Deng?
Fei-Fei
*Stanford University, PPrinceton University
Abstract
Establishing correspondence between images or scenes is significant challenge in computer vision, especially given occlusions, viewpoint changes, and varying object appearances. this paper. we present Siamese Masked Autoencoders (SiamMAE), a simple extension of Masked Autoencoders (MAE) for learning visual correspondence from videos. SiamMAE operates on pairs of randomly sam- pled video frames and asymmetrically masks them. These frames are processed independently by an encoder network, and a decoder composed of sequence of cross-attention layers is tasked with predicting the missing patches in the future frame. By masking large fraction (95%) patches in the future frame while leaving the past frame unchanged, SiamMAE encourages the network to focus on object motion and learn object-centric representations. Despite its conceptual sim- plicity, features learned via SiamMAE outperform -fate-of-the-ee art self- methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks. SiamMAE achieves competitive results without relying data augmentation, handcrafted tracking- hased pretext tasks, or other techniques to prevent representational collapse.
Introduction
"The distinction between the past, present, and future is only stubbornly persistent illusion."
-Albert Einstein
Time is a special dimension in the context visual learning, providing the structure within which sequential events are perceived, cause-effect relationships are Icarned, objects are tracked they move through space, and future events are predicted. Central to all these capabilities is the ability to establish visual correspondence over time. Our visual system is adept at establishing correspondence between scenes despite occlusions, viewpoint changes, and object transformations. This capability is unsupervised, critical to human visual perception, and remains significant challenge in computer vision. Equipping machines with such a capability enables wide range of applications such as object segmentation and tracking in videos, depth and optical flow estimation, and 3D) reconstruction [1-8].
powerful sulf-supervised learning paradigm predictive learning, i.e., predicting any unobserved or hidden part of the signal from any observed or unhidden part of the signal [9]. Notably, this form of predictive learning has been used for learning correspondences [10-12] by predicting the colors of grayscale future frame by observing (colorful) past reference frame. However, the performance of these methods has trailed behind contrastive self-supervised learning [13] approaches. State-of- the-art methods [14-17] for learning correspondence primarily employ some form of contrastive learning [13]. Intuitively, contrastive learning based approaches are well-suited for the task of learning correspondence,
they utilize extensive data augmentation to learn features invariant to changes in pose, lighting, viewpoint, and other factors. However, a major criticism of contrastive approaches is their reliance careful selection of augmentations learn useful invariances [18), along with a suite additional components [19, 20, 17, 21] to prevent representational collapse.
"Correspondence agrim@stanfonoooo edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

frame
patchify
k
Encoder
Decoder
Encoder
target
frame
patchify mask
Figure I: Siamese Masked Autoencoders. During pre-trae pre-training we randomly sample a pair of video frames and randomly mask a huge fraction (95%) of patches of the future frame while leaving the past frame unchanged. The two frames are processed independently by a siamese encoder parametrized by ViT [31]. The decoder consists of a sequence of cross-attention layers and predicts missing patches in the future frame. Vidcos available at this project page.
Recently, predictive learning methods like masked language modelling [22, 23] and masked visual modeling (MVM) [24-26] have demonstrated promising results in natural language processing and computer vision domains. MVM methods like Masked Autoencoders (MAE) learn good visual representations without relying data augmentation by learning to reconstruct the missing patches from randomly masked input image patches. Extending MVM methods from images to videos for learning correspondence is however nontrivial for two reasons. First, features learned by MAEs are specialized for the pixel reconstruction task, which show excellent downstream performance on finetuning, but do not transfer well in zero-shot settings. Second, existing extensions of MAEs in the video domain [27, 28] also symmetrically mask a huge fraction of patches across all frames. Unlike images, which are (approximately) isotropic [29]. the temporal dimension special [30], and all spatio-temporal orientations are equally likely. Hence, symmetrically treating spatial and temporal information might be sub-optimal. Indeed, MAES trained on videos do not outperform MAEs trained on ImageNet on video instance tracking benchmarks (Table 1).
To address these limitations, we present Siamese Masked Autoencoders (SiamMAE): simple extension of MAES for learning visual correspondence from videos. In our approach, two frames randomly selected from video clip, with the future frame having a significant portion (95%) of its patches randomly masked, while the past frame is left intact. These frames are processed independently by an encoder network, and decoder composed of sequence of cross-attention layers is tasked with predicting the missing patches in the future frame. Our asymmetric masking approach encourages the network model object motion, or in other words, to understand what went where [32]. Simple extensions of MAES to frames with symmetric masking wastes model capacity on modeling lewelevel image details. However, by providing the entire past frame as input, our network is primarily focused on propagating the patches from the past frame to their corresponding locations in the future frame. The cross-attention layers our decoder serve function akin to the affinity matrix often employed in self-supervise correspondence learning approaches. Empirically, we find that the combination of asymmetric masking, a siamese encoder, and our decoder can effectively learn features suitable for tasks requiring fine-grained and object-level correspondence.
Despite the conceptual simplicity of our method, it outperforms state-of-the-ar self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation. Moreover, our ViT-S/16 models significantly outperform larger models trained on ImageNet (+8.5% J&FF.. for ViT-B) and Kinetics-4N0 400 (+7.4% J&FFm for ViT-L) via MVM in video object segmen- tation tasks. We also observe significant performance gains across all tasks with models trained with smaller patch sizes (ViT-S/8, VIT-B/8). SiamMAE achieves competitive results without rely- ing on data augmentation [16, 17]. handcrafted tracking-based pretext tasks [15, 14], multi-crop training [17], additional techniques to prevent representational collapse [10, 11, 17, 16] or enhance performance [11] []. We believe that our detailed analysis, straightforward approach, and state-of-the-art performance serve robust bascline for self-supervised correspondence learning.

---

2 Related Work
Temporal correspondence. The visual world is smooth and continuous [33, 34), providing rich source of information for biological and machine vision systems. In biological vision, infants learn about objects and their properties by establishing temporal correspondence, taking advantage of the machine vision, learning fine-grained correspondence from video frames is important problem that has been studied for decades in the form of optical flow and motion estimation [36-42, 5, 6, 43, 44, 7]. However, despite their impressive performance, these methods rely on costly suman-annotail or synthetic data with pixel-level ground truth annotations [45, 46]. A more semantically meaningful task involves determining object-level correspondence i.e., visual object tracking [47-52]. One popular approach is methods that utilize deep features learned via supervised [53, 54] or self-supervised learning (10- 12, 14-16] videos. tfate-of-theeart methods [14-17] for self-supervised feature learning for correspondence primarily employ some form of contrastive learning [13]. Predictive learning has also been used for learning correspondences [10, 11] by predicting the target colors for gray-scale input frame by observing a colorful reference frame. However, the performance of these methods has trailed behind contrastive approaches. In this work, we show that predictive learning based methods can be used for learning fine-grained and object-level correspondence.
inherent smoothness in the visual input [35]. Similarly,
Self-supervised visual representation learning, Self-supervised learning is a way learn gen- cralizable visual representations often using different pretext tasks for pre-training [55-59]. The presence of temporal information in videos has been leveraged in numerous ways for representation learning, including future prediction [60-62, 62-64, 10, 65], temporal ordering [66-70], object motion [71, 72, 58, 14], and temporal coherence [73, 74]. Recently, the community has made great progress in superviedd representation learning via contrastive learning [75, 13). Contrastive meth- ods in the image domain encourage models to learn representations by modeling image similarity and dissimilarity [76. 19, 17, 20] or only similarity [77, 21J. Furthermore, contrastive learning has also been successfully applied to videos [78-82, 82, 83). However, limitation of contrastive approaches is their dependence on careful selection of augmentations to learn useful invariances
[18], and as well as the need for a suite of additional components [19, 20, 17, 21] to prevent representational collapse.
Masked autoencoders. Masked autoencoders are a type of denoising autoencoder [84] that learn representations by reconstructing the original input from corrupted (i.e., masked) inputs. The introduction of masked language modeling in BERT [85] has had a transformative impact on the natural language processing field, particularly when scaled to large datasets and model sizes [86, 87]. Masked autoencoders have also been successfully adapted to learn representations from images [24- 26] and videos [28, 27]. Our work studies a simple extension of MABs [24] videos. However, unlike prior methods [28, 27] that symmetrically mask all frames, we propose an asymmetric masking scheme, leaving the past frame unchanged and masking a higher percentage of the future frame.
Siamese networks. Siamese networks [88] are weight-sharing neural networks used to com- pare entities. They have been widely used in variety of application domains [88-90] including tracking [53]. Siamese networks have been extensively used in modern contrastive learning ap- proaches [76, 19, 17, 20], their design allows easy way to learn invariant visual representations from data. Inspired by the success of masked autoencoders, researchers have also explored combining contrastive learning with siamese networks and masked visual modeling [91, 92]. However, we are not aware of any previous studies that have investigated siamese masked autoencoders using asymmetric masking for representation learning from videos.
3 Method
Our goal is to develop a self-supervised method for learning correspondence. To that end, we study simple extension of MAE [24] to video data (Fig. 1). In this section, we describe the key components of our Siamese Masked Autoencoder.
Patchify. Given a video clip with L frames, we first randomly sample frames fi and f2. The distance between these two frames is determined by selecting random value from the predetermined range of potential frame gaps. Following the original ViT [31], we "patchify" each frame by converting it into sequence of non-overlapping N x N patches. Finally, position embeddings [94]
3

---

Figure 2: Visualizations on the Kinetics-400 [93] validation set (masking ratio 90%). For each video sequence, we sample clip of 8 frames with frame gap of 4 and show the original video (top), SiamMAE output (middle), and masked future frames (bottom). Reconstructions are shown with fi the first frame of the video clip and f2 the remaining frames, using SiamMAE pre-trained ViT-S/8 encoder with masking ratio of 95%.
added to the linear projections [31] of the patches, and [CLS] token is appended. We do not use any temporal position embeddings.
Masking. Natural signals like images and videos are highly redundant, exhibiting spatial and spatio- temporal redundancies, respectively [33, 34]. To create challenging predictive self-supervised learning task, MAEs randomly mask a high percentage (75%) of image patches [24] and extensions to videos [28, 27] use an even higher masking ratio (90%). both images and videos, the masking strategy is symmetric, i.e., all frames have a similar masking ratio. This deliberate design choice prevents the network from leveraging and learning temporal correspondence, leading to sub-optimal performance on correspondence learning benchmarks.
We posit that asymmetric masking can create challenging self-supervised learning task while encouraging the network to learn temporal correlations. Specifically, we do not mask any patches in fi (0%) and mask very high ratio (95%) of patches in fz. By providing the entire past frame as input, the network only needs to propagate the patches from the past frames to their appropriate locations in the future frame. This, in turn, encourages the network to model object motion and focus on object boundaries (Fig. 5). To further increase the difficulty of the task, we sample the two frames with large temporal gap. Although predicting further into the future inherently ambiguous and may yield multiple plausible outcomes, providing small number of patches as input for the second frame results in a challenging yet tractable eel-ssupeeeieed learning task.
Encoder. We explore two different encoder configurations for processing input frames.
A joint encoder is natural extension of image MAEs to pair of frames. The unmasked patches from the two frames are concatenated and then processed by a standard ViT encoder.
A siamese encoder [88] are weight-sharing neural networks used for comparing entities and are an essential component of modern contrastive representation learning methods [21]. Siamese networks have been used for correspondence learning [53, 11, 10] and often require some information bottleneck to prevent the network from learning trivial solutions. For example, Lai and Xie [11] propose to use color channel dropout to force the network to avoid relying colors for matching correspondences.

---

Object Propagation
Pose Propagation
50%
LOOKN
TC
25%
50%
LO0%
2.596
100%
50
25%
505
Semantic Part Propagation
Ref
000%
255%
5.0%
100%
180%
M
Figure Qualitative results on three downstream tasks: video object segmentation (DAVIS- 2017 (95]), human pose propagation (JHMDB [96]) and semantic part propagation (VIP [97]).
We use siamese encoders to process the two frames independently and our asymmetric masking serves an information bottleneck.
Decoder. The output from the encoder is projected using linear layer and [MASK] tokens with position embeddings are added to generate the full set of tokens corresponding to the input frame. We explore three different decoder configurations which operate on the full set of tokens.
A joint decoder applies vanilla Transformer blocks on the concatenation of full set tokens from both frames. A key downside of this approach is substantial increase in GPU memory requirement, especially when using smaller patch sizes.
A cross-self decoder is similar to the original encoder-decoder design of the Transformer [94] model. Each decoder block consists of a cross -atsenattnntoon layer and self-attention layer. The tokens from fe attend to the tokens from fi via the cros-attention layer and then attend to each other via the self-attention layer. We note that the cross-attention layer is functionally similar to the affinity matrix often used in self-supervised correspondence learning approaches [11, 10].
cross decoder consists of decoder blocks with only cross-attention layers, where tokens from f2 attend to the tokens from f1.
Finally, the output sequence of the decoder used predict the normalized pixel values [24] in the masked patches. 12 loss is applied between the prediction of the decoder and the ground truth.
Experiments
In this section, we evaluate our method three different tasks, compare its performance with prior state-of-the-ard methods, and perform extensive ablation studies of different design choices. For qualitative results, see Fig. Fig. 3, Fig. and videos on our project website.
4.1 Experimental Setup
Backbone. We use the ViT-S/16 model for most of our experiments as is similar to ResNet-50 in terms of the number of parameters (21M vs 23M) and allows for fair comparisons across different self-supervised learning and correspondence learning methods.
Pre-training. Models are pre-trained using Kinetics-400 [93] for self-supervised learning. SiamMAE takes as input pairs of randomly sampled frames (224 x 224) with frame gap ranging from 4 to 48 frames at rate of 30 fps. We perform minimal data augmentation: random resized cropping
5

---

DAVIS
VIP mloU
JHMDB
Method
Backbome
Dataset
J&FF
Im
Fm
PCK @0.1
PCK
Supervised [98]
50
ImageNet
66.0
63.7
68.4
39.5
59.2
78.3
SimSiam [20]
ResNet-50
ImageNet
66.3
64.5
68.2
35.0
58.4
77.5
MoCo [19]
ResNet-50
ImageNet
65.4
63.2
67.6
36.1
60.4
79.3
TimeCycle [14]
ResNet-50
VLOG
40.7
41.9
39.4
28.9
57.7
78.5
UVC [12]
ResNet-50
Kinetics
54.5
58.1
34.2
VFS [16]
ResNet-550
Kinetics
68.9
66.5
71.3
43.2
60.9
80.7
MAE-ST [27]
ViT-L/16
Kinetics
54.6
55.5
53.6
33.2
44.4
72.5
MAB [24]
VTT-B/16
ImageNet
53.5
52.1
55.0
28.1
44.6
73.4
VideoMAE [28]
VIT-S/16
Kineties
39.7
38.9
Dino [17]
VIT-S/16
ImageNet
61.8
60.2
63,4
36.2
45.6
75.0
SiamMAE (ours)
VIT-S/16
Kineties
62.0
60.3
37.3
47.0
76.1
SiamMAE (ours)
VIT-B/16
Kinetics
62.8
60.9
64.6
38.4
47.2
76.4
Dino [17]
VIT-S/8
ImageNet
69.9
66.6
73.1
39.5
56.5
80.3
SiamMAE (ours)
VIT-S/8
Kinetics
71.4
68.4
74.5
45.9
61.9
83.8
SiamMAE (ours)
ViT-B/8
Kinetics
72.3
68.9
75.6
46.5
62.4
84.0
Table 1: Comparison with prior work on three downstream tasks: video object segmentation (DAVIS-2017 [95]), human pose propagation (JHMDB [96]) and semantic part propagation (VIP [97]).
and horizontal flipping. Training done for 400 epochs for the ablation studies (Table 2, 3) and for 2000 epochs for the results in Table 1. We adopt repeated sampling factor [99, 27] of 2 and report "effective epochs", i.c., the number of times a training video is viewed during training. We use the AdamW optimizer [100] with a batch size of 2048. Additional training details are in $ A.
Evaluation methodology. We evaluate the quality of learned representations for dense correspon- dence task using k-nearest neighbor inference on three downstream tasks: video object segmenta- tion (DAVIS-2017 [95]). human pose propagation (JHMDB [96]) and semantic part propagation (VIP [97]). Following prior work [14-16], all tasks are formulated as video label propagation: given the ground-truth label for the initial frame, the goal frames of video. We also provide temporal context during inference by maintaining a queue of the last m frames, and we limit the set of source patches considered spatial neighborhood of the query patch. See A for evaluation hyperparameters.
predict the label for each pixel future
4.2 Comparison with Prior Work
Video Object Segmentation. We first evaluate our model on DAVIS 2017 [95], benchmark for video object segmentation, for the task of semi-supervised multi-object segmentation. We follow the evaluation protocol of prior work and use images of 480p resolution for evaluation. We find that SiamMAE significantly outperforms VideoMAE (39.3% 62.0%). which we attribute to the use of tube masking scheme in VideoMAE which prevents the model from learning temporal correspondences. Like DINO [17], we also find that reducing the patch size leads significant performance gains. Our VIT-S/8 (+9.4%) model outperforms all prior contrastive learning and self- supervised correspondence learning approaches. Finally, we note that although the larger MAE-ST models (ViT-1/16, 304M parameters) trained with random masking perform better than VideoMAF, their performance still lags SiamMAE by a considerable margin. Surprisingly, we find that MAEs trained videos perform similarly to image MAEs. Unlike images, which (approximately) isotropic [29], the temporal dimension is special [30], and not all spatio-tempont orientations are equally likely. Hence, symmetrically treating spatial and temporal information might be sub-optimal.
Video Part Segmentation. Next, we evaluate SiamMAE on the Video Instance Parsing (VIP) [97] benchmark, which involves propagating semantic masks for 20 different human parts. Compared to other datasets in our evaluation, VIP is especially challenging, as it involves much longer videos (up to 120 seconds). We follow the evaluation protocol of prior work [12], using 560 560 images and single context frame. this challenging task, our ViT-S/8 model substantially outperforms DINO (39.5 to 45.9). SiamMAE benefits more from smaller patch sizes than DINO, achieving +8.6

---

encoder decoder
J&F
Jm
Fm
joint joint
49.7
48.0
51.5
encoder decoder
mask ratio J&FF
Im
Fm.
joint cross
44.6
43.6
45.7
joint joint
0.50 (8)
51.8
50.7
52.9
joint cross-self
41.1
39.6
42.7
joint joint
0.75 (s)
55.4
54.3
56.6
siam
joint
56.7
55.4
58.1
joint
joint
0.90 (s)
51.9
50.8
52.9
siam cross
52.2
51.2
53.1
siam cross-self
0.95 (a)
58.1
56.6
59.6
siam
cross-sclf
58.1
56.6
59.6
(a) FrameMAE. Simple extension of MAEs to frames does not work.
(b) Encoder-decoder design. The combina-
tion of a siamese encoder and cross-self de- coder works best.
mask ratio
pattern
J&FF
Im
Fm
0.50 (s)
random
41.5
40.2
42.7
mask ratio J&FF
Im
F
0.50
grid
48.2
46.7
49.7
0,50 (a)
49.0
48.4
49.6
0.75
random
52.7
51.3
54.1
0.75 (a)
55.3
54.1
56.4
0.90 (s)
random
51.4
50.0
52.8
0.90 (a)
58.4
57.0
59.8
0.95 (a)
random
58.1
56.6
59.6
0.95 (a)
58.1
56.6
59.6
(c) Symmetric masking. Symmetric random mask-
(d) Asymmetric masking, Extremely high asym- metric masking is essential.
ing degrades performance.
Table SiamMAE ablation experiments on DAVIS [95] with the default setting: siamese encoder, cross-self decoder, asymmetric (a) masking ratio (95%), and frame sampling gap of [4 48).
Default settings marked in blue and (s) denotes symmetric masking.
mloU improvement over DINO's +3.3 mloU. Finally, SiamMAE outperforms all prior contrastive learning and self-supervised correspondence learning approaches.
We evaluate SiamMAE on the task of keypoint propagation, which involves propa- gating 15 keypoints and requires spatially precise correspondence. We follow the evaluation protocol of prior work [12], using 320 x 320 images and a single context frame. SiamMAE outperforms all prior work and benefits more from smaller patch sizes than DINO (+14.9 +10.9 PCK @.1)).
Pose Tracking.
Finally, we test the scalability of SiamMAE by training and evaluating ViT-B models. Across all three tasks, VTT-B models ouiperformed ViT-S models for both patch sizes tested.
4.3 Ablation Studies
We ablate SiamMAE to understand the contribution of each design decision with the default settings: siamese encoder, cross-self decoder, asymmetric masking ratio (95%), frame sampling gap 4 48.
FrameMAE. We compare SiamMAE with FrameMAE (Table 2a), an extension MAES to video frames, joint encoder and joint decoder with symmetric masking ratio. FrameMAE performs significantly worse when the masking ratio too high (90%) or too low (50%). With 90% masking ratio, the task becomes challenging (higher loss) due the insufficient number of patches available to learn temporal correspondence. When the masking ratio is 50%, the task becomes easier (lower loss) and the network reconstruct the frames without relying temporal information, due the spatial redundancy of images. SiamMAE with an asymmetric masking ratio works best.
Encoder-decoder design. An important design decision of SiamMAB is the choice of encoder and decoder. We study the performance of various combinations of encoders and decoders with asymmetric masking in Table 2b. Joint encoders perform significantly worse compared to their siamese counterparts across all decoder designs. This can be attributed to the difference between the training and testing setups, as each frame is processed independently during the testing phase.
Siamese encoder with cross decoder performs worst among siamese encoders. We also observe that the training loss is higher and the reconstructed frames are spatially incoherent, as all patches from f2 are processed independently. Finally, the combination of a siamese encoder with a cross-self decoder outperforms all other pairings. The cross-attention operation is similar to the affinity matrix used
7

---

frame gap
J&Fm
Im
Fm
spatial
color
J&FF
Im
F
4
55.1
53.5
56.7
56.8
55.5
58.1
56.4
54.9
57.8
58.1
56.6
59.6
16
58.0
56.7
59.4
55.8
54.6
57.0
32
57.7
56.3
59.1
56.7
55.4
57.9
4-48
58.1
56.6
59.6
(a) Data augmentation. SiamMAE requires mini- mal data augmentation.
(b) Frame sampling. Random frame gap works the
best.
Table 3: Data augmentation. We ablate the importance of manual (spatial and color jitter) and natural data augmentation (frame sampling) for learning correspondence via our SiamMAE on DAVIS [95]. The table format follows Table 2.
Object Propagation
Semantic Part Propagation
Pose Propagation
71.0
MF-MMaan
65
63
68.1
69.6
40
39.8
40.6
42.6
60
59.1
59.6
60.1
60.5
61.5
61,5
36,8
60
56.9
59.1
35
33.1
50
55
51.6
54.9
32.0
45
45-42.4
43.1
43.4
45.1
46,2
30
1600
100
Epochs (log-scale)
Epochs (log-scale)
Epochs (log-scale)
VIT-S/16
VIT-5/8
Figure 4: Training schedule and patch size. Evaluation of SiamMAE performance for downstream tasks for ViT-S/16 and ViT-S/8 models. Across all tasks, longer training and smaller patch sizes lead to improved performance.
in self-supervised correspondence learning and is also used for label propagation in our evaluation protocol. Hence, by processing the frames independently and decoding them via the cross-self decoder, the network is encouraged to learn good representations for dense visual correspondence.
Masking. Next, we discuss the effect of the masking scheme for the combination of siamese encoder with a self-crross-eccoder. Random symmetric masking performs poorly and is also worse than the corresponding FramcMAA configurations (Table 2a, 2c). We also study the grid-wise mask sampling strategy, which keeps every alternate patch. This is an casier task, as the masking pattern enables the network to exploit and learn spatio-temporal correlations. Although we see significant gains (41.5 to 48.2), performance still significantly poor compared to SiamMAE. In Table 2d, we study the role of different asymmetric masking ratios. We notice clear trend: increasing the masking ratio from 50% to 95% increases the performance (49.0% to 58.1%).
Data augmentation. In Table 3a we study the influence of different data augmentation strategies. Similar to the findings in the image [24] and video [27] domains, we find that SiamMAE does not require extensive data augmentation to achieve competitive performance. Random cropping with a scale range of [0.5, 1] and horizontal flipping works best, and adding color jitter leads to performance degradation. Contrastive methods like DINO show impressive k-NN performance by using extensive data augmentation. In contrast SiamMAE achieves superior results by relying on natural data augmentation available in videos, discussed next.
Frame sampling. Video data a rich source of data augmentation, e.g. variations in pose, lighting viewpoint, occlusions. etc. To effectively leverage this, we study the importance of frame sampling Table 3b. The performance improves as we increase the frame sampling gap. Natural videos frequently exhibit gradual temporal changes; therefore, increasing the frame interval results in more robust natural data augmentation, which turn enhances performance. Our frame sampling strategy is simple and effective: randomly sample frames with a frame gap ranging from 4 to 48 frames.

---

Figure 5: Sell-attention maps. Self-attention maps from a VIT-S/8 model. We examine the self- attention of the [CLS] token the heads of the final layer. Unlike contrastive methods, there is explicit loss function acting on the [CLS] token. These self-attention maps suggest that the model has learned the notion of object boundaries from object motion in videos. See project page for videos.
Training schedule. As noted earlier, our ablations are based 400-epoch pre-training. Figure studies the influence of the training length schedule for V and ViT-S/8 models for the three downstream tasks considered in this work. Due to compute limitations, report evaluations of a single model at different checkpoints. Across both patch sizes and across all tasks, the accuracy improves gradually with longer training.
Prediction target. In Table 5a (see $ B) we study the importance of predicting the future. We consider two additional SiamMAE variations: one where always predict the past frame (fl) and another where the order frame prediction (fl or f2) is randomized. All variations perform reasonably well, with our default setting (i.e., predicting the future) performing the best. We emphasize predicting future behavior due to its natural alignment with most real-world applications, which often necessitate the anticipation prediction of agents' future behavior.
4.4 Attention Map Analysis
In Fig. 5, we visualize the self-attention map of the VIT-S/8 model. We use the [CLS] token as the query and visualize the attention of a single head from the last layer with 720p images from ImageNet. We find that the model attends to the object boundaries. For instance, can clearly delineate iconic objects (such as the sheep in the first row, first column), multiple objects (like the three baseball players in the third row, sixth column), and even when the scene is cluttered (as seen with the bird in the second row, fourth column). While other self-supervised learning approaches [17, 101] have reported emergent object segmentation capabilities, we are unaware of any methods demonstrating an emergent ability to predict object boundaries. This emergent ability unique and surprising since, unlike contrastive learning approaches, no loss function operates on the [CLS] token in SiamMAE (or in MAEs). We attribute the emergence of this ability to our asymmetric masking ratio, which encourages the model to learn about object boundaries from object motion in videos.
4.5 Failure Analysis
We evaluate quality of learnt representations using label propagation and consequently inherit its limitations. Specifically, the inference algorithm lacks semantic understanding. leading to globally inconsistent labels (Fig 6). This limitation can be overcome by fine tuning the learnt representations with task specific architectural changes. Additionally, there are instances where the inference process

---

rsf
pred
pred
(c)
rel
Figure 6: Failure analysis. A key disadvantage of using label propagation is the lack of global semantic understanding of objects. Assigning labels based solely on low-level features lead to globally inconsistent labels, as illustrated by the following examples: (a) a segmentation mask that covers both hands; (b) pose key-point determined using the person's 's hair, rather than their posture; (c) challenges in assigning labels to parts of the object that are occluded in the reference frame; and (d) the inability assign labels to fine object details, such the spokes of a tire.
might miss intricate object details, like the spokes of a tire. While this shortcoming can be mitigated by using a smaller patch size during training and inference, it comes at higher compute cost.
5 Conclusion
In this work, we introduce SiamMAE, simple method for representation learning from videos. Our approach is based on the intuition that the temporal dimension should be treated differently from the spatial dimension. We demonstrate that an asymmetric masking strategy, i.e., masking a high percentage of patches of the future frame while keeping the past frame unchanged, is an effective strategy for learning correspondence. By predicting a majority fraction of the future frame, we find that our SiamMAE able to learn the notion of object boundaries (Fig 5). Moreover, unlike MABs, features learned via our approach can be used in zero-shot manner and outperform state-of-the ofth-offtheeart self-super methods in various tasks, such as video object segmentation, pose keypoint propagation, and semantic part propagation. SiamMAE achieves these competitive results without the need for data augmentation, handcrafted tracking-basse based pretext tasks, or other techniques to prevent representational collapse. We hope our work will encourage further exploration of learning representations by predicting the future.
Future work. Our study focuses on learning correspondences by operating on pairs of video frames. This choice was driven by the empirical success of the approach and the limited computational resources available. Consequently, we believe that further investigation is needed to understand the role of predicting multiple future frames based on past frames, both for general visual representation learning and for correspondence learning specifically. An important future direction is to systemati- cally examine the scalability of our approach in terms of both data and model size. Following previous work, we utilize internet videos for pre-training. However, it is essential to also investigate the impact of different types video data, such as egocentric videos [102] versus "in-the-wild" internet videos. Lastly, our learned representations hold potential for applications involving embodied agents (i.e., robots), as the concept of correspondence could be useful in tasks such as object manipulation, navigation, and interaction within dynamic environments.
We thank Abhishek Kadian for helpful discussions. This research was in part supported by the Stanford Institute for Hluman-Centered Artificial Intelligence (HAT) and ONR MURI N00014-22-1-2740.
Acknowledgments.
10

---

References
[1] Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine- Horning.. A benchmark dataset and evaluation methodology for video object segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 724-732, 2016.
[2] Qiang Wang, Li Zhang. Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: unifying approach. In Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pages 1328-1338, 2019.
[3] Ashutosh Saxena, Jamie Schulte, Andrew Y Ng. et al. Depth estimation using monocular and stereo cues. In IJCAI, volume 7, pages 2197-2203, 2007.
[4] Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. arXiv preprint arXiv:2211.05788 2022.
[5] Alexey Dosovitskiy, Philipp Fischer, Eddy llg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt. Daniel Cremers, and Thomas Brox. Flownet: Learning optical flow with convolutional networks. Proceedings of the IEEE international conference on computer vision, pages 2758 2758-2766, 2015.
[6] Eddy llg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2470, 2017.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Aision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[8] Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cam- bridge university press, 2003.
[9] Self Self-supervised learning: The dark matter of intelligence. https://ai facebook.
/slff-ll---/e supervised sepperiiedd-eernnng cessed: 2023-05-05.
Ac-
[10] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In Proceedings of the European conference computer vision (ECCV), pages 391-408, 2018. 1, 2, 4,
[11] Zihang Lai and Weidi Xie. Self-supervised learning for video correspondence flow. arXiv preprint arXiv: /08005 2019. 2, 2, 4,5
[12] Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming- -suan Yang. Joint-task superviee learning for temporal correspondence. Advances in Neural Informa- tion Processing Systems, 32, 2019. 1, 3, 7,
[13] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 06), volume 2, pages 1735-1742. IEEE, 2006. 1,3
[14] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 2566-2576 2019. 2. 6, 18
[15] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as contrastive random walk. Advances in neural information processing systems, 33: 13:19555-19560, 2020.
[16] Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In Proceedings of the JEFE/CVE International Conference
Computer Vision, pages 10075-10085, 2021. 2, 3, 6, 18

---

[17] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021. 1, 2,
3, 6,9
[18] Tete Xiao, Xiaolong Wang. Alexei Efros, and Trevor Darrell. What should not contrastive in contrastive learning. arXiv preprini arXiv:2008. 2020. 1, 3
[19] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for
unsupervised visual representation learning. In CVPR, pages 9729-9738, 2020. 1,3, 6
[20] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learming, pages 1597-1607. PMLR, 2020. 3, 6
[21] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15750-15758, 2021. 1,3,4
[22] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019. 2
[23] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In NIPS, 2020.
[24] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022. 2, 4, 6, 8, 18
[25] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
[26] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2,3
2022.
[27] Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spatiotemporal learners. arXiv preprint 220.::205.00/13, 2022. 2, 3, 4, 8
[28] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, arXiv preprint arXiv:2203.12602
2022. 2, 3, 4.
[29] Daniel L Ruderman. The statisties of natural images. Network: computation in neural systems, 5(4):517, 1994. 2, 6
[30] Edward H Adelson and James R Bergen. Spatiotemporal energy models for the perception of motion. Josa a, 2(2):284-299, 1985. 2,
[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 2, 3,4
[32] Josh Wills, Sameer Agarwal, and Serge J. Belongie. What went where. In Computer Vision
and Pottern Recognition, 2003. 2
[33] Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):
183, 1954. 3, 4
12

---

[34] Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Annual review of neuroscience, 24(():1193-1216 2001.
[35] Elizabeth Spelke, Peter Vishton, and Claes Von Hofsten. Object perception, object-directed action, and physical knowledge in infancy.". 1995. 3
[36] James J Gibson. The perception of the visual world. 1950.
[37] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17 (1-3):185-203, 1981.
[38] Bruce D Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In IJCA/'81: 7th international joint conference on Artificial intelligence, volume 2, pages 674-679, 1981.
[39] Thomas Brox, Andrés Bruhn, Nils Papenberg, and Joachim Weickert. High accuracy optical flow estimation based on a theory for warping. In Computer Vision- ECCV 2004: 8th European Conference Computer Vision. Prague, Czech Republic, May 11-14, 2004. Proceedings. Part IV 8, pages 25-36. Springer, 2004.
[40] Deqing Sun, Stefan Roth, and Michael J Black. Secrets of optical flow estimation and their principles. In 2010 IEEE computer society conference on computer vision and pattern recognition. pages 2432-2438 IEEE, 2010.
[41] Ce Liu, Jenny Yuen, and Antonio Torralba. Sift flow: Dense correspondence across scenes and its applications. IEEE transactions on pattern analysis and machine intelligence, 33(5): 978-999 2010.
[42] Moritz Menze, Christian Heipke, and Andreas Geiger. Discrete optimization for optical flow. In Pattern Recognition: 37th German Conference, GCPR 2015, Aachen, Germany, October 7-10, 2015, Proceedings 37, pages 16-28. Springer, 2015.
[43] Jia Xu, René Ranftl, and Vladlen Koltun. Accurate optical flow via direct cost volume pro- cessing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1289-1297, 2017.
[44] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. In Proceedings the vision and pattern recognition, pages 8934-8943, 2018. 3
the IEEE conference on computer
[45] D. J. Butler, J. Wulff, G. B. Stanley, and M. Black. naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), editor, European Conf. Computer Vision (ECCV), Part IV, LNCS 7577, pages 61 6-65. Springer- ppringer-Verlag, October 2012.
[46] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1211-1237, 2013.
[47] Ishwar K Sethi and Ramesh Jain. Finding trajectories of feature points monocular image sequence. IEEE Transactions on pattern analysis and machine intelligence, (1):56-73, 1987.
[48] Dorin Comaniciu, Visvanathan Ramesh, and Peter Meer. Kernel-based object tracking. IEEE Transactions pattern analysis and machine intelligence, 25(5):564-577, 2003.
[49] Changjiang Yang. Ramani Duraiswami, and Larry Davis. Efficient mean-shift tracking via new similarity measure. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 005) volume 1, pages 176-183 IEEE, 2005.
[50] Mykhaylo Andriluka. Stefan Roth, and Bernt Schiele. leaee-tracking----eeeeeeee and people-detection- bieptteettiio-b--r-kninnn by-tracking. In 2008 IEEE Conference on computer vision and pattern recognition, pages 1-8. IEEE, 2008
[51] Zdenek Kalal, Krystian Mikolajezyk, and Jiri Matas. Tracking-lamiingddteeee IEEE transactions pattern analysis and machine intelligence, 34(7):1409-1422 2011.
13

---

[52] Philipp Bergmann. Tim Meinhardt, and Laura Taixe Tracking without bells and whistles. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 941-951,
2019. 3
[53] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convolutional stamese networks for object tracking. In Computer Vision-ECCY 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part 14, pages 850-865. Springer, 2016. 3,
[54] Jack Valmadre, Luca Bertinetto, Joao Henriques, Andrea Vedaldi, and Philip HS Torr. End-to- end representation learning for correlation filter based tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2805-2811 2017. 3
[55] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference computer vision, pages (422-1430) 2015.
[56] Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI, pages 69-84. Springer, 2016.
[57] Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Computer Eision-ECCY 2016: 14th European Conference, Amsterdam, The Netherlands, October 14, 2016, Proceedings, Part III 14, pages 649-666. Springer, 2016.
[58] Deepak Pathak, Ross Girshick, Piotr Dollár, Trevor Darrell, and Bharath Hariharan. Learning features by watching objects move. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2701-2710, 2017.
[59] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL https: http:///pppeeeeeeeeeeee wwt.//nee/eeee.....
wwt.:ooruffffermo fwr.m? id=S1v42200
[60] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video representations using Istms. In International conference on machine learning, pages 843-852. PMLR, 2015.
[61] Jacob Walker, Carl Doersch, Abhinav Gupta, and Martial Hebert. An uncertain future: Fore- casting from static images using variational autoencoders. In Computer Vision-ECCH 2016: 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part VII 14, pages 835 851. Springer, 2016.
[62] Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Anticipating visual representations from unlabeled video. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 98-106, 2016. 3
[63] Michael Mathicu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. arXiv preprint 1rX1v:1511.05440, 2015.
[64] William Lotter, Gabriel Kreiman, and David Cox. Deep predictive coding networks for video prediction and unsupervised learning. arXiv preprint arXiv: 1605.0660000004 2016. 3
[65] Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martín, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviee wwt.wiwwwwweee.rreeeeeeee id=QAV2CcLEDh. 3
[66] Ishan Misra, Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Computer Vision-ECCC 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 527-544. Springer, 2016. 3
14

---

[67] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3636-3645, 2017.
[68] Hsin- Ying Lee, Jia- Bin Huang, Maneesh Singh, and Ming- Hsuan Yang. Unsupervised repre- sentation learning by sorting sequences. In Proceedings of the IEEE international conference on computer vision, pages 667-676, 2017.
[69] Donglai Wei, Joseph J Lim, Andrew Zisserman, and William T Freeman. Learning and using the arrow of time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8052-8060, 2018
[70] Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-supervised spatiotemporal learning via video clip order prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10334-10343, 2019.
[71] Pulkit Agrawal, Joao Carreira, and Jitendra Malik. Learning to see by moving. In Proceedings of the IEEE international conference on computer vision, pages 37 37-45, 2015.
[72] Xiaolong Wang and Abhinav Gupta. Unsupervised learning of visual representations using videos. In Proceedings of the IEEE international conference on computer vision, pages 2794 2802, 2015. 3
[73] Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715-770 2002. 3
[74] Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, and Yann LeCun. Unsupervised learning of spatiotemporally coherent metrics. In Proceedings of the IEEE international conference on computer vision, pages 4086-4093, 2015.
[75] Suzanna Becker and Geoffrey E Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355(6356) 1992.
[76] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-pane parameraetrric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2018. 3
[77] lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self sufe-suee learning. Advances in neural information processing systems, 33:21271-21288 2020 3
[78] Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In 2018 IEEE international conference on robotics and automation (ICRA), pages 1134-1141. IEEE, 2018. 3
[79] Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning video representa- tions using contrastive bidirectional transformer. arXiv preprint arXiv: 190c.0906.05 2019.
[80] Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[81] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large- scale study on unsupervised spatiotemporal representation learning. Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 3299-3309 2021.
[82] Adria Recasens, Pauline Luc, Ban-iBap Alayrac, Luyu Wang, Florian Strub, Corentin Tallec, Mateusz Malinowski, Viorica Pătráucean, Florent Altché, Michal Valko, et al. Broaden your views for self-super video learning. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 1255-1265, 2021. 3
15

---

[83] Rui Qian, Tianjian Meng, Boqing Gong, Ming- Hung- Yang, Huisheng Wang, Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2021.
[84) Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning, pages 1096-1103, 2008. 3
[85] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019. 3
[86] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. 3
[87] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019. 3
[88] Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993. 3, 4
[89] Yaniv Taigman, Ming Yang, Marc' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1701-1708, 2014.
[90] Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, al. Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille, 2015. 3
[91] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832 2021.
[92] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In Computer Vision-ECCY 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pages 456-473. Springer, 2022.
[93] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. arXiv preprint arXiv: /055006070500500 2017. 4,5
[94] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017. 5
[95] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Homung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:/704.0000 2017. 5, 6, 7,8
[96] Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael Black. Towards understanding action recognition. In Proceedings of the IEEE international conference on computer vision, pages 3192-3199, 2013. 5,6
[97] Qixian Zhou, Xiaodan Liang, Ke Gong, and Liang Lin. Adaptive temporal encoding network for video instance-level human parsing. In Proceedings of the 26th ACM international conference on Multimedia, pages 1527-1535, 2018. 5,6
[98] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of IEFE conference on computer vision and pattern recognition, pages 770-778, 2016. 6
16

---

[99] Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8129-81.38 -1388... 2020. 6, 18
[100] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. URL https::/oppereviiii net/forum? id-Bkg6Ricqy7. 6, 18
[101] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representations, 2022. URL
net hett/Eorummid-ybbp-e-lll
[102] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. arXiv preprint arXiv:21 10.00755,, 2021. 10
[103] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages 1691-1703, PMLR, 2020. 18
[104] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint
2016.
18
[105] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jía, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.0266 2017. 18
17

---

A Implementation Details
Training. Our training settings follow [24] and we build on the open-source implementation of MAEs btttps://gl thub,
for all our experiments. We use the parameters specified in the original implementation unless specified otherwise in Table 4a. All our experiments are performed on 4 Nvidia Titan RTX GPUs for ViT-S/16 models, and on 8 Nvidia Titan RTX GPUs for ViT-S/8 models and ViT-B models.
Evaluation methodology. Our evaluation methodology follows prior work [14-16] and in Table we report results previously reported in these studies. For recent self-supervised learning approaches like DINO, MAEs, MAE-ST and VideoMAE, we carry out comprehensive grid search on the evaluation hyperparameters listed in Table 4b, and report the optimal results obtained. The evaluation parameters for SiamMAE can be found in Table 4b.
config
value
optimizer
AdamW [100]
optimizer momentum
81, 8209, 0.95 [103]
weight decay
0.05
learning rate
1.5e-4
config
DAVIS
VIP
JHMDB
learning rate schedule warmup epochs [105]
cosine decay [104]
top-k
7
10
7
queue length
20
20
epochs
2000 (ahlations 400)
neighborhood size
20
20
repeated sampling [99]
2 hflip, crop |0.5, 1 2048 (S) 1024 (B)
augmentation
batch size
frame sampling gap
[4,48] 48
(a) Kinetics pre-training setting.
(b) Evaluation setting.
Table 4: Training and evaluation hyperparameters.
B Additional Ablations
Prediction target. In Table 5a we study the importance of predicting the future. We consider two additional SiamMAE variations: one where we always predict the past frame (fl) and another where the order of frame prediction (f1 or f2) is randomized. All variations perform reasonably well, with our default setting (i.e., predicting the future) performing the best. We emphasize predicting future behavior due its natural alignment with most real-world applications, which often necessitate the anticipation or prediction of agents' future behavior.
Frame overlap analysis. To perform frame overlap analysis, sampled video frames from the Kinetics-400 validation set with the specified frame gap and calculated two image similarity metrics: mean squared error (mse) and structural similarity index measure (ssim). We observed that either very high overlap (low frame gap, high ssim, and low mse) or a low overlap (high frame gap. low ssim, and high mse) adversely affects performance. Sampling with frame gap of 16 within range of [4, 48] yields the best results. Interestingly, the overlap metrics for frame gap of 16 and [4, 48) are comparable, suggesting that particular degree of overlap important for best results.
frame gap
ssim
msc
J&F
4
0.6231
0.0230
55.1
pred, target
J&F
Im
Fm
8
0.5343
0.0360
56.4
fl (past)
57.5
56.0
59.0
16
0.4749
0.0480
58.0
random (fl, 12)
57.8
56.3
59.2
32
0.4221
0.0597
57.7
12 (future)
58.1
56.6
59.6
4-48
0.4548
0.0528
58.1
(a) Prediction target. Predicting the future works the best.
(b) Frame sampling. Random frame gap works the best.
Table 5: Additional ablations. We ablate the importance predicting the future and perform overlap analysis for different frame gaps. The table format follows Table 2.
18


Reviews:
[{"review_id": "CkKk5a7mFK", "reviewer": "Reviewer_e5XC", "summary": "This paper considers the problem of using self-supervised learning from video to obtain a representation that is well-suited to the task of estimating correspondence between a pair of images. They propose a significant modification of the MAE training procedure which is adapted for estimating correspondence: one image is not masked at all while the other has most (90%+) of its tokens masked, and the same (i.e. a siamese) encoder is applied to both. This is designed to require the model to internally establish correspondence. The model is trained for pixel prediction on Kinetics-400 and evaluated on several propagation tasks (object mask, part masks, human pose) using kNN inference to establish a dense correspondence field. SiamMAE is shown to greatly outperform existing self-supervised learning procedures with comparable backbone architectures, including methods trained on video. Ablative experiments confirm the importance of combining a siamese encoder with asymmetric masking. Visualization of the attention maps show that the model pays strong attention to object boundaries, seemingly a novel attribute.", "questions": "(Please also address or correct weaknesses above.)\n\n1. What is the purpose of the [CLS] tokens? Is the model much less effective without it? (It is surprising that the attention masks for these tokens were so salient given that they are not involved directly in the loss.)\n\nSuggestions:\n1. I wonder whether it's possible to identify a general principle/strategy for preventing shortcut learning, of which this is just one instance?\n\nMinor edits: (no need to respond)\n1. Bold values in Tables 2d and 3b seem incorrect (e.g. 58.4 > 58.1, 56.7 > 56.5).", "limitations": "Additional limitation:\n1. Even though Kinetics-400 was used without labels, it is possible that its image distribution is quite similar to that of the downstream tasks. It would be good to discuss this potentialityu, and say that the impact of dataset similarity has not been investigated?\n\nI do not see any negative societal impacts stemming from this individual paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Good motivation and contextualization with respect to past work.\n1. The modification of the MAE procedure is simple but clever, and manages to extract much more information for correspondence from video than past methods.\n1. Visualization of predicted images is quite impressive (Figure 2), despite the main goal being to learn a feature extractor for correspondence.\n1. Comprehensive evaluation with 3 different tasks and wide selection of relevant baselines.\n1. Ablative experiments verify the importance of each component of the design.\n1. Hyper-parameters provided for reproducibility.", "weaknesses": "1. I'm not sure about the emphasis on predicting the _future_. It seems that the temporal order could be reversed (i.e. predict the past given the future) or randomized and I would expect similar results. Has this already been tested?\n1. It wasn't abundantly clear how the patch-patch similarity was obtained. It seems to be taken from the cross-attention values within the decoder (line 239). However, this could be more clear since the decoder may contain multiple cross-attention layers with each having multiple heads?\n1. It wasn't clear how k-NN and the queue were used to perform propagation. This should be explained in more detail or a reference provided.\n1. No code provided at this stage.\n1. Lack of confidence intervals (not a major issue - delta is quite large in most cases).\n1. No evaluation of ViT-B and ViT-L (not a major issue - impressive results obtained with smaller model).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. We address the reviewer concerns below:\n\n>*Emphasis on predicting the future*\n\nWe agree with the reviewer that reversing the temporal order should not significantly alter the results. We conducted two additional ablation studies: one where we always predict the past frame (f1) and another where the order of frame prediction (f1 or f2) is randomized. All the settings for the results below follow Table 2.\n\n| prediction target | J & Fm    |\n| ----------------- | --------- |\n| f1 (past)         | 57.5      |\n| random [f1, f2]   | 57.8      |\n| **f2 (future)**   | **58.1**  | \n\nAll settings perform reasonably well, with our default setting (i.e., predicting the future) performing the best. We emphasize predicting future behavior due to its natural alignment with most real-world applications, which often necessitate the anticipation or prediction of agents' future behavior. We will update the text with this discussion and ablation. \n\n>*Patch-patch similarity calculation*\n\nWe follow prior work on representation learning (e.g. MAE) and use the output of the encoder for calculating patch-patch similarity. The decoder is only used during pre-training. We will update the text to clarify the same.\n\n>*Evaluation methodology* \n\nOur evaluation methodology follows prior work and we provide a reference for the same in the text (please see L-189, references 14-16). For completeness we provide a short description here. \n\nAll evaluation tasks are cast as video label propagation, where the goal is to predict labels for each pixel in the target frames of a video, using only the ground-truth of the initial frame (i.e., the source). We measure the cosine similarity of each pixel, or patch, in the target frame with all the patches within its spatial neighborhood from the preceding m frames. The label assignment is then based on the labels of the top-k patches that have the highest similarity. Please note, the term 'queue' in this context refers to the usage of predicted labels from the past m frames.\n\n>*Code release*\n\nWe will release the code and pre-trained checkpoints upon acceptance. For reproducibility we build on the open source implementation of MAE and provide all the relevant hyper parameters in the appendix. \n\n>*Confidence intervals*\n\nOur inference procedure is fully deterministic given a pre-trained model. We confirmed the same by running our inference with 5 different seeds for our ViT-S/16 models trained for 2000 epochs. \n\n| Model    | Run 1     | Run 2     | Run 3     | Run 4     | Run 5     |\n| -------- | --------- | --------- | --------- | --------- | --------- |\n| ViT-S/16 | 62.0      | 62.0      | 62.0      | 62.0      | 62.0      |\n\n>*Larger models ViT-B & ViT-L*\n\nWe address this in our global response and re-iterate here for convenience.\n\nWe note that ViT-S has approximately the same number of parameters as ResNet-50, enabling us to compare our method across a diverse set of baselines. Moreover, ResNet-50 is the largest backbone explored by prior work [12, 14, 16], all of which aimed at improving representations for correspondence. However, we agree that a systematic investigation into the scalability of our method in terms of model size (refer to L297) would be valuable. Practically speaking, we are constrained by resources. For instance, training a ViT-S/8 model on 8 Titan RTX GPUs for 2000 epochs requires approximately 16 days. Given the compute and time constraints, we trained a ViT-B/16 model for 400 epochs. \n\n| Model      | J & Fm    |\n| ---------- | --------- |\n| ViT-S/16   | 58.1      |\n| ViT-B/16   | 58.6      |\n\nThe improvement in performance is encouraging. To contextualize the magnitude of the improvement, we note that scaling DINO from ViT-S/16 to ViT-B/16 results in an improvement of 0.5 J&Fm. \n\n>*Role of [CLS] token*\n\nFollowing the original ViT paper which appends the [CLS] token during supervised training and uses the output corresponding to the [CLS] token for predicting class labels, almost all follow up work on representation learning using ViTs has followed this practice. Here, the role of [CLS] token is similar and is typically used for evaluation of learnt representation via linear probing. MAEs also included [CLS] token, likely to maintain consistency with existing literature and evaluations. In SiamMAE, much like in MAEs, the [CLS] token doesn't play any role during pre-training. This design choice was inherited from both ViT and MAE. Our ablation study shows that while the [CLS] token isn't crucial for achieving good results, omitting it slightly hurts the performance.\n\n| [CLS] token  | J & Fm    |\n| ------------ | --------- |\n|              | 57.5      |\n| &#10003;     | **58.1**  |\n\n>*General principle/strategy for preventing shortcut learning*\n\nWe agree that a general strategy of preventing shortcut learning would be great and is indeed an open question in the field of representation learning for computer vision. \n\n>*Impact of dataset similarity has not been investigated*\n\nWe agree that the role of dataset similarity between the train and test tasks has not been studied and will add this limitation in the text. However, we would like to point out that most of the prior work we compare with (UVC, VFS, MAE-ST and VideoMAE) were trained on the same dataset i.e. Kinetics-400. \n"}, {"review_id": "ujFPN8rAX9", "reviewer": "Reviewer_9jyT", "summary": "This paper focuses on the self-supervised learning for video representations. The proposed SiamMAE operates on pairs of randomly sampled video frames and asymmetrically masks them, and then predicts the missing patches for visual representation learning. SiamMAE achieves significant performance and outperforms state-of-the-art self-supervised methods on video object segmentation, pose keypoint propagation, and semantic part propagation tasks.\n", "questions": "My concerns mainly lie in the experiments. ", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 The motivation is clear and strong.\n\n2 The proposed asymmetric masking and cross-self decoder are effective and achieve good performance.\n", "weaknesses": "1 This paper mainly evaluates the proposed method on tracking problems. How is the performance on video classification tasks, such as UCF101 and HMDB51?\n\n2 In Table 2(c), grid mask achieves better performance than random mask with 0.5 mask ratio. Increasing the mask ratio will improve the performance of random masking. How is the performance when increasing the mask ratio for grid masking?\n\n3 This paper only investigates the ViT-S backbone. It is better to also leverage larger models, such as ViT-B, to verify the effectiveness of the proposed method.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. We address the reviewer concerns below:\n\n>*Results on video classification tasks*\n\nWe address this in our global response and re-iterate here for convenience.\n\nWe agree with the general sentiment of the comment, emphasizing the evaluation of a self-supervised representation learning method across a wide array of tasks. While we aim to demonstrate the versatility of our method, following prior work [12, 14, 16], we've limited our claims and experiments to validate our method's effectiveness at learning representations for visual correspondence. A key advantage of current evaluation strategy is its computational efficiency as it requires no training. Evaluation on video recognition benchmarks often involves fine tuning models for 300 epochs (as seen in VideoMAE, MAE-ST), requiring approximately 32-64 industrial-grade GPUs (such as V100, A100).\n\n>*Role of grid masking*\n\nAs suggested, we conducted additional ablations for grid masking with different masking ratios. We find that the performance increases when we increase the grid masking ratio to 0.75. However, it decreases when we further increase the masking ratio to 0.95. An advantage of grid masking is that the masking pattern encourages the network to exploit spatio-temporal correlations. However, with a very high masking ratio the network can no longer rely on temporal correlations, leading to worse performance. \n\nAll the settings for the results below follow Table 2c.\n\n| mask ratio      | pattern      | J & Fm    |\n| --------------- | ------------ | --------- |\n| 0.50 (s)        | grid         | 48.2      |\n| 0.75 (s)        | grid         | 53.8      |\n| 0.95 (s)        | grid         | 49.0      |\n|  **0.95 (a)**   | **random**   | **58.1**  |\n\n>*Results on larger backbones ViT-B*\n\nWe address this in our global response and re-iterate here for convenience.\n\nWe note that ViT-S has approximately the same number of parameters as ResNet-50, enabling us to compare our method across a diverse set of baselines. Moreover, ResNet-50 is the largest backbone explored by prior work [12, 14, 16], all of which aimed at improving representations for correspondence. However, we agree that a systematic investigation into the scalability of our method in terms of model size (refer to L297) would be valuable. Practically speaking, we are constrained by resources. For instance, training a ViT-S/8 model on 8 Titan RTX GPUs for 2000 epochs requires approximately 16 days. Given the compute and time constraints, we trained a ViT-B/16 model for 400 epochs. \n\n| Model      | J & Fm    |\n| ---------- | --------- |\n| ViT-S/16   | 58.1      |\n| ViT-B/16   | 58.6      |\n\nThe improvement in performance is encouraging. To contextualize the magnitude of the improvement, we note that scaling DINO from ViT-S/16 to ViT-B/16 results in an improvement of 0.5 J&Fm. \n"}, {"review_id": "G3L7I8XZP0", "reviewer": "Reviewer_h4WK", "summary": "* This paper propose Siamese Masked Autoencoders for learning visual correspondence from videos called SiamMAE. \n\n* SiamMAE randomly sample a pair of video frames and randomly mask 95% of patches of the future frame, and the pair of video frames are passed into visual encoder(VIT), and cross attention decoder to reconstruct the target.\n\n* The authors conduct several experiments on downstream tasks(vos, human pose propagation), showing superior performance.", "questions": "please refer to the strength and weakness", "limitations": "no limitations", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper presents a simple, yet highly effective method for the challenging video self-supervised framework.\n\n* The proposed method is well motivated and intuitive with excellent performance.\n\n* The ablation is sufficient and the writing is excellent.", "weaknesses": "* Experiments on video recognition experiments should be reported. \n\n* Although SiamMAE is pretraining on video frames, i still think the authors should conduct experiments on image downstream tasks(such coco detection, segmentation) to show image representation capacity.\n\n* In Table1, The previous video ssl method(VPS) is also pretrained on Kinetics with ResNet50，however，the performance of SiamMAE with ViT-S/8 is similar in DAVIS & VIP & JHMDB, any explanation for this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. We address the reviewer concerns below:\n\n>*Experiments on video and image recognition*\n\nWe address this in our global response and re-iterate here for convenience.\n\nWe agree with the general sentiment of the comment, emphasizing the evaluation of a self-supervised representation learning method across a wide array of tasks. While we aim to demonstrate the versatility of our method, following prior work [12, 14, 16], we've limited our claims and experiments to validate our method's effectiveness at learning representations for visual correspondence. A key advantage of current evaluation strategy is its computational efficiency as it requires no training. Evaluation on video recognition benchmarks often involves fine tuning models for 300 epochs (as seen in VideoMAE, MAE-ST), requiring approximately 32-64 industrial-grade GPUs (such as V100, A100).\n\n>*VFS performance*\n\nVFS is a state-of-the-art contrastive self-supervised representation learning method for visual correspondence. VFS learns representations by maximizing similarity across different frames from the same video. This training objective is directly aligned with how the downstream performance is measured i.e., by calculating the similarity of patches for label propagation. \n\nIn this work, our goal was to develop a predictive learning method which can match or outperform contrastive learning approaches for learning visual correspondence. Historically, the performance of predictive learning methods has trailed behind contrastive self-supervised learning approaches. \nThis can be attributed to the training objective of predictive learning methods like SiamMAE, which focuses on low level pixel details and is not directly aligned with the downstream evaluation procedure.\nDespite this shared disadvantage, we significantly outperform prior predictive learning based methods (+22.7 improvement over VideoMAE). Finally, we achieve our stated goal as the ViT-S/8 model improves over VFS by +2.5 J&Fm on DAVIS, +2.7 mIOU on VIP, and +1.0 PCK\\@0.1 on JHMDB. \n"}, {"review_id": "PQ0PkVnesp", "reviewer": "Reviewer_uZFR", "summary": "The paper proposes to use Siamese Masked Encoders for establishing correspondence for video input data. Uses the concept of predictive learning based on Masked Auto Encoder. Paper proposes to use asymmetric masking for present and future frames. Achieves best results in self-supervised setting for video label propagation tasks.\n", "questions": "Questions:\n1) Ablation experiment: Comparison with FrameMAE. In case of FrameMAE, When we do provide 2 frames to the joint encoder, a better comparison would be comparing the asymmetric masking with FrameMAE, since from Table 2 (b, d) results as presented does not seem too behind for FrameMAE.\n\n3) Section 4.4: This subsection would be more aptly titled as \"Futher Insights\" rather Qualitative Results.", "limitations": "Overall the idea presented in the paper is lucid and necessary details regarding reproducing the experiments are mentioned. I think the paper puts a right step in the direction of improving well know correspondence (object level) problem using predictive learning. However I feel what this paper lacks is drawing a complete comparison for an image overlap percentage based study. Presenting zero-shot results for occluded scenes and object parts and discussing failure cases (as seen from qualitative results obtained)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1) The paper is well written for most of the parts and aptly elucidates the advantage of using a masked auto-encoder-based method for object-based correspondence. \n2) Discusses in detail the architecture design choices for the encoder and decoder, ans uses a final design which is intuitive and simple and  also focuses on relevant ablation studies.\n3) The paper shows and achieves better results for object segmentation, part segmentation, and pose propagation tasks. \n\n", "weaknesses": "Weakness:\n1) Frame sampling: Table 3b discusses the effect of frame gap, however, it misses to comment on what could be the minimum frame overlap between the pair of frames for which predictions are out to be done. Apart from asymmetric masking, the degree of overlap between consecutive frames is also an important factor.\n\n2) Fails to discuss the probable failure cases given the limitations from more qualitative results analysis belonging to various tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. We address the reviewer concerns below:\n\n>*Frame sampling with overlap analysis*\n\nTo perform overlap analysis, we sampled video frames from the Kinetics-400 validation set with the specified frame gap and calculated two image similarity metrics: mean squared error (mse) and structural similarity index measure (ssim). We observed that either a very high overlap (low frame gap, high ssim, and low mse) or a low overlap (high frame gap, low ssim, and high mse) adversely affects performance. \nSampling with a frame gap of 16 or within a range of [4, 48] yields the best results. Interestingly, the overlap metrics for a frame gap of 16 and [4, 48] are comparable, suggesting that a particular degree of overlap is important for best results. \n\n\n| frame gap  | ssim     | mse     | J & Fm    |\n| ---------- | -------- | ------- | --------- |\n| 4          | 0.6231   | 0.0230  | 56.7      |\n| 8          | 0.5343   | 0.0360  | 57.8      |\n| 16         | 0.4749   | 0.0480  | 58.0      |\n| 32         | 0.4221   | 0.0597  | 56.3      |\n| 4-48       | 0.4548   | 0.0528  | 58.1      |\n\n>*Qualitative analysis & discussion of failures*\n\nWe've included a qualitative failure analysis in a file attached to the global response and will incorporate it into the paper.\n\nWe evaluate the quality of learnt representations using label propagation and consequently inherit its limitations. Specifically, the inference algorithm lacks semantic understanding, leading to globally inconsistent labels (refer to examples in the figure). This limitation can be overcome by fine tuning the learnt representations with task specific architectural changes. Additionally, there are instances where the inference process might miss intricate object details, like the spokes of a tire. While this shortcoming can be mitigated by using a smaller patch size during training and inference, it comes at a higher compute cost.\n\n>*Comparing asymmetric masking with FrameMAE*\n\nThe results in Table 2b include the comparison suggested i.e. FrameMAE with asymmetric masking. We will update the text to clarify this. For completeness the table below compares relevant FrameMAE variations with SiamMAE. The combination of siamese encoder, a cross-self decoder with asymmetric masking works the best. In the table below (a) denotes asymmetric masking and (s) denotes symmetric masking. \n\n| encoder     | decoder        | mask ratio   |  J & Fm   |\n| ----------- | -------------- | ------------ | --------- |\n| joint       | joint          | 0.50 (s)     | 51.8      |\n| joint       | joint          | 0.75 (s)     | 55.4      |\n| joint       | joint          | 0.90 (s)     | 51.9      |\n| joint       | joint          | 0.95 (a)     | 49.7      |\n| **siam**    | **cross-self** | **0.95 (a)** | **58.1**  |\n\n>*Change title of section 4.4 to Further Insights*\n\nWe will update the text to incorporate the suggestion."}, {"review_id": "C0MbuOa97j", "reviewer": "Reviewer_5RkZ", "summary": "In this paper, the authors propose a simple extension to Masked Autoencoders (MAE) to be able to pre-train on videos: SiamMAE. Two frames are sampled, independently encoded, and then asymmetrically masked. A transformer decoder is used to predict the missing patches in the masked image. The authors show that by masking a high proportion of patches (0.95) in the future frame and leaving the past frame unmasked, they are able to encourage the network to learn a more object-centric representation and focus on object motion rather than low-level image details. The authors show that this simple approach outperforms previous methods on many down-stream tasks and perform an extensive ablation to examine the architecture choices.", "questions": "With standard MAE, it is possible to also mask the input images for the downstream tasks to speed-up computation. It seems that this avenue for speed-up would not be possible with this current approach since the previous frame has to be unmasked?\n\nCould the authors clarify the reason for not including temporal positional encoding (was this ablated?)\n\nIt would be interesting to see a bit more how this frame-based method compares to video-based methods on downstream tasks (for example more comparisons like VideoMAE). Perhaps focusing on compute efficiency?\n", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Very well written paper. Some results are particularly impressive (e.g. more than 20% gain over VideoMAE)\n\nSimple but effective method and the honesty of presenting this as-is (instead of disguising the method as being more complicated than it needs to be) I think should be appreciated.\n\nGood comparison to other work and a great ablation section explaining many design choices\n", "weaknesses": "Can only see two minor weaknesses:\n\n1.\tThe approach adds little over MAE and feels incremental, however it works very well and hasn’t been done before.\n2.\tThis approach relies on the temporal smoothness that is found in many curated datasets, such as Kinetics. However, for in-the-wild videos, with many sharp scene changes, this assumption becomes less likely to hold. However, it should be possible to split long videos into scenes in an unsupervised manner and then sample frames within a scene. It would be nice to a discussion on how this can be applied to large-scale internet datasets (which are becoming very popular for foundation models)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and suggestions. We address the reviewer concerns below:\n\n>*Novelty over MAE*\n\nWe agree that our method is a simple modification of MAE albeit one which has not been explored in the past.  We hope that the simplicity, efficacy and extensive empirical analysis of our method is a valuable contribution to the community.  \n\n>*Application to “in-the-wild” datasets*\n\nPredicting the future frame based on the past frame to learn correspondence is effective because the two frames often have some degree of overlap. We consider a maximum frame gap of 48 frames, equivalent to 1.6s at 30fps. Over such short time horizons, temporal smoothness is a reasonable assumption. However, we agree that \"in-the-wild\" datasets, particularly egocentric datasets like Ego4D, might have frequent sharp scene changes due to head movements. Exploring these datasets is a promising future direction (refer to L298), especially when examining if the current strategy of random sampling remains effective. We will include this discussion in the revised text.\n\n>*Masking input images for downstream tasks to speed-up computation*\n\nTo the best of our knowledge the masking strategy of MAEs is known to speed up the training process as the encoder acts on a small set of unmasked tokens. During inference generally the entire image is processed by the encoder [1, 2]. It is possible that some works have done inference on masked images however we are not aware of such works. \n\n[1] He, Kaiming, et al. \"Masked autoencoders are scalable vision learners.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Li, Yanghao, et al. \"Scaling language-image pre-training via masking.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n>*Reason for no temporal position encoding*\n\nTemporal position embeddings (TPE) enable the network in distinguishing between tokens from distinct frames. In SiamMAE, the encoder processes each frame individually and the sole interaction between tokens of different frames occurs through the cross-attention layer in our cross-self decoder. Hence, the network does not need TPEs for achieving good performance. We validated this hypothesis by conducting the proposed ablation study, adding TPEs to the decoder. When using a joint decoder, TPEs improve the performance. Ideally, if position embeddings are redundant, the network should achieve similar performance with zero embedding weights. However, in practice we found that when using a cross-self decoder, the presence of TPEs hurts the performance.\n\n\n| encoder     | decoder        | temporal pos. embed  |  J & Fm   |\n| ----------- | -------------- | :------------------: | --------- |\n| siam        | joint          | &#10003;             | 57.3      |\n| siam        | joint          |                      | 56.7      |\n| siam        | cross-self     | &#10003;             | 57.6      |\n| **siam**    | **cross-self** |                      | **58.1**  |\n\n>*Comparison on additional downstream tasks*\n\nWe address this in our global response and re-iterate here for convenience.\n\nWe agree with the general sentiment of the comment, emphasizing the evaluation of a self-supervised representation learning method across a wide array of tasks. While we aim to demonstrate the versatility of our method, following prior work [12, 14, 16], we've limited our claims and experiments to validate our method's effectiveness at learning representations for visual correspondence. A key advantage of current evaluation strategy is its computational efficiency as it requires no training. Evaluation on video recognition benchmarks often involves fine tuning models for 300 epochs (as seen in VideoMAE, MAE-ST), requiring approximately 32-64 industrial-grade GPUs (such as V100, A100).\n"}]

Author rebuttal:
We sincerely thank the reviewers for their thoughtful and constructive feedback. We are glad that **all** the reviewers found our method simple, clever and intuitive, our analysis and ablations to be thorough and convincing, and results impressive. The main aim of this rebuttal is to improve this work further by incorporating reviewer suggestions and comments. Specifically, we have conducted all additional ablations and experiments suggested by the reviewers. Here we comment on the feasibility and scope of two suggested avenues of improvement and provide some encouraging preliminary results. 

>*Training larger backbones like ViT-B & ViT-L*

We note that ViT-S has approximately the same number of parameters as ResNet-50, enabling us to compare our method across a diverse set of baselines. Moreover, ResNet-50 is the largest backbone explored by prior work [12, 14, 16], all of which aimed at improving representations for correspondence. However, we agree that a systematic investigation into the scalability of our method in terms of model size (refer to L297) would be valuable. Practically speaking, we are constrained by resources. For instance, training a ViT-S/8 model on 8 Titan RTX GPUs for 2000 epochs requires approximately 16 days. Given the compute and time constraints, we trained a ViT-B/16 model for 400 epochs. 

| Model      | J & Fm    |
| ---------- | --------- |
| ViT-S/16   | 58.1      |
| ViT-B/16   | 58.6      |

The improvement in performance is encouraging. To contextualize the magnitude of the improvement, we note that scaling DINO from ViT-S/16 to ViT-B/16 results in an improvement of 0.5 J&Fm. 

>*Evaluation on video action recognition and image recognition tasks*

We agree with the general sentiment of the comment, emphasizing the evaluation of a self-supervised representation learning method across a wide array of tasks. While we aim to demonstrate the versatility of our method, following prior work [12, 14, 16], we've limited our claims and experiments to validate our method's effectiveness at learning representations for visual correspondence. A key advantage of current evaluation strategy is its computational efficiency as it requires no training. Evaluation on video recognition benchmarks often involves fine tuning models for 300 epochs (as seen in VideoMAE, MAE-ST), requiring approximately 32-64 industrial-grade GPUs (such as V100, A100).

Meta-review: All reviewers find the paper to be novel with impressive experiment results. The rebuttal additionally resolves minor concerns regarding technical details and additional experimental comparisons. The authors are suggested to incorporate constructive feedbacks from the review comments into the finalized version. Based on high quality of this paper and the potential impact to the community, reviewers generally nominated it to be an Oral.
Decision: Accept (oral)