Extreme Masking for Learning Instance and Distributed Visual Representations
Authors: Zhirong Wu
Zihang Lai
Xiao Sun
Stephen Lin
Venue: Accepted by TMLR ()

Abstract: The paper presents a scalable approach for learning spatially distributed visual representations over individual tokens and a holistic instance representation simultaneously. We use self-attention blocks to represent spatially distributed tokens, followed by cross-attention blocks to aggregate the holistic instance. The core of the approach is the use of extremely large token masking (75\%-90\%) as the data augmentation for supervision. Our model, named ExtreMA, follows the plain BYOL approach where the instance representation from the unmasked subset is trained to predict that from the intact input. Instead of encouraging invariance across inputs, learning requires the model to capture informative variations in an image.

The paper makes three contributions: 1) It presents random masking as a strong and computationally efficient data augmentation for siamese representation learning. 2) With multiple sampling per instance, extreme masking greatly speeds up learning and improves performance with more data. 3)  ExtreMA obtains stronger linear probing performance than masked modeling methods, and better transfer performance than prior contrastive models.

Published in Transactions on Machine Learning Research (03/2023)
Extreme Masking for Learning Instance and Distributed Visual Representations
Zhirong Wu'
Zihang Lai
Xiao Sun'
Stephen Lin'
Microsoft Research Asia
Carnegie Mellon University²
Reviewed OpenReview:
Abstract
The paper presents a scalable approach for learning spatially distributed visual representations over individual tokens and holistic inslance representalion simultaneously. We use self- attention blocks to represent spatially distributed tokens, followed by cross-attention blocks to aggregate the holistic image instance. The core of the approach the use of extremely large token masking (75%-90%) as the data augmentation for supervision. Our model, named ExtreMA, follows the plain BYOL approach where the instance representation from the unmasked subset is trained to predict that from the intact input. Instead of encouraging invariance across inputs, the model is required to caplure informative variations in an image.
The paper makes three contributions: 1) It presents random masking as strong and computationally efficient data augmentation for siamese representation learning. 2) With multiple sampling per instance, extreme masking greatly speeds up learning and improves performance with more data. 3) ExtreMA obtains stronger linear probing performance than masked modeling methods, and better transfer performance than prior contrastive models.
1 Introduction
Masked modeling DDelinn et al. 2019) has emerged as viable approach for visual representation learning. On a generie transformer architecture (Vaswani et al. 2017). it optimizes a learning objective based on the masked signal prediction popularized in natural language understanding (Devlin et al. 2019), without reliance on heavily engineered image augmentations (Wu al. 2018 Chen et al. 2020b Grill el al. 2020). Superior
finetuning performance has been demonstrated with this approach; however, the pretrained representation does not work competitively off-tho-shelf (Bao et al. 2021 e.g., for k-nearest-neighhor retrieval.
On the other hand, Siamese networks trained with contrastive objectives (Oord et al. 2018) are strong for learning off- off-the-shelf representations (Radford et al. 2021). This fundamental difference lies in the way they represent data. Siamese networks extract a holistic instance representation for an image, whereas masked modeling acquires spatially distributed representation over individual tokens that comprise an image H No instance representation is explicitly modeled or provided supervision in masked modeling approaches (Devlin et al. 2019 Bao et al.) 2021b.
In this paper, study the connections between the instance and the distributed representations, and we explore self-supervision for learning these representations. We start with an observation that random masking could be viewed as a novel data augmentation scheme not previously exploited in Siamese networks. For the masked area, its potential degrecs of freedom grow combinatorially large with its size, allowing for richer superviinn than conventional augmentations such as cropping and scaling, which are heavily
In this paper, the term instance representation refers to the feature representation for the holistic image, not for object Instances used instance segmentation literatures. The term distributed representation intends mean that the image is represented by pattern activity distributed patch tokens.

---

Published in Transactions on Machine Learning Research (03/2023)
biased towards areas around the image center (He et al. 2022 More importantly, the self-supervision from conventional augmentations lead to a common representation vector that encompasses multiple augmentations of an instance (Tian et al.) 2020 Purushwalkam & Gupta 2020), and this invariance may degrade the sensitivity of the representation to spatial locality. On the contrary, random masking preserves the original content of the unmasked area and the geometrical structure of the data.
We propose a simple model, called ExtreMA, where the instance representation from masked input is trained to predict that from the full view, in plain BYOL fashion (Grill et al., 2020). The information gap created by masking encourages the student network to encode much information possible, and hence bootstrap the teacher network to be stronger. We adopt the vision transformer ViT (Dosovitskiy et al.] 2021 ) to embed distributed representations over patches, and this is followed by cross-attention blocks (Touvron et al. 2021b) to aggregate the distributed representations into the instance representation. The instance-level learning objective provides the supervision for both of the representations. In our model, the distributed representations are only implicitly learned without the corresponding token-level objective used in masked modeling. However, through investigating and visualizing the atlention maps (in Figure D. we find that the output representations of our model maintain accurate correspondences with the input tokens and that semantic clusters tend to emerge from the learned distributed reprosentations.
A notable distinction of this model is its effectiveness with an extremely large masking ratio (75% 90%). while typical masked modeling approaches work best between the range of 50% to 75% (El-Nouby et al. 2021 Bao et al. 2021 He et al. 2022). Besides the computational efficiency that this brings, a key aspect of extreme masking the complementarity that arises among multiple samples. Due to the high redundancy in visual data, the visible content for samples with different masking becomes independent only when the masking ratio becomes large. In addition, multiple masks speed up learning and convergence significantly, making the system fast learner that is hungry for more data. In practice, multiple sampling is also computationally appealing, as the teacher network for processing the full content needs only to be forwarded once,
ExtreMA enjoys the favorable properties of Siamese representation ation learning. The instance representation from the model can be used off-the-shelf for measuring semantic similarities. The framework also welcomes other data augmentations besides masking for applications with different ends. However, unlike conventional contrastive learning, ExtreMA does not rely data augmentation induced invariances to achieve generalization. Rather, ExtreMA preserves all possible useful information from the masked view in order to recover the full image. The generative aspects of our model are exhibited in Figure 2 It can faithfully inpaint the masked pixels through network inversion (Zhao et al. 2020). Moreover, the instance representation is shown to be sensitive to spatial and scale variations for localizing objects in Figure R. These properties demonstrates that ExtreMA learns both instance and distributed representations that well captures scale, location, and color intensities.
In the experiments, we systematically study the model behavior under different masking ratios, its convergence properties using multiple masks on larger datasets, and integration with various other data augmentations. Based on the study observations, we also propose a new augmentation scheme which uses shared image crops but different colors for the two input views. Our main results on ImageNet outperform prior masked modeling approaches on both finetuning and linear probing metrics. Notably, this is achieved by training ExtreMA using single node of V100 GPUs in about two days for a ViT-Base model. We also evaluate the transfer performance for semi-supervised learning and semantic segmentation. For both applications, EXtreMA produces superior results compared with prior arts.
2 Related Works
In self-supervised representation learning, labels are mined from the data itself to achieve generalization beyond that from human annotations, especially when the training data is at scale. Past works demonstrate generalization through k-nearest-neighhors (Wu et al., 2018) and zero-shot classification on the learned features (Radford et al.) 2021), or finetuning the model for a limited schedule (Chen et al. 2020cc He et al.) 2022). The central problem under investigation is how to extract the training labels automatically and formulate the pretext tasks. In high-level vision, such pretext tasks include predicting colors from a grayscale image (Zhang et al. 2016), inpainting pixels given the spatial context (Doersch et al. 2015) or through
2

---

Published in Transactions on Machine Learning Research (03/2023)
autoregression Chen et al. 2020a), predicting the orientation of a rotated image (Kmoddkii & Gidaris 2018); solving a Jigsaw puzzle given shuffled patches (Noroozi & Favaro 2016), and others (Donahue & Simonyan 2019 Zhang et al. 2017]. The key idea is that the network has to learn semantics in order to solve the pretext tasks. Recently, there has been a resurgence of the context prodiction pretext task (Bao et al., 2021 Li et a 2021) that has accompanied the rise of vision transformers (Dosovitskiy et al.] 2021p. Inpul tokens are masked, and the model is trained to predict the masked tokens from the visible tokens in BERT fashion. The target tokens could be represented by dVAE tokens (Bao et al.;
2021), ; raw pixel values (He
et al. 2022), features (Wei et al. 2022 Dong et al., 2021 from online learned encoder (Zhou et al.
2021 El-Nouby et al. 2021 Baevski et al. 2022; Chen et al. 2022 Tao et al.
2022 Such representations
are shown to surpass prior art when finetuned for downstream tasks (He et al.
2022
Contrastive learning is a special pretext task of instance discrimination to learn view-invariant representations from data augmentations. It encourages different views of an image instance have similar representations relative to negative samples (Wu et al.) 2018 He et al. 2020 Chen et al.) (020a), negative clusters (Caron
et al. 2020), or even without using negalives (Grill el. al.
2020) al all. Views of image are commonly processed by Siamese network (Chen et al. 2020b Li et al. 2022) with a momentum encoder (He et al.
2020) on one of its branches. To boost performance, the community has crafted various data angmentations including color jittering (Wu et al.]
2018). Gaussian blurring (Chen et al.) 2020b), solarization (Grill et al. 2020j, and copy-and-paste (Zhao et al. 2021), as well as determined their optimal hyper-parameters. Multiple augmentations (Fort et al. 2021) per input are investigated to improve test time performance and convergence speed for supervised learning. In self-supervised learning, DINO uses local crops to improve representation quality. Its effect on convergence speed remains under explored. Contrastive models trained at scale shown
to perform on par with supervised learning (Goyal et al. 2019 2021 The contrastive lcarning framework has the flexibility to handle various data augmentations, while traditional pretext lasks need non-trivial engineering to be trained in a multi-task manner (Doersch& & Zisserman 2017). Recently, there have been efforts (EI-Nouby et al. 2021 Zhou et al. 2021 Mishra et al.J 2022) to combine contrastive learning with masked modeling objectives in a multi-task manner. The two tasks complement each other, but the intrinsic
connection remains unclear.
The technique of masking originates from representation learning languages (Devlin et al. 2019 Liu et al.
2019 and is especially suited for transformer architectures. In computer vision, Block-wise masking (Bao
et al.
2021) and random masking (He et al. 2022) are investigated to cope with the 2D nature of images.
Aside from BERT-like training approaches [Bao et al., 2021 He et al.) 2022 El-Nouby et al., 2021 Zhou
ct al.
2021 Bacyski et al.) 2022). a special type of random masking in the form of small local image crops
has also been adopted contrastive models (Caron el al. 2021
2020 However, the local crop augmentation introduced in (Caron et al.] 2020 mainly designed for computational efficiency, without revealing the impact
of substantial content removal on representation quality. A recent work MSN (Assran et al.] 2022) explores
the application of masking in Siamese networks. However, its masking ratio is low (30% for ViT base), and
it heavily relies on other augmentations besides masking to work. ADIOS (Shi et al. 2022) adversarially optimizes masking as a novel data augmentation for contrastive lcarning. It shows masking is complementary to existing conventional image augmentations. Data2vec (Baevski et al. 2022) formulates a masked image prediction task, with the prediction target bootstrapped from a momentum encoder. Similar to Data2vec, our work explores representation learning with the Siamese architecture and the masking mochanism. Differently, our model predicts the holistic instance representation without using the mask token throughout the norw Hybrid models (Zhou et al.] 2021 Tao et al.. 2022 Mishra et al., 2022) of contrastive learning and masked image modeling formulate a multi- task problem to enjoy the best of two paradigms. However, the technique of masking is mainly to supervise masked modeling, not as an augmentation for contrastive learning. Specifically, Zhou el al. (2021) finds that adding masking augmentation actually hurts representation quality. Our work reveals the power of masking even when used as a data augmentation, without explicit supervision for token-level predictions.
3
The ExtreMA Approach
This work explores the use of random masking as data augmentation for Siamese representation learning with instance-level supervision. The model uses an architecture of two parallel networks, where the momentum
3

---

Published in Transactions on Machine Learning Research (03/2023)
supervision
prediction
projection
projection
class attention
cross attention
cross uttention
vision
vision transformer
distributed attention
transformer
masking
patchify
Figure 1: Overview of ExtreMA. Our model follows the Siamese network approach for representation learning, The momentum encoder processes the full view while the base encoder processes partial view from extreme masked sampling. Input tokens are encoded into distributed representations via vision transformers and gathered into an instance representation via cross-attention blocks with an appended [CL.S] token. Self-supervisiin is applied at the instance level. We visualize attention maps for four query patches in the last layer of ViT and cross atlention for the class token.
encoder processes the full image crop and the base encoder processes the masked image crop. The information gap, i.c. the masked image region, is the basis of the supervision for training the base encoder. An overview of our approach is illustrated in Figure I We describe details about masking and the model architecture follows.
Extreme Masking. Given an image, we first divide it into non-overlapping patches to be fed into the vision transformer. A fixed sinusoidal positional encoding is added to each embedded patch, and few of the embedded patches are sampled randomly (He et al. 2022) according to the masking ratio.
A key aspect of our approach is that it achieves its best performance with an extremely large masking ratio of 75%-90%, leaving the base encoder to process just a fraction (10%-25%) of the patches. This is contrast. to masked image modeling where the performance degrades when the masking ratio exceeds 75% (He et al., 2022 The extremely high masking ratio sets a very hard pretext task for the network. We hypothesize that the ability of EXtreMA to succeed with extreme masking is in part due to the momentum encoder processing the entire image, whereas the encoder in masked image modeling never receives the full view as input. This may create train-test discrepancy for masked modeling, as the attention blocks need to generalize from processing a fraction of the tokens to the full set.
Extreme masking can be further assisted with the multi-masking technique, where multiple paralleled basc encoders processing different masked versions of the image are used to accelerate learning. Multiple student networks can share the same learning target from the momentum encoder, which makes multi-mash maattiimmsiing efficient learner. We note that such multi-maskngg technique is not immediately applicable or efficient for masked modeling since the decoder needs to run independently for each masked input without being able to share the learning larget.
Extreme masking also introduces new challenges. We observe that extreme masking tends to overfit to the training data especially when multi-masking enabled. Concretely, the model obtains good classification performance the training dataset, but less generalizable to unseen data. This is different from "cheating",
4

---

Published in Transactions on Machine Learning Research (03/2023)
masked input
supervised
DINO
MAE
ours
ground truth
MAE (details)
ours (details)
mask 50%
mask 75%
mask 90%
Figure 2: Generative properties of the distributed representations at various masking ratios. We use the deep image prior technique to invert the representations. Our reconstruction result shows the best quality overall. Supervised ViT fails to produce meaningful content. MAE fails to inpaint proper colors due to the use of normalized pixels. DINO loses information about spatial locality.
as the feature effectively learns semantics on the training data from supervipnnn Using more unlabeled training data, viable way to mitigate the overfitting phenomenon. This is discussed further in the experiment section.
Distributed and Instance Representations. We adopt the vision transformer (Dosovitskiy et al. 2021 for its efficiency and flexibility in handling input content of variable size. The vision transformer embeds the input visual tokens into spatially distributed representation via the self-attention mechanism. An instance representation is desired in order to allow supervision from the instance level. To achieve this, we use cross-attention blocks [Touvron et al., 2021bb to aggregate the distributed patch-level representations into a single representation with an additional appended class token. Only the class token acts the query in the cross-attention blocks while the patch tokens remain frozen without updates. This makes the cross- attention blocks lightweight with O(N) complexity compared to O(N2) in self-attention. The overall encoder architecture follows Cayl-style (Touvron et al. 2021b design. The projection head and the prediction head follow the instance representation.
We have investigated two other alternatives to represent an instance, but neither works well. If we feed the instance token as an input to the transformer as ViT, optimization becomes unstable, potentially due to the large masking rate for the input. With average pooling over the token representations as the instance representation, the model finds a shorteut on learning averaged patch features without learning attention across patches. Details provided in the appendix.
5

---

Published in Transactions on Machine Learning Research (03/2023)
Learning Objective. The instance representation from the masked input is trained to predict that from the unmasked input by simply minimizing the cosine distance ween the two representations. We primarily follow BYOL for simplicity in this paper, but our approach is also found to work with the contrastive loss (Wu ct al.
2018 Oord et al.) 2018) in our experiments. Our learning objective differs with the conventional BYOL in the following aspect. Conventional BYOL adopls a symmelric loss where the two views are learned to predict each other. This will drive the representation to find the common subspace shared by the two views, with invariance over other information (Tian et al. 2020). In our case, since the shared information between the two views is obvious, finding the commonality between the views does not make for meaningful self-supervision. Thus, we adopt the asymmetric loss where only the masked view is trained to predict the intact view. The information gap created by masking encourages the network to extrapolate the masked regions, instead of seeking a shared subspace.
Compared with masked modeling approaches, our learning objective fundamentally different. The instance level supervision does not explicitly enforce spatial reasoning for cach individual token. Nonetheless, we find strong evidence that our model learns distributed representation over the tokens. We visualize the attention maps for four query patches in the last layer of the transformer block in Figure [ The shown visualization is averaged across 12 attention heads. We observe that patch tokens tend to group into meaningful semantic clusters.
BYOL Details. Our design choices for the projection and the prediction head are even simpler than the original BYOL (Grill et al., 2020). We replace the BatchNorm with LayerNorm and the RcLU activations with GeLU activations, making the overall framework free of BatchNorm and consistent with the rest of the transformer blocks. Our work also incidentally demonstrates that BYOL does not rely BatchNorm to prevent collapse (Richemond et al. 2020). The projection head and the prediction head have 3 and 2 hidden layers respectively, a hidden dimension of 4096, and an output dimension of 256, following the original design.
Limitations. It remains as a limitation without being able to fully understand why CaiT-style architec- ture (Touvron et al., 2021b) is crucial for stable convergence and why conventional ViT architecture fails. More discussions to this problem is included in the appendix. When training the ExtreMA model on a dataset, the training epochs may need to be validated and tuned in order to prevent overfitting. The optimal masking ratio may also varies depending on the property of the dataset.
4 Representation Properties
To understand the distributed representation and the instance representation trained with extreme masking augmentation, we invert the distributed representations into the pixel space and examine the sensitivity to locality of the instance representation. These results give further evidence that the model learns a meaningful distributed representation without a BERT-like objective, and that the instance representation preserves detailed visual information. The following analysis is not meant to compare different models in quantitative way, but to provide intuitive insights on what have been learned by the models.
Generative Propertics. Given a pretrained model and a masked image, we can encode the visible patches using the distributed representations, and invert these partial representations back to the pixel space to reconstruct the masked patches. Specifically, we follow the technique of deep image prior (Ulyanov et al. 2018 and minimize the I.2 distance on the visible representations between the masked image and the reconstruction. Details of the method is included in the appendix. This reconstruction techniiue allows us to examine the content of the encoded features without the need to further train new generative model. Figure 2 shows the reconstruction result. We vary the masking ratio and compare results with supervised DeiT [Tovrro et al. 2021a), DINO and MAE using this inversion method. Supervised DeiT is unable to produce any meaningful content. MAE fails to inpaint proper colors due to the use of normalized pixels. DINO is inaccurate with spatial localities. Our result is spatially smooth and accurate in color. The inversion technique suffers when the masking ratio is very large, due to limited ability to inpaint unseen semantic areas.
We acknowledge that MAE is able do amazing reconstruction using its decoder. The inversion experiment for MAE is mainly to investigate its encoder in fair comparison with competing approaches.
6

---

Published in Transactions on Machine Learning Research (03/2023)
query crop
candidate crops
prediction
query crop
candidate crops
prediction
Figure 3: ExtreMA is sensitive to spatial and scale variations. We randomly sample 25 candidate bounding boxes of 5 scales and 5 random locations from test image and use the query crop to relrieve the closest bounding box in the test image. The highest ranked crop is shown as the prediction. The instance representation from our model able to identify the correct scale and location, suggesting that ExtreMA is sensitive to information beyond semantics.
Table 1: Mask ratio.
Table 2: Multi-masks trained on IMlk.
Table 3: Trained on IM22k.
ratio
ft.
lin.
ratio 75%
ratio 80%
ratio 90%
ralio 90%
50%
81.9
36.3
num
ft.
lin.
mum
ft.
lin.
mum
ft.
lin.
mum
ft.
lin.
70%
82.3
64.4
82.4
64.8
1
82.4
67.3
82.3
61.6
1
82.5
60.4
80%
82.4
67.3
82.7
67.2
2
82.6
68.8
2
82.5
64.0
2
82.7
65.6
85%
82.4
66.3
82.9
67.7
4
82.8
67.5
82.7
63.1
4
83.0
69.0
90%
82.3
61.6
5
82.9
67.1
8
82.9
60.3
8
83.2
71.4
95%
81.6
49.3
10
83.0
59.3
10
83.2
72.0
Locality Properties. We use the k-nearest neighbor technique to probe the instance representation. We first generate a small gallery set by random sampling of image crops that vary spatially and scale from a single image, We then use another query image crop from the same semantic category rank the gallery set. We resize these image crops to 224 224 and extract the instance representations for measuring similarities. In Figure 3 the top nearest retrieval returns the image bounding box with the closest spatial and scale configuration as the query crop. The results suggest that the instance representation is sensitive to spatial and scale changes, and the learned high-level representation is not a result of invariance but is more powerful and generalizable. This example also demonstrates form of zero-shot detection using exemplars (Malisiewicz et al. 2011).
5 Experiments
5.1
Ablation Studies
We pretrain the representation on ImageNet and evaluate on finetuning (ft) and linear probe (lin) in our ablations. We finetune the model on top of the distributed representation, and conduct linear probes with the instance representation. The evaluation protocol mainly follows BEiT and MAE.
Implementation Details. We use the original ViT-base (Dosovitskiy et al. 2021 as the backbone architecture without the layer scale technique [Touvron et al. 2021b). The class attention follows the original design in Touvron al. (021b) with default of two transformer blocks and layer scale hyper-parameter of 0.1. We train our model using the AdamW optimizer (Loshchiloy & Hutter 2018) with a batch size of 2048, initial base learning rate of 1.5e-4, and weight decay of 0.1. The exponential averaging weight for
7

---

Published in Transactions on Machine Learning Research (03/2023)
mulli-masking at ratio 75%
multi-masking at ratio 80%
multi- masking at ratio 90%
53-
52
zum-1
2000
11211 2
5430
10
30m-10
52
160 258
200
150
250
220
50
160
150
203
25c
training epochs
training epochs
training epochs
Figure 4: Convergence curves for multi-masking ImageNetlk. Each plot shows kNN accuracy the validation set with respect to training epochs. Multi-masking with 80% and 90% ratio enjoys sleep learning curves, but suffers from overfitting on ImageNetlk.
Table 4: Other augmentations.
Table 5: Cross attention block.
Table 6: Training loss.
1 aug
2 augs
#blocks
ft. lin.
objective ft. lin.
augment ft.
lin.
ft. lin.
82.6
66.2
BYOL 82.6 68.8
none
82.3 55.1
2 82.6 68.8
InfoNCE 82.8 66.8
color
82.5 62.0 83.3 62.4
3
82.6 68.5
rand size crop
82.6 68.8
82.2 66.8
crop + color
82.6
69.0
83.1
73.3
shared crop + color
83.3 73.1
the momentum encoder initialized to 0.996 and increased to 1.0 following a cosine schedule. The default augmentation is random resized cropping and random flipping. All models are trained for 300 epochs.
Masking Ratio. We first vary the masking ratio using a single mask for training. In Table | the finetuning performance plateaus across a wide range from 70% to 90%, while the linear probe performance peaks at ratio 80%. Notably, an extremely large masking ratio of 90% also achieves reasonably good performance. The performance degrades beyond 90%.
Multi-Masking and Convergence Speed. For each image instance, we generate multiple masks without replacement for the student network. The loss as well as the gradient are averaged over multiple masked inputs for each parameter update. The learning rate schedule is unaffected by multi-masking and kept unchanged with single masking. We investigate the behavior of multi-masking under the ratios of 75%, 80%, and 90% in Table 2 Finel performance consistently improves with more masked inputs. However, the linear probe performance degrades when too many masks are used, especially when the masking ratio gets larger. We take a close look at this phenomenon and find that the training accuracy for lincar probing actually improves with greater multi-masking. This suggests that the model overfits to the training data without using labels. In Figure 4 we plot the n-iiabo classification curves on the validation set with respect to training epochs. The ImageNet training set. Masking with ratio 75% does not suffer from overfitting with multi-masking, but converges less quickly. Multi-masking with extreme ratio 90% has the steepest learning curve, but it tends to saturate and degrade after 120 epochs. We hypothesize that this is because masked inputs become more independent and bring complementary learning signal when the masking ratio grows larger.
k is set to 200 and the gallery is set to 10% of the
To combat overfitting while preserving fast learning, a straightforward solution is to use larger datasets. We therefore study multi-masking on ImageNet22k, which about 10 times larger in total images. We train the model for 30 epochs, which maintains the effective number of optimization iterations and reveals the impact solely from data scale. The evaluations for finetuning and linear probing are all conducted with ImageNet In Table 3 at the masking ratio of 90%, ExtreMA no longer suffers from overfitting as the number of masks increases. The model performance is also consistently better than using ImageNet1k training data. This shows that our model benefits from more data for large-scale representation learning.
Other Augmentations. Supervision at the instance level enables integration of other augmentations for both the student and teacher networks. We set the default masking ratio to 80% with two masks in the following ablations, as it does not suffer from overfitting. We consider the augmentations of cropping (random
8

---

Published in Transactions on Machine Learning Research (03/2023)
Table 7: ImageNetlk classification comparison.
Table 8: ViT-B Wall-clock time comparison using
ViT-S
ViT-B
single node of 8xV100 GPUs.
methods
epochs
ft.
lin.
ft.
lin.
methods
epochs
time
MoCo-v3
300
81.5
73.1
83.2
76.2
DINO
400
300 hrs
DINO
400
81.7
77.0
83.6
78.2
MSN
600
700 hrs
MSN
600
81.6
76.9
83.4
76.8
BETT
800
240 hrs
BEYT
800
83.2
37.6
MAE
1600
650 hrs
MAE
1600
83.6
67.8
Data2vec
800
250 hrs
Data2vec
800
84.2
60.8
ExtreMA (80% ratio x1)
300
29 hrs
ExtreMA (1k)
300
81.8
69.4
83.7
73.3
ExtreMA (80% ratio x2)
300
36 hrs
ExTreMA (22k)
30
81.5
65.7
83.9
74.5
ExtreMA (80% ratio x5)
300
60 hrs
resized crop + flipping) and color (color jittering and random grayscaling). We do not consider Gaussian blurring and solarization as their effects are marginal.
We first examine the case of single augmentation, where the input of the teacher branch is augmented and the student branch takes a random masking sample of the teacher's imput. Such a scheme is akin to enlarging the training dataset without introducing other supervision from augmentations. As reported in Table 4 by just using a center crop, our model achieves a reasonable result of 82.3% finetuning and 55.1% linear probing performance. Color and cropping augmentations improve the overall performance individually but their effects are marginal when both are used.
We next consider two independent augmentations, student's input undergoing masking well. The self-supervision in such scheme introduces invariance, such as spatial, scale, and color intensity, similar to prior contrastive models (Caron et al. 2021 Chen et al. 2021 h.
for the student and one for the teacher, with the
Crucially, we find that adding spatial and scale invariance by two crops of an image may hurt representation quality, with finetuning decreased by 0.4% and linear probing decreased by 2.0%. On the other hand, color invariance is shown to be beneficial, leading a signíficant 0.8% gain from 82.5% to 83.3%. Using cropping and color augmentations combined, the linear probing performance improves substantially to 73.3% while the finetuning performance drops 0.2% spatial invariance may hurt generalization.
Based on these observations, we propose another augmentation scheme that uses a shared spatial crop for the two network branches, but two different color augmentations. Such scheme achieves the best overall performance for finetuning and linear probing.
Cross Attention Blocks. We use cross-attention heads Touvron et al. 2021b) to aggregate the distributod representations into the instance representation. We ablate the number of blocks for this design in Table 5 The finetuning performance is not affected by the depth of the cross-attention blocks, while the linear probing performance is improved by 2% with two blocks and saturates for more blocks.
Training Objective. Besides BYOL, ExtreMA also works with other Siamese representation learning objectives, such as InfoNCE (Oord et al. 2018) with negatives. In Table G we provide the result with a MoCo-v3 implementation using a contrast temperature of 0.2. Compared with the BYOL objective, the finetuning performance improved by 0.2% and linear probing drops by 2.0%.
5.2 ImageNet Comparisons with Previous Methods
We compare with representative contrastive methods MoCo-v3, DINO and MSN, as well as masked image modeling methods BeiT, MAE and Data2vec on ImageNet classification. We use our strongest model with five masks of ratio 80% and color augmentations. The finetuning takes 200 epochs for ViT-S and 100 epochs for VIT-B following prior works. The results are summarized in Table 7 Our approach outperforms MoCo-v3, DINO, MSN, MAF, BEiT for finetuning evaluations but underperforms Data2vec. This is likely due to the layer-averaged targets in Datavec provides substantial improvement. Our linear probing outperforms the masked image modeling methods by a large margin but underperforms contrastive counterparts. This may
We note that ExtreMA can be reproduced stably with small variance of 0.1% % for linear evaluation and finetuning.
9

---

Published in Transactions on Machine Learning Research (03/2023)
Table 9: Semantic segmentation on ADE20K.
Table 10: Semi-supervised classification.
methods
epochs
mloU
methods
epochs
1%
10%
DINO
400
47.2
scratch
9.0
44.8
MSN
600
47.1
BETT
800
35.9
69.7
MoCo-v3
300
47.3
DINO
400
64.7
75.9
BEiT
800
47.1
MoCo-v3
300
57.2
75.8
MAE
1600
48.1
MSN
600
66.6
76.8
ExtreMA (1k)
300
47.9
MAE
1600
52.7
72.1
ExtreMA (22k)
30
48.4
ExtreMA (1k)
300
67.3
76.1
be due to the lack of global crops and other heavy image augmentations. Since no holistic representation is modeled for the image in masked modeling, BETT, MAE and Data2vec perform less competitive for linear probing with features average pooled from tokens. The VIT-S model does not scale well with large data, potentially limited by its model size.
A notable advantage for ExtreMA is its computational efficiency and fast convergence speed. This allows us train ViT-Base models of 300 epochs using single node of Vx0000 GPUs for 29 hours to hours shource-limitee academic
depending on the choice of multi-masking, Such hardware requirement friendly labs. On the contrary, prior self-supervised reprosentation models require multi-node training and lengthy optimization. We summarize the wall clock times of representative models in Table g Since official releases of prior methods are reported with multi-node training, we estimate their wall-clock time using just single node of 8x V100 GPUs. ExtreMA achieves 5x to 10x speedups for visual representation learning.
5.3 Transfer Learning Results
We consider two transfer learning scenarios with limited target labels: semi-supervised image classification and semantic segmentation. For both experiments, we use our strongest model with five masks of ratio 80% and color augmentations.
Semi-supervised Learning. Given the pretrained model, we use a small fraction of the ImageNet1k training labels (1% 10%) for semi-supervised finetuning. We append the classification head on the first output layer of the projection head following SimCLR-v2 Chen et al. (020c).. The finetuning protocol and data augmentation mainly follows BETT. The model is optimized using AdamW with an initial learning rate of 5e-6 for 1000 epochs and batch size of 1024. Comparison results are shown in Table Lo ExtreMA outperforms the masked image modeling approaches MAE and BEiT by a large margin of 12% and 30% using 1% of the labels. Surprisingly, ExtreMA obtains better results than DINO, which is heavily tuned for ImageNet classification with higher linear probing performance than our approach. ExtreMA is also on par with MSN which is specifically designed for semi-supervised learning. The models pretrained with ImageNet22k perform worse, because the majority of unlabeled classes are less relevant to the target 1k classes. We thus follow the prior evaluation practice and omit the TmageNet22k entry.
Semantic Segmentation. We evaluate semantic segmentation performance on the ADE20K (Zhou et al., 2017) dataset. Following prior works, we initialize the UperNet framework (Xiao et al. 2018) using our pretrained model and finetune the segmentation model end-to-end. The model optimized using AdamW for 80k iterations with an initial lcarning rate of 1o-4 and a batch size of 16. We the weight decay to 0.05 and layer-wise learning rate decay to 0.85. The results are shown in Table g Our method able to outperform competitive representation learning baselines such as DINO, MSN, MoCo-v3 and BETT. Our model is even comparable to MAE, which is trained with much heavier schedule (300 epochs vs. 1600 epochs). By scaling the training data to the larger ImageNet-22K dataset while keeping the total number of iterations unchanged, our model performance improves by 0.5 mloU, surpassing all prior arts by a significant margin. This indicates that our model scales well with data.
Object Detection and Instance Segmentation. We evaluate the transfer performance on the MSCOCO dataset. We adopt the Mask-RCNN framework for object detection and instance segmentation using the
10

---

Published in Transactions on Machine Learning Research (03/2023)
Table 11: Object detection and instance segmentation transfer on COCO.
object delection
instance segmentation
methods
epochs
AP
APso
AP75
AP
APso
APrs
DINO
400
46.8
68.6
50.9
41.5
65.3
44.5
MSN
600
45.8
68.2
49.5
40.6
64.8
43.2
MoCo-v3
300
45.5
67.1
49.4
40.5
63.7
43.4
BEiT
800
42.1
63.3
46.0
37.8
60.1
40.6
MAE
300
45.4
66.4
49.6
40.6
63.4
43.7
MAE
1600
48.4
69.4
53.1
42.6
66.1
45.9
ExtreMA (1k)
300
47.5
68.9
51.9
42.0
65.6
45.1
ExtreMA (22k)
30
48.5
69.8
53.1
42.7
66.5
46.0
ViT-base architecture. ure. We fine-tune the model for 12 epochs and evaluate the performance on the validation set. The results are summarized in Table 11 ExtreMA outperforms DINO DINO/MSN MMSS/MMoC-x3/BBBT using a lot less compute. ExtreMA outperforms MAE with the same number of pretraining epochs, but underforms MAE if MAE is trained longer. When pretrained on the ImageNet22k datasel, ExtreMA improves the performance by about 1% AP.
/BEi while
6 Conclusions
This work explores masking a novel augmentation for Siamese representation learning. The investigated approach, ExtreMA, learns strong instance and distributed representations through data augmentations without masked modeling supervision. This work is inspired by the masking operation in masked modeling. However, it makes no claims that ExtreMA works in a similar way as masked modeling, since the learning objectives are different. ExtreMA exhibits several unique characteristics: 1) the use of extremely large masking ratios, 75%-90%; 2) fast convergence speed with multi-masking and scalability to large data; 3) low consumption of computational resources. Its ability encoding precise locality for the instance representation may open up possibility for detection transfer.
Broader Impacts
The proposed method learns represenlations from a specific dataset and as such may reflect biases contained in the dataset. Building downstream application which finetunes the model on a customized dataset may also reflect bias and potentially negative societal impacts in the pretrained representations. The study this paper is limited curated datascts, and research on uncurated datasets warrants future rescarch.
References
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Mike Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. In European Conference Computer Vision, pp. 456 473. Springer, 2022.
Alexci Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. data2vec: A general framework for self-supervised learning speech, vision and language. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research. PMLR, 17-23 Jul 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. International Conference on Learning Representations, 2021.
11

---

Published in Transactions on Machine Learning Research (03/2023)
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Newral Information Processing Systems, 33:9912 9924, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1550-9660, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and llya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning, pp. 1691-1703. PMLR, 2020a.
Ting Chen, Simon Korublith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Simon Korublith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33: 22243 22255, 2020c.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning. arXiv preprint 2rXi 03026, 2022.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 0640-9649, 2021.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2051-2060, 2017.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEFE international conference computer vision, pp. 1422-1430, 2015.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in Newral Information Processing Systems, 32, 2019.
Xiaoyi Dong, Jianmin Bao, Ting Zhang. Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu Peco. Perceptual codebook for bert pre-training of vision transformers. arXio preprinl arXix:2111.12711 1(3):7, 2021.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are largo-scale datascts necessary for self-supervised pre-training? arXiv preprint (tf:::212.111... 2021.
Stanislav Fort, Andrew Brock, Razvan Pascanu, Soham De, and Samuel L Smith. Drawing multiple augmen- tation samples per image during training efficiently decreases test error. arXiv preprint arXiv:2105.180: 13343, 2021.
Priya Goyal, Dhruv Mahajan, Abhinav Gupta, and Ishan Misra. Scaling and benchmarking self-supervised visual representation learning. In Proceedings of the IEEE/CVF international conference on computer nision, pp. 6391-6400) 2019.
12

---

Published in Transactions on Machine Learning Research (03/2023)
Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, et al. Self-supervised pretraining of visual features in the wild. arXiv preprint arXiv:2103.01985 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-2182 21284, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick, Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Nikos Komodakis and Spyros Gidaris. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Hepresentations (ICLR), 2018.
Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao. Efficient self-supervised vision transformers for representation learning. In International Conference on Learning Representations, 2022.
Zhaowen Li, Zhiyang Chen, Fan Yang, Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation. Advances in Neural Information Processing Systems, 34:13165 13176, 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer, and Veselin Stoyanov. Roberla: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. 2019.
Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2018.
Tomasz Malisiewicz, Abhinav Gupta, and Alexei A Efros. Ensemble of exemplar-svms for object detection and beyond. In 2011 International conference on computer vision, pp. 89-96. IEEE, 2011.
Shlok Mishra, Joshua Robinson, Huiwen Chang, David Jacobs, Aaron Sarna, Aaron Maschinot, and Dilip Krishman. A simple, efficient and scalable contrastive masked autoencoder for learning visual representations. arXiv preprint arXiw:2210.1687) 2022.
Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European conference on computer vision, pp. 69 84. Springer, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiu preprint veXiv:180.00777 2018
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Advances in Neural Information Processing Systems, 33:3407-3418, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Pierre Richemond, Jean-Bastien Grill, Florent Altché, Corentin Talloc, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. In NewrIPS 2020 Workshop: Self-Supervised Learning- Theerry and Practice, 2020.
13

---

Published in Transactions on Machine Learning Research (03/2023)
Yuge Shi, N Siddharth, Philip Torr, and Adam R Kosiorek. Adversarial masking for self-supervised learning. In International Conference on Machine Learning, pp.
PMLR, 2022.
Chenxin Tao, Xizhou Zhu, Gao Huang, Yu Qiao, Xiaogang Wang, and Jifeng Dai. Siamese image modeling for self-supervised vision representation learning. arXiv preprint arXi::220600120 01200 2022.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827 6839, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-cfficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347 10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 32 42, 2021b.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446 9454, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the JEFE/CC CCV Conference on Computer Vision and Pattern Recognition, pp. 14668-148 14668-14678, 2022.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-paramel ric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Tele Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In European Conference on Computer Vision. Springer, 2018.
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In European conference computer vision, pp. 649-666. Springer, 2016.
Richard Zhang, Phillip Isola, and Alexei Efros. Split-brain autoencoders: Unsupervised learning by cross- channel prediction. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition, pp. 1058-1067. 2017.
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? In International Conference on Learning Representations, 2020.
Nanxuan Zhao, Zhirong Wu, Rynson WH Lau, and Stephen Lin. Distilling localization for self-supervised representation learning. In Proceedings of the AAAT Conference on Artificial Intelligence, 2021.
Bolci Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEFE Conference on Compuler Vision and Pallern Recognition, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations, 2021.
14

---

Published in Transactions on Machine Learning Research (03/2023)
A Discussions on CaiT-Style Architecture
ExtreMA follows the CaiT--stye transformer architecture Touvron et al. (2021b) where the class token is appended later in the attention blocks. We find that such design is critical for ExtreMA to stabilize learning, whereas the conventional ViT class token design failed to converge properly. Additionally, we also investigate a third option on using average pooling across tokens to aggregate the holistic representation. In Figure 5 we plot the training loss and the kNN classification accuracy for different class token designs. The ViT class token design leads to unstable optimization, and average pooling finds representation shortcut. The CaiT-style architecture works desired.
It remains as a limitation of this work to fully understand the training dynamics for the class token design. We hypothesize that the problem originates from the Siamese networks processing input sequences with very different lengths. This makes the learning the class token representation harder, when is processed throughout the network.
VIT
Avy
Poul
8.8
Lzaaining loes
8NN accourocc
A18
training opocle
training epocias
Figure 5: Training loss and kNN accuracy curves for three class token designs: ViT, average pooling, and CaiT. The network is trained with the ExtreMA objective with masking ratio 90% of 8 crops, using the ViT-small architecture.
B
Multi-masking Performance Efficiency
ExtreMA greally accelerates learning by multi-masking because the momentum encoder for processing the full tokens may be shared for multiple students. In the following Table [2] further investigate its performance efficiency. We compare the model trained with 2 masks and 300 epochs against the model trained with single mask and 600 epochs. The performance for these two settings are similar while multi multi-masking is significantly faster.
Table 12: Multi-masking performance efficiency.
ratio 75%
ratio 80%
ratio 90%
mask num.
epochs
ft.
lin.
time
ft.
lin.
time
ft.
lin.
time
1
300
82.4
64.8
30 hrs
82.4
67.3
29 hrs
82.3
61.6
28 hrs
2
300
82.7
67.2
40 hrs
82.6
68.8
36 hrs
82.5
64.0
30 hrs
1
600
82.6
67.8
60 hrs
82.6
69.1
58 hrs
82.5
63.4
58 hrs
C
Technical Details of the Generative Property
The pretrained ExtreMA model is able inpaint the masked region given the visible patches. We follow the method of Deep Image Prior Ulyanov al. (2018) to invert the representation back to pixels. Denote
15

---

Published in Transactions on Machine Learning Research (03/2023)
the pretrained ViT encoder as f() and an additional reconstruction network as re(2u), where 20 is fixed random noise and 0 is the network parameter. Given input image x processed by random mask M, the ViI encoder fMM z) extracts feature representations for the visible tokens. The reconstruction network ro(2)) generates a full image Zo. which is then passed to the encoder f(zo). We minimize a L2 energy function on the visible token represenlations between the masked input image and the reconstruction,
E (f(M 2), M f(z))).
Zo rr(2o))
(1)
The optimization takes 3000 iterations with an Adamn optimizer of learning rate 0.001. The pretrained ViT encoder remains fixed and the reconstruction network is optimized per input image. 2o is the inpainted image.
D Additional Comparisons of Locality Properties
We compare the performance on localization with other works, MAE/ DINO MoCo-v3. We use the [cls] token representation from these models. In Figure G we find that DINO performs favorably well, and that MAF/ MoCo-v3 degrades the performance notably. MAF does not supervise an instance representation in the formulation, and hence its instance representation is weaker. MoCo-v3 suffers from the heavy use of spatial cropping augmentation, and DINO improves by using small local crops for localization.
query
candidate crops
MAE
DINO
MoCo-v3
ours
Figure 6: Stale-oftthe-arl comparisons with other models for localization.
E
Details of Evaluation Protocols
The evaluation protocols for end-to-end finetuning and linear probing largely follow BETT and MAE. The byper-parameter configurations are detailed in Table I and Table 14. We finetune ViT-Small models for 200 epochs and ViT-Base models for 100 epochs. We base learning rate 1e-3 and layer decay 0.75 for
16

---

Published in Transactions on Machine Learning Research (03/2023)
ImageNetlk pretrained models, and slightly smaller learning rate 5e-4 and smaller layer decay 0.65 for ImageNet22k pretrained models. The linear probing configuration is adopted consistently for all reported entries.
Table 13: End-to-end fine-tuning protocol.
config
value
optimizer
AdamW
Table 14: Linear probing protocol.
base learning rale
le-3
config
value
weight decay
0.05
optimizer
LARS
optimizer momentum
91.92=0.9,000 0.999
base learning rate
0.1
layer-wise decay
0.75
weight decay
0
batch size
1024
optimizer momentum
0.9
learning rate schedule
cosine decay
batch size
4096
warmup epochs
5
learning rate schedule
cosine decay
training epochs
200 (S), 100 (B)
warmup epochs
10
augmentation
RandAug (9, 0.5)
training epochs
90
label smoothing
0.1
augmentation
RandommeesizeCCrp
mixup
0.8
cutmix
1.0
drop path
0.1
17

---

Published in Transactions on Machine Learning Research (03/2023)
F
Additional Visualizations
We provide additional visualizations on the generative aspects of our model in Fig. T and attention maps of the distributed representations in Fig. 9 Both visualizations reveal properties of the distributed representations. These representations maintain accurate correspondences with the input tokens, while inferring meaningful semantic relationships among tokens. We also append failure examples on the reconstructions in Figure g The feature inversion technique fails inpaint novel semantic areas especially when the masking ratio is large.
18

---

Published in Transactions on Machine Learning Research (03/2023)
masked imput
supervised
DINO
MAE
ground truth
mask 50%
mask 75%
mask 90%
Figure Additional examples inpainting at various masking ratios.

---

Published in Transactions on Machine Learning Research (03/2023)
masked input
reconstruction
ground truth
masked input
reconstruction
ground truth
mask 50%
mask 75%
PEPSI
mask 90%
Figure 8: Failure examples of our reconstructions on the generative aspect of our model.
20

---

Published in Transactions on Machine Learning Research (03/2023)
input image
ours
input image
ours
DINO
DINO
input image
ours
input image
ours
DINO
DINO
Figure 9: Attention maps on the last layer of the ViT encoder. We average the responses for 12 attention heads for visualization. Our model produces diverse and distributed attention maps, whereas DINO Caron al. (2020) mainly attends the foreground object, ignoring the others. The border color of the attention map corresponds to the colored query in the input image.
21


Reviews:
[{"review_id": "0drDa13CPL", "reviewer": "Reviewer_g5tf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- This work notes that MAEs train purely on masked inputs, and never observe _unmasked_ inputs during training, although all test inputs are unmasked.\n  By regressing the representation of unmasked inputs from the masked inputs, this work is able to learn from images like test inputs.\n  This is a correct and useful observation; however, it has been made in concurrent work like MSN (Assran et al. '22).\n  Furthermore, the claim in this work that the teacher input avoids train-test discrepancy for the method is not entirely justified, because it is only the teacher that processes full images, and not the student.\n- ExtreMA shows that extremely high masking ratios (75-90%) can be effective with the right asymmetric training scheme, which is in contrast to prior methods that used masking rates of around 60% or even as low as 30%.\n- The proposed multiple maskings of the input are efficient and improve optimization. Multiple maskings are computationally efficient, in that the same encoding of the full input can be used as a target for each, and the proposed method has no decoder, which requires more computation in the original MAE. This reduces the per-step and total computation time (Table 8). Additionally, multiple maskings speed up convergence, and while they lead to overfitting on IN-1k, they further improves results for the larger dataset of IN-22k (while single masking does not improve over IN-1k for the same number of optimization steps).\n- The proposed extreme masking and multi-masking are effective at the stated purpose of improving the accuracy of linear probing (Tables 1-3). However, there is little to no effect on fine-tuning accuracy with differences of less than 1 percentage point.\n\n**Weaknesses**\n\n- Key related work on masking and contrastive learning is not reflected in the claims nor included in related work.\n  - ADIOS [A] adversarially optimizes input masks as a kind of data augmentation for contrastive learning and compares with random masking (at the same rate as the original MAE).\n  This work is related by its consideration of masking for contrastive learning, although not to the rate of ExtreMA, and in its consideration of local/non-spurious representations (ADIOS, Table 5) without per-token supervision.\n  - CAN [B] combines masking and contrasting, along with noise prediction, in order to learn representations that are transferrable by fine-tuning and linear probing, with better linear probing accuracy than the original MAE. While different in technique, CAN shares some goals with this work, such as improving linear probing accuracy above pure masking methods.\n  - data2vec and MSN [Assran et al. ECCV'22] are credited in the related work, but not accounted for in the claimed contributions of this submission, in particular (1): data2vec, MSN, and the original MAE for that matter all apply masking for the self-supversion of attention networks.\n- The method is not fully explained and self-contained.\n  First, the architecture is not fully specified. The schematic of the architecure (Figure 1) and does not include the projection and prediction heads. While these are common to asymmetric siamese network methods, it is still preferable to identify their use in the student and teacher.\n  Second, the use of multiple maskings of each input implies several design choices: how to alter the batch size (by growing it or having fewer inputs with more maskings), how to count steps for optimization (does this work measure an epoch as equal to one pass over the dataset, which would take more steps due to multi-masking, or as equal to the original number of batches without duplicates), and relatedly how to adjust hyperparameters accordingly.\n- The use of multiple maskings per input is novel, but more narrowly, because multiple augmentations have been studied in prior work [C].\n  More specifically, multiple augmentations for a single input during training has been applied for supervised learning [C].\n  However, the prior work did not look at self-supervision, and it may include conflicting findings, in that it reports slower convergence on the training set but higher accuracy on the test set.\n- Experiments are incomplete w.r.t. prior work and lack some controls.\n  - It is not clear if the alternative representations are transferred and evaluated as intended in their original papers. For instance, MAE and adata2vec apply mean pooling to learned tokens for the purpose of classification (see the MAE code https://github.com/facebookresearch/mae or data2vec paper Sec. 4.1). However, this work comments that \"Since no explicit supervisions are applied on the [cls] token in masked modeling, BEiT, MAE and Data2vec perform less competitive for linear probing.\" which suggests a different choice of input is used for linear probing.\n  - Semi-supervised classification (Table 10) excludes MSN, which is cited in related work, but MSN reports comparable or better results for both ViT-B and ViT-S in this setting. Some reuslts of DINO are also higher in the paper than in this submission. Where are the differences?\n  - The quality of reconstructions/inversions is only evaluated with qualitative examples (Figure 2), and it is not specified how these examples are chosen. Is this a random sampling or a cherry-picking of results? For what it's worth the column of ExtreMA results does look better, but there is no non-qualitative assessment, such as an FID score, or retrieval scores for matching inversions with the original input images across the dataset.\n  - Localization (see \"Locality Properties\") is only evaluated with qualitative examples (Figure 3), but other work considers linear or nearest neighbor evaluations of local tasks like segmentation, such as DINO, which include ground truths and quantitative scores. As a first step, this work could at least evaluate the localization task of ILSVRC (assuming Figure 3 is showing images from ImageNet, which is not specified in the caption).\n  - Multi-masking is proposed as a key part of ExtreMA, but multiple augmentations of the same input could likewise be applied in the other self-distillation methods. This may have some potential benefit for optimization with other types of augmentations too, as suggested by [C], and so it would be informative to compare with at least one of MoCo/DINO/etc. with multiple augmentations of each input.\n- (Minor) The chosen terminology for \"instance\" and \"distributed\" representations is likely to confuse different kinds of readers.\n  For example, \"instance\" could refer to a single input image, an object within an image, or a unique individual object across images (such as a particular person).\n  At the same time, a \"distributed\" representation already has a defined meaning in neuroscience and machine learning that indicates a representation includes multiple dimensions or units.\n  Consider terms that are shared by other work, such as image/global/holistic representations of a whole input image, and local/dense/spatial representations of the tokens across space.\n  Choosing clearer terminology could work better than trying to fix the confusion with footnote 1.\n\n**Summary**\n\nThis is a borderline submission. On the positive side, it does explore a different use of masking for visual representation learning, and highlights the use of extreme and multiple transformations of single inputs, and the experiments and ablations show improvements from these choices. This is either in contrast to other works on self-supervision, or in addition to findings for supervised learning (concerning the use of multiple augmentations of the same input).\nOn the negative side, the claimed contributions 1 & 3 are not entirely compatible with existing papers, and there are some concerns about the comparability of exprimental settings across papers.\n\n- Claim #1. data2vec, MSN, and CAN (plus even the original MAE) could be considered to satisfy this claim. This should be sharpened into a claim about ExtreMA itself.\n- Claim #2. This is explained in the paper and supported by clear exprimental evidence: multi-masking is computationally efficient, speeds up convergence, and improves accuracy given a larger pre-training dataset.\n- Claim #3. Is this not already shown by works on self-supervised learning of image-level representations, like MoCo and DINO, which evaluate on global and local downstream tasks? Their level of supervision is also on the image or \"instance\" level of representation. This claim should be adjusted to highlight what further information is contributed by the downstream experiments for ExtreMA.\n\nNote: I have marked \"No\" to the question on Claims and Evidence for now due to the discprepancies between the claims, results, and prior work and due to the lack of precision in some of the statements in the paper. However, I will reconsider this pending the response and revision provided for this submission.\n\n**Other Feedback and Questions**\n\n- \"Learning requires the model to capture informative variations in an instance, instead of encouraging invariance.\" How? Invariance to transformations will also improve the accuracy of the regression for self-distillation. Is this saying that this learning target does not collapse, or in other words cannot be invariant to the input? If so, please re-phrase, perhaps by saying \"instead of encouraging invariance (collapse) across all inputs\".\n- How is the overfitting of extreme masking on the training set, as discussed on page 4, different from \"cheating\"? If it \"effectively learns semantics\" than why does it not generalize. Please make this statement more precise.\n- \"CaiT-style architecture\" likely indicates cross-attention transformers, as in Touvron et al. 2021b, but this abbreviation is not introduced or explained. Please spell out its first usage.\n- Masked Contrastive Representation Learning [D] is yet another hybrid scheme for masking and contrasting, and while it is only on arxiv, and differs from ExtreMA in including a local reconstruction loss, it may nevertheless be worth a look. (To be clear: this arXiv has not been considered as prior work for the purpose of this review.)\n\n**References**\n\n- [A] [Adversarial masking for self-supervised learning](https://arxiv.org/abs/2201.13100). Shi et al. ICML'22.\n- [B] [A simple, efficient and scalable contrastive masked autoencoder for learning visual representations](https://arxiv.org/abs/2210.16870). Mishra & Robinson et al. arXiv'22.\n- [C] [Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error](https://arxiv.org/abs/2105.13343). Fort et al. arXiv'21.\n- [D] [Masked Contrastive Representation Learning](https://arxiv.org/abs/2211.06012). Yao et al. arXiv'22.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Self-supervised learning of visual representations without human annotations now achieves accuracies comparable to those of supervised learning, but research continues to seek higher accuracies, more computational efficiency, and more generality w.r.t. input data and its augmentations.\nThis work studies self-supervised representation learning for vision by student-teacher/self-distillation training of siamese networks where the teacher takes unmasked images and the student takes masked images.\nIn general, this follows asymmetric siamese network training methods like MoCo, BYOL, DINO, and the like, but this work focuses more specifically on two aspects: (1) masking as the input transformation for self-distillation and (2) designing the attention architecture and learning targets for efficient training.\nFor masking, a high rate (75-90%) is applied to the student input, which is indeed higher than the rate for alternative methods.\nThis contributes an existence proof for the use of such high masking, in showing experimentally that it can improve over other masking approaches like the MAE, and in particular it achieves\nhigher linear probing accuracies than the standard MAE by its BYOL-like scheme (Table 7), but these are lower than other contrastive and self-distillation methods like MoCo and DINO.\nFor attention architecture, the vision transformer (ViT) for pre-training includes both its spatial or \"distributed\" tokens and its global or \"instance\" token (the CLS token), and the CLS token is selected as the representation for self-distillation.\nThat is, the CLS token of the teacher serves as the target for regression the CLS token of the student.\nThe choice of CLS token regression makes the method more efficient, compared to masking with the MAE, because it requires no decoding from its latents to the images.\nThis contributes a slightly different architecture for asymmetric siamese network learning.\nFurthermore, the teacher representation of a single, unmasked input can be shared as the regression target of multiple student representations from multiple maskings of the same image.\nThis reduces computation, again relative to the MAE, and relative to alternative contrastive and self-distillation schemes that transform both inputs.\nThis contributes a more efficient training computation, both per step and in the total number of steps (Figure 4, Table 8).\nThe method as a whole is called ExtreMA, after its extreme masking, but it is likewise defined by its choice of attention architecture and learning targets.\nThe experiments contribute an ablation of ExtreMA's design choices and an evaluation of its self-supervised representation in to with multiple current methods that make use of masking (MAE, BEiT) and self-distillation (MoCo, DINO).\nThe ablation covers masking rate, multi-masking, additional data augmentations, architecture choices like cross-attention depth, and the choice of loss between BYOL (without negatives) and InfoNCE (with negatives).\nThe evaluation of the presentation covers ImageNet itself (the pre-training data) and downstream tasks (semantic segmentation on ADE20K, detection on COCO).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical Changes for Acceptance**\n\n- Please discuss the potentially related works [A, B], or explain in the response how they are not sufficiently related.\n- Report the results for MSN as prior or concurrent work in the tables comparing with other methods on ImageNet and downstream tasks (Tables 7-11).\n  It is likewise a masking and self-distillation method and so it deserves to be in these comparison.\n- Specify how multiple masking interacts with training batches and updates.\n  Are the epochs the number of passes through the dataset, or the number of updates made?\n  This question is relevant because of multiple masking.\n  For example, if there are two masks per input, then are twice the number of update steps made in an epoch?\n  Or does optimization accumulate gradients across computational batches so that the same number of updates are made with or without multiple masking?\n- Please clarify the architecture used for section 5.3 and whether or not the results for the comparison methods (1) control for the architecture and (2) are taken from the prior work or re-evaluated.\n  This is needed because some results seem to differ from those reported elsewhere, such as the DINO results in this submission's Table 10 vs. Table 2 in MSN.\n- Please clarify the linear probing evaluation of the compared methods, such as the MAE and data2vec, and whether or not the approach used in this evaluation differs from the original papers.\n\n**Minor Changes for Improvement**\n\n- Consider rephrasing Claim #2 about data hunger as more precise and measurable claim, perhaps about accuracy vs. the number of training inputs, or simply saying that ExtreMA continues to improve on larger datasets like IN-22k.\n- Table 8: consider equalizing the number of epochs so that methods can be compared on the same scale. It is possible to see that there is a difference as it is, but one has to divide the number of epochs and hours for a closer comparison.\n\n**Update following Rebuttal and Revision**\n\nThe critical changes were addressed by the rebuttal and corrected in the revision. While potential improvements remain the claims and the evidence are now in agreement with accurate, convincing, and clear materials.\n\nI still hold that certain choices in the exposition are suboptimal for clarity and accessibility, such as the wording for \"instance\" and \"distributed\" and the discussion of \"cheating\" during training, but these have been discussed in this review and the review by xj2Y so the feedback is there for the authors to decide on. These are not issues that preclude acceptance.", "broader_impact_concerns": "This submission is about a general method for self-supervision of deep networks on image data and so it does not have any specific ethical concerns.\nA broader impacts statement is included to warn against the possibility of bias in pre-training data and the need to check for issues when fine-tuning to a given downstream task.\nWhile generic, this warning is sufficient for this work.\n\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1jyaCAUBA8", "reviewer": "Reviewer_xj2Y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of the method is reasonable and interesting \n2. Good empirical results on a few vision tasks (ImageNet classification, semi-supervised classification, instance segmentation, semantic segmentation), while training is quite a bit faster than some competing methods.\n3. Good ablation studies of different aspects of the method\n4. Mostly quite good writing\n\nWeaknesses:\n\n1. No empirical comparisons to data2vec and a limited discussion of the relationship between data2vec and the proposed method, while as far as I can tell the approach is very related, the main conceptual difference being training on aggregated whole-image vs spatially distributed per-token representation (as well as many other technical details). There should be a proper discussion of the relationship and a proper comparison.\n\n2. Related to the previous, there is no systematic evaluation of aggregated whole-image vs spatially distributed per-token representations, while training on whole-image representation is more or less the main contribution of the paper. It would be very helpful to do some apples-to-apples comparisons, in terms of training speed and final performance. Otherwise, it remains unclear if the good performance of the method is specifically thanks to the use of whole-image representation or some other tricks.\n\n3. Some unfounded statements:  \n3a. \"the self-supervision from conventional augmentations lead to a common representation vector that encompasses multiple augmentations of an instance, and this invariance degrades the sensitivity of the representation to spatial locality\" - need to support this statement with a reference or experiment.  \n3b. \"The ability of ExtreMA to succeed with extreme masking is in part due to the momentum encoder processing the entire image, whereas the encoder in the masked image modeling never receives the full view as input. This creates train-test discrepancy for masked modeling, as the attention blocks need to generalize from processing a fraction of the tokens to the full set.\" - is there direct experimental evidence to support this? Stated as a fact, although I believe this is an unconfirmed hypothesis.  \n3c. \"Extreme masking provides two other critical benefits...\" etc - this paragraph needs empirical support, I would advise the authors to mostly stick to describing the method in the method section and analyze its properties later in the experiments.\n\n\n4. Writing/presentation issues:  \n4a. Many papers cited as arxiv, while they have been officially published already  \n4b. Inconsistent use of \"vision transformer\" and \"visual transformer\". ViT stands for \"vision transformer\".  \n4c. Terms \"instance\" and \"holistic instance\" are not quite clear, in my opinion. I think by these terms the authors mean the whole image here, but I would say a more standard meaning of \"instance\" would be one object in an image, like in \"instance segmentation\". I would recommend rephrasing these, or at least very clearly explaining what is meant.  \n4d. The use of the term \"distributed representation\" is also confusing - here the authors mean the \"per-token\", \"spatially distributed\" representation before pooling over the image, while a more standard meaning, as far as I know, has to do not with the spatial organization of features but with how features code for the properties of the model input (see e.g. http://www.cs.toronto.edu/~bonner/courses/2014s/csc321/lectures/lec5.pdf, slide 6; or https://web.stanford.edu/~jlmcc/papers/PDP/Chapter3.pdf)  \n4e. \"The space of masking augmentations becomes combinatorially smaller when the masking ratio increases.\" I do not quite get this statement. Yes, the space of 90%-masking masks if smaller than the space of 70%-masking masks, but still it's a huge space - assuming 196 patches, C(196,19) is over 10^26, which is way-way more than the model will ever be able to observe during training. How does it explain overfitting? I personally would vaguely hypothesize that the model overfits because the task is very difficult and the model tends to learn images by heart as a shortcut.  \n4f. \"This is different from “cheating”, as the feature effectively learns semantics on the training data.\" - I do not get this phrase, please rephrase. I guess the intended meaning is that the model does not fail on representation learning completely like some other self-supervised methods, but this needs to be expressed more clearly.  \n4g.  \"The overfitting phenomenon simply suggests that extreme masking is hungry for more training data\" - this sentence does not sound exactly scientific, better rephrase. Also, yes overfitting could be solved by using more data, and it could be also solved in other ways, why state specifically this one here?  \n4h. \"The distributed representations are frozen without updates, and this makes the cross-attention blocks lightweight with O(N) complexity compared to O(N^2) in self-attention\" - I don't get it, what does the complexity of cross-attention have to do with the representations being frozen? And why/when are they frozen?  \n4i. \"Conventional BYOL adopts a symmetric loss where the two views are learned to predict each other. <...> Thus, we adopt the asymmetric loss.\" - I'm unsure what's meant by symmetric and asymmetric here, clarifications are needed. Would be good to add a formula for the loss too.  \n4j. \"Specifically, we follow the technique of deep image prior (Ulyanov et al., 2018) and minimize the L2 distance on the visible representations between the masked image and the reconstruction.\" - this needs to be explained in much more detail (potentially in the appendix). I know the Deep Image Prior and still, it's far from obvious how exactly it's used. Some readers may not know it at all.  \n4k. Visualizations in Section 4 are interesting but purely qualitative, so it's difficult to really make any proper conclusions based on them.  \n4l. Why some results (say, tables 1-7) are with linear probing, while others with kNN (say, Figure 4)? Would be nice to report kNN in tables too.  \n4m. \"our model wants more data\" - sounds unscientific, better smth like \"benefits from more data\" or \"can make use of more data\"  \n4n. \"Such hardware requirement is amenable to resource-limited academic labs\" - I'm not sure the word \"amenable\" works here  \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a method for self-supervised representation learning from images that can be seen as a combination of masked modeling and BYOL - namely, masking (with large masked fractions, around 80%) is used as an augmentation in the BYOL training setting. Note that, unlike data2vec, training is performed on the aggregated whole-image representation, not on the spatially distributed per-token representation. The method is evaluated on a few tasks (ImageNet classification, semi-supervised classification, instance segmentation, semantic segmentation) and shown to perform favorably compared to baselines while being faster to (pre-)train.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Fix all the issues mentioned in \"weaknesses\" (see there for more detail):\n1. (critical) Empirically compare to data2vec and add a somewhat detailed discussion of the relationship between data2vec and the proposed method\n2. (very desirable) Add apples-to-apples comparisons of training on whole-image vs spatially distributed per-token representations\n3. (critical) Rephrase unsupported claims or add evidence\n4. (critical) Fix writing/presentation issues", "broader_impact_concerns": "The paper works on improving self-supervised image representation learning, I don't see any specific concerns, other than that generally better computer vision systems can be used both for good and for bad.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0WtmKMpaPE", "reviewer": "Reviewer_h3DV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strenghts\n------------------------\n1. The paper identifies a gap between MAE based methods and contrastive methods for downstream instance level tasks and the need for both image level and spatially discriminative representations. The proposed method serves to both bridge this gap and provide a versatile representation that can be used for various downstream tasks.\n\n2. The paper is well written. The analysis of the representation in figures 2 and 3 is particularly helpful.\n\n3. Extensive ablations on masking ratio, multi-masking (using more than 1 mask for the student network), and an analysis of why the performance drops with more masks in Table 2, help the reader understand the efficacy of the proposed method and its hyper-parameters.\n\n4. The paper makes some interesting and creative choices:\n(a) An asymmetric instance discrimination loss suitable for the proposed method,\n(b) A novel shared crop + color augmentation strategy that helps get high linear eval and fine-tuning Top-1 accuracy on ImNet-1k.\n\n------------------------\n\n### Weaknesses\n------------------------\n1. The effect of multi-masking in studied by controlling the number of training iterations. However, in most use cases the limiting resource is computation time. Is multi-masking beneficial when trained for the same number of total hours?\n\n2. Figure 3: Are these results cherry picked or purely random. Please include some failure cases.\n\n3. No error bars are included in any of the results. This is likely due to the computational requirements of each experiment. One work around is to include error bars for the default method (Say ExtreMA (80% ratio x N) ) for downstream ImageNet-1k linear eval. This will help readers when they try to reproduce the results and also for understanding which comparisons might be within error bars.\n\nBonus: The model is trained only on ImNet-1k / ImNet-22k. The readers would benefit from knowing what happens when the model is trained on long tailed data such as random video frames or scene centric images such as in the MS-COCO dataset.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a novel method for self-supervised representation learning for downstream recognition tasks. The key idea is two fold:\n\n(a) Use a cross attention layer on top of a Vision Transformer backbone to obtain image level representations (as opposed to global average pooling).\n\n(b) Use instance discrimination as the self-supervision proxy task but use \"extreme\" masking as an aggressive data augmentation strategy. Extreme masking here refers to the kind of masking done in Masked Autoencoders but with an even higher masking percentage (~80%)\n\nThe findings are:\n\n(a) The proposed method benefits from extreme masking up-to even 90%.\n\n(b) It learns representations that are both suitable for image level tasks (image classification) and spatial tasks (object detection, semantic segmentation, instance segmentation)\n\n(c) The method benefits from pre-training on bigger datasets (ImNet-22k > ImNet-1k).\n\n(d) Linear probing performance of MAE based methods is now on-par with SoTA contrastive methods (previously these were much worse, Table 7).\n\n\n--------\n### Post discussion update\n\nMy concerns have been addressed. I have updated \"claims and evidence\" to \"yes\".", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Minor changes\n--------------------------------------\nIn page 4: \"or efficient for masked modeling ...\"  ... I think you mean that the decoder needs to be rerun for each mask, making it inefficient. Or did I misunderstand this. Please clarify in text.\n\nIn section 5.2.: You could revisit, in the first paragraph, the point made in the introduction about MAE methods not being suitable for a direct evaluation.\n\n---------------------------------\n### Major changes\n---------------------------------\nAddressing weaknesses (1), (3) would be important for acceptance. Addressing these would flip by answer under \"Claims and evidence\" to Yes. (2) would be great and the \"bonus\" would strengthen the paper.\n\nPlease include a discussion of the limitations of the method (could be in the appendix as well).\n", "broader_impact_concerns": "The paper does not include a broader impact statement. The proposed method does not directly create ethical concerns, but all recognition methods especially those that do object detection could have ethical implications. A brief comment about this would be great.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers lean towards acceptance of the paper. The AE thanks the authors and reviewers for engaging in productive discussions that have improved the manuscript. The AE supports accepting the paper as is -- there have been substantial revisions to address reviewer concerns.
Decision: Accept (poster)