MST: Masked Self-Supervised Transformer for Visual Representation
Authors: Zhaowen Li
Zhiyang Chen
Fan Yang
Wei Li
Yousong Zhu
Chaoyang Zhao
Rui Deng
Liwei Wu
Rui Zhao
Ming Tang
Jinqiao Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-supervised
Transformer
Attention-guided mask strategy

Abstract: Transformer has been widely used for self-supervised pre-training in Natural Language Processing (NLP) and achieved great success. However, it has not been fully explored in visual self-supervised learning. Meanwhile, previous methods only consider the high-level feature and learning representation from a global perspective, which may fail to transfer to the downstream dense prediction tasks focusing on local features. In this paper, we present a novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image while preserving the global semantic information. Specifically, inspired by the Masked Language Modeling (MLM) in NLP, we propose a masked token strategy based on the multi-head self-attention map, which dynamically masks some tokens of local patches without damaging the crucial structure for self-supervised learning. More importantly, the masked tokens together with the remaining tokens are further recovered by a global image decoder, which preserves the spatial information of the image and is more friendly to the downstream dense prediction tasks. The experiments on multiple datasets demonstrate the effectiveness and generality of the proposed method. For instance, MST achieves Top-1 accuracy of 76.9% with DeiT-S only using 300-epoch pre-training by linear evaluation, which outperforms supervised methods with the same epoch by 0.4% and its comparable variant DINO by 1.0%. For dense prediction tasks, MST also achieves 42.7% mAP on MS COCO object detection and 74.04% mIoU on Cityscapes segmentation only with 100-epoch pre-training.

MST: Masked Self-Supervised Transformer for Visual Representation
Zhaowen Lit** Zhiyang Chen
Fan Yang
Wei Li
Yousong Zhu Ming Tang"
Chaoyang Zhao
Rui Dengov Liwei Wu
Rui Zhao
Jinqiao Wang**
National Laboratory of Pattern Recognition, Institute of Automation, CAS
*School of Artificial Intelligence, University of Chinese Academy of Sciences
"SenseTime Research
V University of California, Los Angeles
(zhaowen. 1i, zhiyaann........... then,yousong zhu, chaoyang (ttpaaannn..ahnnnl//a
zeazzahhhlpppp ia. ac.
@nlpr. ia. ac. cn
(yangfan1, liweil, dengrui, wuliwei, zhaorui]@sensettime.
Abstract
Transformer has been widely used for self-supervised pre-training in Natural Language Processing (NLP) and achieved great success. However, it has not been fully explored visual self-supervised learning. Meanwhile, previous methods only consider the high-level feature and learning representation from global perspective, which may fail to transfer to the downstream dense prediction tasks focusing on local features. In this paper, present novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image while preserving the global semantic information. Specifically, inspired by the Masked Language Modeling (MLM) in NLP, we propose a masked token strategy based on the multi-head self-attention map, which dynamically masks some tokens local patches without damaging the crucial structure for self-supervised learning. More importantly, the masked tokens together with the remaining tokens are further recovered by a global image decoder, which preserves the spatial information of the image and is more friendly the downstream dense prediction tasks. The experiments on multiple datasets demonstrate the effectiveneess and generality of the proposed method. For instance, MST achieves Top-1 accuracy of 76.9% with DeiT-S only using 300-epoch pre-training by linear evaluation, which outperforms supervised methods with the same epoch by 0.4% and its comparable variant DINO by 1.0%. For dense prediction tasks, MST also achieves 42.7% mAP MS COCO object detection and 74.04% mloU on Cityscapes segmentation only with 100-epoch pre-training.
Introduction
As Yann LeCun said, "if intelligence is cake, the bulk of the cake is unsupervised learning". This sentence reflects that learning. Common approaches focus on designing different pretext tasks [10] 29 14 36 and aim to learn useful representations of the input data without relying on human annotations It then uses those representations downstream tasks, such as image classification, objection detection, and semantic segmentation.
Learning played central role in the resurgence of deep 6 [3]
3
Work done as an intern at SenseTime Research.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

In computer vision, previous methods focus on designing different pretext tasks. One of the most promising directions among them is contrastive learning/instance discrimination [77 23 which regards each instance in the training dataset as single category. Based on instance discrimination 13 some methods show the effectiveness in the image classification task. They successfully bridge the performance gap between self-supervised and full-supervised methods. However, almost all of self-supervised learning methods, which formulate the learning as image-level prediction using global features, are suboptimal in the pixel-level predictions [14 I [3), such as object detection and semantic segmentation. Also, InfoMin [35] finds that high-level features do not truly matter in transferring dense prediction tasks. Here, current self supervise learning may overfit to image classification while not being well tamed for downstream tasks requiring dense prediction.
14
Meanwhile, large-scale pre-trained models have become the prevailing formula for a wide variety of Natural Language Processing (NLP) tasks due to its impressive empirical performance. These models typically abstract semantic information from massive unlabeled corpora in self-supervised manner. The Masked Language Modeling (MLM) [10] has been widely utilized as the objective for pre-training language models. the MLM setup, a certain percentage of tokens within the input sentence are randomly masked, and the objective is to predict the original information of the masked tokens based only on its context. In NLP tasks, we found that the different mask strategies used in the MLM framework had a great impact on the performance of the model. However, in the field of vision, images have information of imput images is randomly distributed in tokens. If tokens are randomly masked, it will lead to poor performance. Some of previous methods use random tokens, such as iGPT B and ViT [1]. iGPT trains self-supervised Transformers using an amount of 6801M parameters and achieves 72.0% Top-1 accuracy on ImageNet by masking and reconstructing pixels, while VIT trains ViT-B model on the JFT-300M dataset, and the result is significantly lower than the supervised model.
noisy, and redundant format compared to text. The main
The random MLM is prone to mask the tokens of crucial region for images, resulting misunder- standing, and is not suitable for directly applying to self-supervised vision Transformers. In order to avoid masking the tokens of crucial region, we propose a masked token strategy based on the multi-head self-attention map, which dynamically masks some tokens of patches without damaging the crucial structure for self-supervised learning. Notably, the strategy would not increase the training time. Also, predicting original tokens alone may cause the model to over-emphasize local region, and therefore suppress the ability to recognize objects. Hence, in this paper, we present a novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image while preserving the global semantic information. In addition, a global image decoder is further exploited recover the spatial information of the image and is thus more friendly the downstream dense prediction tasks.
We validate our method on multiple visual tasks. In particular, on the ImageNet linear evaluation protocol, we reach 76.9% top-1 accuracy with DeiT-S and achieve the state-of-thee-ar performance. Overall, make the following contributions:
We propose a new masked self-supervised transformer approach called MST. of self-attention map guide the masking of local patches, thus enhancing the understanding of local context semantics in pre-training without damaging the crucial structure.
It makes full use
Our method can effectively recover the spatial information of the image by global image decoder, which is vital for the downstream dense prediction task and greatly improves the versatility and scalability of the pre-training model.
Extensive experiments demonstrate the effectiveness and transfer ability of our method. Specifi- cally, the results on ImageNet [9]. MS COCO [18] and Cityscapes [8] show that our method outperforms previous tate-of-the--ar methods.
2 Related Works
2.1 Self-supervised visual representation learning
Following MLM paradigm in NL. 10 25 iGPT 3 trains self-supervised Transformers by masking and reconstructing pixels, while ViT masks and reconstructs patches. Recently, the most competitive pretext task for self-supervised visual representation learning is instance discrimination

---

Teacher
Projection
Encoder
Head
Map
MLP Head
Lineur Projection
Transformct Encuder
Decuder
Student
Attuntlas-gaäle Maaca Straltgy
Estra learnabic [chea] emboliling
Extra Iearaaic [musk] cnbeciling
Postiam zmbbilling
Probubility distrbutions
The pipeline of our MST. Both student and teacher share the architecture with different parameters. Inspired by the MLM NLP, the attention guidedd mask strategy first introduced to mask tokens of student network based on the output self-altention map of the teacher network. The basic principle to mask some patches with low responses and does not destroying the important foreground regions. Then, global image decoder is used to reconstruct the original image based the masked and unmasked tokens. Finally, the total function consists of the sule-supe cross entropy loss and the restoring loss.
Figure 1:
4 13 I.. The learning objective is simply to learn representations by distinguishing each image from others, and this approach is quite intractable for large-scale datasets. MoCo [14] improves the training of instance discrimination methods by storing representations from a momentum encoder instead of the trained network. SimCLR [4] shows that the memory bank can be entirely replaced with the elements from the same batch if the batch is large enough. In order to avoid comparing every pair of images and incur overfitting, BYOL [3] directly bootstraps the representations by attracting the different features from the same instance. SwAV [I] maps the image features to a set of trainable prototype vectors and proposes multi-crop data augmentation for self-supervised learning to increase the number of views of an image. MoCov3 (7) and DINO (2) apply the self-supervised learning methods of computer vision to Transformers and achieve superior performance in image classification task. These works achieve comparable results compared to supervised ImageNet [9] pre-training. The success of these methods suggest that is central importance to learn invariant features by matching positive samples. However, almost all of these self-supervised learning methods formulate the learning process as image-level prediction using global features, so they lack the ability to pay attention local features.
2.2 Self-supervised dense prediction learning
Based on the existing instance discrimination, some researchers propose self-supervised dense prediction methods. Self-EMD [99 adopts Earth Mover's Distance (EMD) to compute the similarity between two embedding. Insloc (33 pastes image instances at various locations and scales onto background images. The pretext task is predict the instance category given the composited images as well as the foreground bounding boxes. PixPro (31 directly applies contrastive learning at the pixel level. DenseCL 28 presents dense contrastive learning by optimizing a pairwise contrastive loss at the pixel level between two views of input images. VADeR [24 and FlowE [32 also learn dense image representations
downstream detection and segmentation tasks. Meanwhile, HED focuses downstream segmentation task. These methods also show the effectiveness in detection and segmentation tasks but get poor performance on image classification tasks. In word, these methods overfit single task and cannot train general pre-training model.

---

Figure 2: Illustration of attention-guided mask strategy. It improves preserving key patterns in images, compared the original random mask. Description images from left right: (a) the input image. (b) attention map obtained by self-attention module, (c) random mask strategy which may cause loss of crucial features, (d) our dntention-guider mask strategy that only masks nonessential regions. In fact, the masked strategy is to mask tokens.
3 Methods
The pipeline our proposed MST is shown in Figure We propose a Masked Self-supervised Transformer (MST) approach, which creatively introduces attention-guided mask strategy and uses to complete image restoration task. Our method combined with some classical components of instance discrimination, such the momentum design, asymmetric data augmentations, and multi-crop strategies. Here, we first review the basic instance discrimination method in.3.1 Then, the mechanism and effect of our attention-guide the reconstruction branch and the training target of our method in 3.3
mask strategy explained in 3.2] Finally, we describe
3.1 The basic instance discrimination method
29 many existing augmentation policies adopt random resized cropping, horizontal flipping, color jittering and so on. We generate multiple views for each image a under random data augmentation according to multi-crop [[]. This operation can acquire two standard resolution crops and 22 representing the global view and sample N low-resolution crops indicating partial view. They are encoded by two encoders, teacher network fa and student network fa. parameterized by 8, and 0g respectively, and outputting vectors O, and Os. Both encoder and fi consist of a Transformer backbone and a projection head (5), which share the same architecture with different parameters. The parameters 0, of fixed encoder fo is updated by the moving-average average of 9, according to Eq
As noted in prior works
θ: = m 0g + m.) * A.
(1)
Given a fixed teacher network fr, the student network f. learns the parameters 0, by minimizing cross entropy loos as Eq (2.
N+2
LCE(0.)=
Σ
SE(L2)
3.2 Masked token strategy
Random mask strategy. Inspired of the MLM strategy for natural language pre-training, we apply the random mask strategy to self-supervised learning. Given dataset Q without manual annotations, and e= (e1.... e,) denote image of tokens, where i 1, 1,.., n. Let m = (m....., m,,) denote a binary vector of length n, where m, 1), representing the mask over image. According to BERT 10), the m can be obtained with probability p by Eq (3), and the is 0.15 by default.
f1. 1. prob, < p
m;
(3)
0, otherwise

---

Algorithm Pseudo code of attention-guiddd mask strategy in a Py PyTrchh-like style.
# 10: linear projection
f_a: backbone projection head
f f_t:
backbone + projection head
mask embedding: learnable
# mask probability
f.t.1(x) linear projection atteniit,, f_t.Transee (ransormmerett))
importance measure hatpaaree.mpprracccccaac..eeaaee
i acquire threshold
f_a.1(x) linear projection
mask M(e_s, patch_attention importance) # generate mask
ma_aakedd (1-mask)
mask embedding
f_s. Tra.srrrnorreeee.... (e___maske))
def M(embedding, patch_attentine attention, importance):
L,, embedding shape
mask_tokess ceros((B) L))
mask_remove bernoulli (B, L)) p)
maak where (importace aatchiattention, tokeet mask removeemove)
According Eq (3), the tokens of crucial and nonessential regions have the same probability of being masked. As shown in Figure we observe that the random mask strategy may eliminate tokens of crucial regions that are responsible for recognizing objects, resulting indistinguishable semantic features for input images. The random mask strategy is prone to mask crucial regions for images, and suppress the ability of network to recognize objects. It is not suitable to directly apply this strategy to self-supervised vision Transformers and the overall performance would deteriorate if the mask strategy not properly modulated.
iitention-guieed mask strategy. In this section, we propose our attention-guide mask strategy for dynamically controlling the fidelity of masked tokens and thereby decreasing the probability of masking crucial regions in self-supervised Transformer. Meanwhile, our strategy does not increase additional time consumption. Our algorithm shown as Alg. T
Our framework consists of two networks, teacher network fc and student network fa, with the same transformer architecture. Let æ denotes the input image. It is firstly projected to a sequence of 1-d tokens e É1, .... En. and then processed by several self-attention layers. Each self-attention layer 27 owns three groups of embeddings for one token, denoted as Qı(query), K, (key), V,(value). The attention map is calculated as the correlation between the query embedding of class token Qets and key embeddings of all other patches K. is averaged for all heads as Eq (4). We output the attention map from the last layer in the teacher network to guide our strategy.
1
H
Softmax Softmax(Qele KT
Attn
(4)
H h=1
We sort the attention of different patches for each image in ascending order, and take the sorted attention value of 1/ 1/num of total tokens as the threshold т, where num the hyperparameter for selection. This means that the lowest 1/num of total tokens selected the masked candidates. The student model receives the importance of different patches and generates the mask m with probability p. according the Bernoulli distribution Eq (5) prob; refers to the probability of randomly generated.
prob: < p and Attn, {1,
m.
0, otherwise
(5)

---

We use m @ to denote the final masked tokens as Eq 66. Follow the BERT [T0], the masked regions are filled with a learnable mask embedding [MASK]. Our strategy can ensure the patches with the highest scores are always presented (in Figure 2
[MASK],
m4 = 1
(6)
eis
mh = 0
The sitention-guided mask strategy can benefit pre-training models in two ways:
1.
The models utilize contextual information to understand the relationship of different patches, thus preserving the global semantic information of the image while paying more attention to the local details of the image.
Our strategy can avoid masking crucial regions while replacing nonessential regions with the learnable mask embedding, making the models foeus the crucial regions.
3.3 Masked self-supervised transformer
In MLM, mask denotes the complementary set of mask, that is, mask = 1 mask. The loss function of MLM pre- -re-training strategy one data is shown Eq 7 where P(x: (8, mask @ t) the probability the network correctly predicting 4 given the masked token. t indicates the text tokens. That is, the network only restores the masked tokens.
UMLM (0;t,m) (logP(mmss)) @ t/0, mask - E logP(t,)0, mask @ t) fm,=1
There are a sub-sequence .M [1, such that each index independently has probability p of appearing in M, and the overall loss function for training the network is shown as Eq (8). Q is the dataset in Eq (8) In pre-training, the MLM strategy minimizes the overall loss over pre-training dataset.
LMLM (0) = E E (M; t, mask). t~Q M
(8)
However, MLM only predicts the masked tokens according to Eq (8). Different from original MLM, our method encourage the network reconstruct the original input images. We argue that pixel-level restoration task can make the network avoid overfitting patch prediction. therefore enhancing the ability capture the pixel-level information and recovering spatial structure from a finer grain. Since convolution neural networks (CNNs) have the ability of inductive biases, the restoration task adopts CNN as the decoder module, with convolution layers and up-sampling operations alternately stacked. To maximally mitigate the adversarial effect (37, the up-sampling operations are restricted to 2x. Hence, total of operations are needed for reaching the full resolution from 16 x W And the 16.
running mean and running variance of BN are only updated from the global views. The global image decoder consists of the Transformer and decoder. The restoration task is only performed on the student network fa (-). For decoder g(.) with parameters Og, its loss function over image E RH# W and a mask E (0, 1)" as Eq
brestoring (0,, 0y: x. m)
E
Jr f( m))l
(9)
The overall loss function for training the network is shown as Eq (10). and we only need the parameters 8, of student network fe.
Trrestoring (0.) ====Trasae Trretoorngg 0g)=
E
E |r
(10gifs(0.;00 m))l
(10)
x~X
HxW
Therefore, the total loss is shown as Eq III and the MST minimizes the loss over ImageNet 9 dataset in pre-training.
Liotat (0.) X1 LCE(0siz) + 22 Lreetoring (0.iz)
(11)

---

Table 1: Comparison of popular self-supervise learning methods ImageNet. Throughput (im/s)
calculated on single NVIDIA V100 GPU batch size 128. adopts the linear probing of DINO.
Method
Architecture
Parameters
epoch
im/s
Linear
k-NN
Supervised
100
1237
76.5
McCov2 6
800
1237
71.1
61.9
Res50
23
BYOL
13
1000
1237
74.4
64.8
SwAV
800
1237
75.3
65.7
Supervised
300
1007
76.4
SWAV I
300
1007
67.1
SimCLR 4
300
1007
69.0
BYOL [3
300
1007
71.0
MoCov3 7
300
1007
72.5
MOBY 30
300
1007
BYOL 15
800
1007
71.4
66.6
MoCov2
6
DeiT- DeiT-S
21
800
1007
72.7
64.4
SwAV |
800
1007
73.5
66.3
DINO
2
300
1007
75.2
72.8
DINO
300
1007
75.9
72.8
DINO
800
1007
77.0
74.5
Ours
100
1007
75.0
72.1
Ours
300
1007
76.3
75.0
Ours
300
1007
76.9
75.0
Supervised
300
755
81.2
MoBY 30
Swin-T 20
28
100
755
70.9
57.34
Ours
100
755
73.8
66.20
4
Experiments
Several experiments with MST are conducted in this section. We first train self-supervised models with different transformer architectures on ImageNet benchmark, and then examine their transfer capacity with downstream tasks like object detection and semantic segmentation. After that, ablation studies are introduced to elaborate on how our method could achieve state-of-the-an performance.
4.1 Pre-training settings
Dataset and Models Our method is validated on the popular ImageNet 1k dataset [9] This dataset contains 1.28M images in the training set and 50K images in the validation set from 1000 classes. We only use the training set during the process of self-supervised learning. As to models, we choose the classical DeiT-S (26) and popular Swin-T (00 as representatives of all transformer-based architectures. After the backbone, 3-layer MLP with hidden dimension 2048 is added as the projection head. When evaluating our pretrained model, we both use the k-NN algorithm and train a linear classification for 100 epochs former works. Top-1 accuracy is reported.
Training Configurations Our model is optimized by AdamW 20 with learning rate 10-3 and batch size 1024. Weight decay is set to be 0.04. We adopt learning rate warmup [2] in the first 10 epochs, and after warmup the learning rate follows a cosine decay schedule (21. The model uses multi-crop similar to [ and data augmentations similar to [3 The setting of momentum, tempera- ture coefficient, and weight decay follows (2). The coefficient A₁ of basic instance discrimination task is set as 1.0 while the restoration task X2 is set as 0.6.
4.2 Compared with other methods on ImageNet
We compare our method with other prevailing algorithms in Table [ All these methods share the same backbone for fair comparison. Our 300-epoch model achieves 76.9% top-1 accuracy with linear probing. It outperforms previous best algorithm DINO by 1.7% at the same training epochs, and even approaches the performance of DINO with much longer training schedule (77.0% with 800 epochs). should be emphasized that our algorithm relieves the need of extreme long training time for self-supervised learning, and is able to obtain decent result (75.0%) with only 100 epochs.

---

MST is general to be applied with any other transformer-base architectures. Here we use the popular Swin-T for an example. has similar amount of parameters with DeïT-S. Using the same training epochs, MST outpeforrs MoBY by 1.8%, which is self-supervised learning method designed delicately for Swin-T. Swin-T shares the same hyperparameters with DeïT-S, there it can still be improved by further tuning.
4.3 Object detection and instance segmentation
achieves state-of-the-ar under supervised training, it is adopted as the backbone to validate the transfer ability our method in the task of object detection and instance segmentation. We perform object detection experiments with MS COCO [88] dataset and Mask R-CNN detector [5] framework. MS COCO a popular benchmark for object detection, with 118K images training set and 5K images for validation. This dataset contains annotations for 80 classes. Box AP and mask AP are reported on the validation set. As to training settings, we follow the default Ix schedule with 12 epochs. The shorter edges of the input images resized be 800 and the longer edges are limited by 1333 pixels. AdamW optimizer is used, and all hyper-parametest follow the original paper.
Since
In Table 2 we show the performance the learned representation by different self-supervised methods and supervised training. For fair comparison, all these methods pre-trained with 100 epochs. We observe that our method achieves the best results with 42.7% bbox mAP and 38.8% mask mAP. It outperforms the ImageNet supervised model by 1.2% and 0.5%, and MoBY results by 1.2% and 0.5% with the same epoch. The results indicate that MST not only performs well on image classification task, but also performs well on downstream dense prediction task. Therefore it has strong transfer ability.
Table 2: Results of object detection and instance segmentation fine-tuned on MS COCO.
box AP
mask AP
Method
Backbone
Epoch
A Abbbox
A Abbbox
A Apbbox
Apmask
A Ammsk
Apmask
75
300
43.7
66.6
47.7
39.8
63.3
42.7
Supervised
Swin-T
20
100
41.6
64.6
45.4
38.4
61.5
41.0
MoBY
30
41.5
64.1
45.2
38.3
61.0
40.8
DINO Swin-T
20
100
42.2
64.6
46.3
38.7
61.5
41.3
Ours
42.7
65.1
46.7
38.8
61.8
42.5
4.4 Semantic segmentation
provide semantic segmentation framework for standard Vision Transformer. Hence, we adopt the SETR as the semantic segmentation strategy on Cityscapes 8 Cityscapes contains 5000 images, with object categories annotated in pixel level. There are 2975, 500, and 1525 images in training, validation, and testing set respectively. We follow the training config original SETR. For fair comparison, we both use the 300-epoch pretrained model for DINO and our method.
SETR 37
As shown in Table 3 it illustrates the comparison of supervised method, DINO, and our method on this evaluation. Our method achieves the highest mloU 74.7% and mAce 82.35%. outperforms both supervised results (+2.7 71 % mloU and +2.05% mAcc) and DINO pretrained results (+ 108% mloU and +1.03% mAce). Our model is also suitable to transfer for the semantic segmentation task.
Table 3: Results of semantic segmentation fine-tuned Cityscapes.
Method
Backbone
Pre-Epochs
Schedule
mIoU
mAcc
aAcc
Supervised
DeiT-S [26
300
40K
71.33
80.30
94.99
DINO [2]
72.96
81.32
95.37
DeiT-S 26
100
40K
Ours
74.04
82.35
95.42

---

Table 5: The setting of hyper-paramelens for attention- based mask strategy.
Table 4: Linear probe results of different mask strategy (DeiT-S).
0.05
0.10
0.15
Mask Strategy
Top-1 acc (%)
TEum
1
63.2
61.4
60.6
None
73.1
2
73.7
64.4
62.7
Random Mask
63.2
4
73.6
73.6
66.7
Crention-GGider
73.7
8
73.6
73.9
73.6
4.5
Ablation studies
In this section, conduct ablation studies elaborate the effectiveness of our method. All ablation experiments are conducted under 100-epoch setting. By default, only the cls token from the last layer is used to train the linear classifier.
4.5.1 Impact of different mask strategy
Table shows the impact of different mask strategies. We train DeïT-S with random mask strategy [0] attention-guider mask strategy and no mask. For fair comparison, all methods mask with the same probability p. It can be observed that the performance of random mask strategy degrades. This strategy would probably suppress the ability to recognize the object in the images (from 73.1 63.2). Random mask strategy may destroy the tokens of crucial regions of original image which may be indispensable for recognizing object. The masked input may have incomplete or even misleading information. On the contrary, the performance of our attention -uideed mask strategy has steady improvement (from 73.1 to 73.7). Essential regions are mostly preserved, which could be a strong proof of our hypothesis.
4.5.2 Impact of different mask hyper-parameters
T validates the performance of different mask syper-parametery under attention- guideed mask strategy. We sort the attention map of different patches for each image in ascending order, and split the first 1/num patches as the masked candidates. Removing these candidates can force the network learn local features from adjacent patches, therefore strengthening the capacity of modeling local context without destroying the semantics. These candidates are masked according to the probability p. Top-1 accuracy of linear evaluation ImageNet shown in Table 3 When num is set 8, any choice of p can get robust result, which suggests that the last 1/8 patches are relatively safe to be mask candidates.
4.6 Impact of w/o BN
Former work (2) found that the performance will be better if dropping BN in the projection head. We argue that the degradation not caused by BN. As shown in Table 6 normal BN downgrades the performance of baseline model, while the update rule introduced in Section 3.3 helps improve top-
accuracy slightly. This may be due the need to keep consistent structure with the global image Decoder since the image Decoder consists of Conv-BN--e BN-ReLu.
Table 6: Impact of Batch Normalization in projection head.
w/o BN
w/ BN
Baseline
72,4
71.6 73.9
Ours
73.1

---

Conclusion
In this paper, we investigate the two problems of current visual self-supervised learning, namely lack of local information extraction and loss of spatial information. To overcome the above problems, propose a new self-supervised learning method based on transformer called MST. The proposed MST exploits attention attention-guided mask strategy capture the local relationships between patches while also preserving the global semantic information. noted that the attention-guul -uiiddd mask strategy is based on the multi-head self-attention map extracted from the teacher model and does not cause extra computation cost. In addition, a global image decoder is further used under the attention-guider guriddd mask strategy recover the spatial information of the image, which is vital for dense prediction tasks. The proposed method shows good versatility and scalability in multiple downstream visual tasks.
Broader Impact
The MST is to provide pre-trained models with better feature extraction capabilities for popular computer vision tasks. Therefore, the potential positive societal impact of our method:
Improved road safety in autonomous driving by detecting pedestrians.
Safe human robot collaboration in factories with robots taking on risk prone jobs thus saving lives.
Assistive robots for elderly care.
Meanwhile, the potential negative societal impact:
Large scale drone surveillance.
Face recognition leads privacy exposure.
Acknowledgments and Disclosure of Funding
This work was supported by National Natural Science Foundation of China under Grants No.61772527, No.61976210, No.62002357, No.61876086, No.61806200, No.62002356, No.62006230, and No.62076235.
References
[1] Caron, M., Misra, L,, Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning
of visual features by contrasting cluster assignments. In: Advances in Neural Information Processing Systems. vol. 33, pp. 9912-9924 (2020)
[2] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerg- ing properties in self-supervised vision transformers. arXiv: Computer Vision and Pattern Recognition (2021)
[3] Chen, M., Radford, A., Child, R., Wu, J.K., Jun, H., Luan, D., Sutskever, I.: Generative pretraining from pixels. In: Proceedings of the International Conference on Machine Learning (ICML). vol. 1, pp. 1691-1703 (2020)
[4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.0577 (2020)
[5] Chen, T., Korblithh S., Swersky, K., Norouzi, M., Hinton, G.E.: Big self-supervised models
strong semi-supervised learners. In: Advances in Neural Information Processing Systems. vol. 33, pp. 22243-22255 (2020)
[6] Chen, X., Fan, H., Girshick, R.. He, K.: Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.044:
(2020)
[7] Chen, X., Xie, S., He, K.: An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2 arXiv:2104.0200 104.0205 (2021)
10

---

[8] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S.. Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR). pp- 3213-3223 (2016)
[9] Deng, J., Dong, W., Socher, R., Li, L.J,, Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (2009)
[10] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.N.; Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp- 4171-41881 (2018)
[11] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x 16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021)
[12] Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A.,
Jia, Y., He, K.: Accurate, large minibarch sgd: Training imagenet in I hour. arXiv preprint (2017)
[13] Grill, Strub, F,, Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, Pires, B.A., Guo, Z.D., Azar, M.G., Piot, B., Kavukcuoglu, K., Munos, R., Valko, M.: Bootstrap your own latent: A new approach to self-supervised learning. In: Advances in Neural Information
Processing Systems (NeurIPS). vol. 33, pp. 21271-21284 (2020)
[14] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. arXiv preprint 191: 1et::1111..000 1.57222 (2019)
[15] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the International Conference Computer Vision (ICCV) (2017)
[16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceed- ings of the Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
[17] Hjelm, R.D., Fedorov, A., Mavoie-Mar,hildon, S., Grewal, K., Bachman, P., Trischler, A., Bengio, Y.: Learning deep representations by mutual information estimation and maximization. International Conference on Learning Representations (ICLR) (2019)
[18] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Proceedings of the European Conference on Computer Vision (ECCV) (2014)
[19] Liu, S., Li, Z. Sun, J.: Self-emd: Self-supervised object detection without imagenet. arXiv preprint arXiv:201 1.13677 (2020)
[20] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.1400 14030 (2021)
[21] Loshchilov, I., Hutter, F: Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0399 (2016)
[22] Loshchilov, I., Hutter, F:: Decoupled weight decay regularization. In: International Conference on Learning Representations (2018)
[23] Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv: 807 00748 (2018)
[24] Pinheiro, P.O., Almahairi, A., Benmalek, R. R.Y. Golemo, F., Courville, A.: Unsupervised learning of dense visual representations. arXiv preprint Ttt::2011,054999 (2020)
[25] Radford, A.. Narasimhan, K.. Salimans, T., Sutskever, L.: Improving language understanding by generative pre- pre-training (2018)
[26] Touvron, H., Cord, M., Douze, M., Massa, F,, Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. arXiv preprint 12X1 (2020)
[27] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, I.., Gomez, A.N., Kaiser, I.,, Polosukhin, I.: Attention all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems. vol. 30, pp. -098-6008 (2017)
11

---

[28] Wang, X., Zhang, R., Shen, C., Kong, T., Li. L.: Dense contrastive learning for self-supervised visual pre-training arXiv preprint BrXiv:2011.0911 (2020)
[29] Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via con-parametric instance discrimination. In: Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
[30] Xie. Z.,, Lin, Y., Yao, Z.., Zhang, Z., Dai, Q. Cao, Y., Hu, H.: Self-supervised learning with swin transformers, arXiv preprint arXiv:2105.045: .05553 (2021)
[31] Xie, Z., Lin, Y., Zhang, Z.,, Cao, Y., Lin, S., Hu, H.: Propagate yourself: Exploring pixel-level
consistency for unsupervised visual representation learning. arXiv preprint arXiv:2011. 1000000411004 (2020)
[32] Xiong, Y., Ren, M., Zeng. W., Urtasun, R.: Self-supervised representation learning from flow equivariance, arXiv preprint arXiv:2101.06553 (2021)
[33] Yang, C., Wu, Z., Zhou, B., Lin, S.: Instance localization for self-supervised detection pretrain- ing. arXiv preprint arXiv:2102.08311 (2021)
[34] Zhang, X., Maire, M.: Self-supervised visual representation learning from hierarchical grouping.
arXiv preprint arXiv:201 (2020)
[35] Zhao, N., Wu, Z., Lau, R. W.H., Lin, S.: What makes instance discrimination good for transfer
learning. In: International Conference on Learning Representations (ICLR) (2021)
[36] Zhao, Z., Samel, K., Chen, B., Song. L.: Proto: crngram-guideee transformer for program-gune guided
tasks. arXiv preprint urXiv:2110.000
(2021)
[37] Zheng, S., Lu, J., Zhao, H., Zhu, X., Luo, Z.., Wang, Y., Fu, Y., Feng. J., Xiang, T., Torr, P.H.S.,
Zhang, I..: Rethinking semantic segmentation from a sequence-to-sequenee perspective with transformers. arXiv preprint arXiv:2012. 15840 (2020)
12


Reviews:
[{"review_id": "xegT8CEBjNC", "reviewer": "Reviewer_hFAf", "summary": "The paper proposes to add an image reconstruction task to the existing instance discrimination task, as the pre-text tasks for self-supervised learning. Specifically, the self-attention map from the teacher model is used to guide the random mask process, that the attended foreground regions will not be masked. A decoder is used to restore the original image from the masked image.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall idea of combining image reconstruction and instance discrimination as pre-text tasks makes sense. \nHowever, the paper is poorly written. Both the ablations and analysis are insufficient to demonstrate the effectiveness of the proposed method.\n\n- The authors are suggested to provide the experiments of only removing the reconstruction loss. Because the gains could be from the data augmentation itself.\n- More ablation study is needed. For example, random mask strategy with different sampling ratios,  the impact of loss weight $\\lambda_2$.\n- The authors are suggested to demonstrate their method by first reporting the results of the baseline and then adding each small component of the proposed method.\n- L181, what's 'global crops'. \n- The authors should pay attention to the notations. For example, what's 'Q' in Eq.(8), 't' on L167, 'num' on L152, 'd' on Eq.(4), 'prob_j' in Eq.(3). Although I can get the meaning by guess or by experience, they should be clearly explained.\n\n\nOther minors:\n- L20-L21, it's meaningless to report a single number on downstream detection and segmentation tasks. Reporting the gains from the reconstruction task should make more sense.\n- L225, COCO has 80 categories, not 81.\n- Many typos and grammar errors, e.g., L25, L57, L114, L145, L166.\n\n## After Rebuttal\nThe rebuttal and discussion have addressed most of my concerns. Thus I raise my rating from 4 to 6.\nThe authors are suggested to add all these ablations and discussion, and further revise the paper, especially the presentation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors didn't provide a statement of broader impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "tCt53Kjex7W", "reviewer": "Reviewer_KqLC", "summary": "This submission combine the task of mask language modeling (MLM) and a contrastive learning task for self-supervised visual representation learning. The authors claim that due to he preserve of spatial information of an image, the proposed approach is more friendly to the downstream dense prediction tasks. The effectiveness of the approach is verified on ImageNet-1K linear evaluation task, and downstream tasks of COCO object detection and ADE20K semantic segmentation.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\n\n-- While the MLM alone shows inferior performance (iGPT and in ViT), this submission tries MLM as a complementary pretext task to the conventional contrastive method, and demonstrates superior performance than contrastive learning alone. It also proposes a new masking strategy according to the importance of a token. The originality is acceptable for NeurIPS but not outstanding.\n\nQuality\n\n-- The quality is borderline. The combination approach makes sense. But the effects of specific designs are not well ablated, and it is thus unclear to the readers what the real effective components are:\n1) what is the performance by contrastive learning alone? Masking can be also regarded as a kind of data augmentation. Does it help?\n2) what is the performance by mask language modeling alone?\n3) can the two pretext tasks really benefit each other？ what is the effect if \\lambda_2 varies?\n4) Is random masking well tuned? seems it may need very small p, but the results at smaller p such as 0.01 are not provided. Does the poor performance by random masking come from poor contrastive learning or poor mask language modeling.\n5) what would it perform with longer training, e.g. 500 or 800 epochs?\n6) why there are no finetuning results on COCO and ADE using 300-ep pretrained models?\n7) What is the speed compared to contrastive approaches? It seems that the masking strategy requires a pre-compute of the teacher network, and the two branches must be conduct in sequential. Does this sequential computation affect running speed?\n\nClarity\n\n--Overall, the method is presented clear. But there lack a lot of details and hyper-parameters: are the two image views the same cropping/resizing augmentation? it seems to be different as in Line 205, but if the two views are different, how do you estimate \\tau by the teacher network as their patches are not the same?\n\nSignificance\n\n--Satisfied if the components are solidly verified.\n\n\n\n\n ablation is not  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the questions above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "Y5U1YCYfLJu", "reviewer": "Reviewer_VUto", "summary": "The authors propose a new framework for self-supervised learning from natural images. The proposed objective is a linear combination of two previously proposed objectives, namely student-teacher self-distillation (as in DINO), and masked-language modeling as is commonly used in NLP. The masking strategy is guided by the attention masks learned by the transformer backbone. The authors evaluate their model on standard image-SSL benchmarks: ImageNet classification and COCO detection and segmentation.   ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main weakness of the work is its novelty. The proposed objective simply adds a masked-language modeling loss to the objective in DINO. The question is whether the improvement upon DINO is worth the additional complexity, conceptual and otherwise. The improvement upon DINO is on the order of 1% for linear classification on ImageNet, detection on COCO, and segmentation on Cityscapes. This is fine in itself, the issue is that it is unclear whether there is much room to build on the work, as it is itself a simple combination of existing objective. \n\nThe author do highlight the importance of guiding the masking strategy with the learned attention masks however, showing it performs much better than random masking. This constitutes the main technical contribution of the paper, but wether this in itself justifies publication is unclear. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "MOjPn4hYtVd", "reviewer": "Reviewer_2YYa", "summary": "This paper presents a new self-supervised learning method based on a combination of mean-teacher distillation (in this case DINO) and masked patch modelling (like masked language modelling in NLP). They use this to train transformer based vision models and show downstream results for ImageNet classification using linear probing and knn eval, MS COCO object detection and instance segmentation using Mask-RCNN framework and Cityscapes semantic segmentation using SERT framework.\n\nTheir key technical contributions are (a) the overall training framework combining DINO with masked patch modelling. The latter requires a CNN decoder and reconstructs the original global image and not just the masked patches and (b) a smart mask generation scheme based on the attention map of the CLS token which turns out to be crucial for good results. They use the attention maps corresponding to the CLS token to select patches that are not important. The idea is that if important regions of the image are masked out then the model will end up modelling only local statistics.\n\nAt a deeper level, their method is trying to retain spatial structure in the latent representation while capturing global image properties. If I understood correctly, the spatial structure comes from masked patch modelling and global image properties come from DINO.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Both technical contributions listed above are novel to the best of my knowledge. The method is of significance to the community as self-supervised learning now outperforms supervised pre-training and could potentially help the community save on data annotation costs. It is also significant in that a lot of prior self-supervised learning literature learns global image representations which may not adequately preserve spatial structure. The proposed method addresses this issue.\n\nThe performance on established benchmarks is strong. The explanation is mostly clear. Some areas for improving the paper are discussed below.\n\nCitations\n------------\nPlease cite \n* [1] Self-Supervised Representation Learning from Flow Equivariance. Yuwen Xiong, Mengye Ren, Wenyuan Zeng, Raquel Urtasun, Tech report, arXiv, Jan. 2021 which like DenseCL and PixPro also learns dense image representations for downstream detection and segmentation tasks.\n* [2] Unsupervised Learning of Dense Visual Representations. Pedro O. Pinheiro, Amjad Almahairi, Ryan Y. Benmalek,\nFlorian Golemo, Aaron Courville, NeurIPS 2020 is also such a method.\n* [3] https://papers.nips.cc/paper/2020/file/c1502ae5a4d514baec129f72948c266e-Paper.pdf also focus on downstream segmentation and identify the same issue that MoCo learns only global image representations.\n* In line 179, what does adversarial effect refer to? A citation here might help clarify what the authors mean.\n* Numbers for prior methods in the experiments section were in some cases taken from prior papers. Making the source of each number explicit in the table caption will make it easier for the reader to dig out the technical details for each entry. Also especially clarify if any of these numbers are based on you running the opensource code from prior work.\n* In line 201, does \"former works\" refer to DINO?\n* Vision Transformers are a new phenomenon. Unlike ResNet and VGG most readers would not know what DeIT-S and Swin-T are. A 3-5 line summary of these models will be of great value in the related works section.\n\nImportant missing ablations\n---------------------------------------\n* What is the performance of just L_restoring? Does it have to be combined with DINO for this to work? Concurrent work on MLM for Vision Transformer (https://arxiv.org/pdf/2106.08254.pdf) is able to work purely using an MLM objective. Answering this question in an ablation experiment will be super useful to the community.\n* In line 174, the authors mention that unlike original MLM where the loss only looks at masked tokens, they reconstruct the full input image. Please include an ablation to support this design choice. Also clarify how multi scale cropping effects the MLM part of the method. Is the MLM part also exposed to lower resolution crops?\n* In line 178, the authors argue how inductive biases in CNNs inspire them to use CNNs to decode the image. One could alternatively just decode using a projection into 16x16x3 patches and concatenate them together, or decoder into a lower resolution image. Basically how crucial is it to have the CNN inductive biases in this part of the training framework?\n* Masking is done after the linear projection. I found this only when looking carefully at figure 1. Is this important?\n\nDifferences with BERT\n------------------------------\n* BERT randomly replaces the masked positions with the learned [mask] token or with other words. The image equivalent would be to replace chosen patches with either the [mask] token or other patches. However, the proposed method only replaces chosen patches with [mask]. Did the usual BERT approach not work? Sorry I am not familiar with NLP literature, so maybe this is no longer the current practice in NLP either.\n* Do NLP methods also need to be some attention guided masking? If not, then why do those methods not suffer from masking out important words during training?\n\nMinor issues\n-----------------\n* Table 6 caption: Impact of batch normalization \"in projection head\".\n* Line 283, global image decoder is further used \"under\" the attention-guided mask ...\n* Line 167, 't' is a new variable. Did you mean 'e'?\n* Line 196, 50k images not 5k.\n* Line 183, what is $H^n$?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are not discussed. What would be something the community should address moving forward?\n\nSocietal impact is not discussed. The authors state that \"This work dose not present any foreseeable societal consequence.\". The method improves object detection and instance segmentation performance. These datasets prominently include the \"person\" category. Some suggestions are below.\n\nPotential positive societal impact:\n* Improved road safety in autonomous driving by detecting pedestrians.\n* Safe human robot collaboration in factories with robots taking on risk prone jobs thus saving lives.\n* Assistive robots for elderly care.\n\nPotential negative societal impact:\n* Large scale drone surveillance.\n\nThe societal impact of anything in computer vision is massive! I agree that the above areas are not a direct outcome of the proposed method but are a step in that direction.\n\nThere are no error bars. The authors say error bars are in the appendix. But they are missing. Training for 100 epochs makes it super difficult to repeat the experiment 3-5 times. If possible, maybe just run the main proposed setting 3 times at 100 epoch schedule and do knn eval on imagenet. This way we can get some idea of the variance.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
MST proposes to combine the task of mask language modeling with instance discrimination. 

While adding non-negligible complexity to training, MST demonstrate gain of the order of 2% on various standard SSL tasks (ImageNet linear eval, MS-COCO, Cityscape). Additionally, reviewers all agreed that the ablation experiments performed during the rebuttal period clarified the importance of the contributions. Overall, it is unclear if the performance gain fully justifies the extra-complexity of the training pipeline.

Given the positive feedback from the reviewers and the good empirical results, I am in favor of acceptance

Decision: Accept (Poster)