Contextual Image Masking Modeling via Synergized Contrasting without View Augmentation for Faster and Better Visual Pretraining
Authors: Shaofeng Zhang
Feng Zhu
Rui Zhao
Junchi Yan
Venue: ICLR 2023 poster (2023)
Keywords: Mask Image Modeling
Self-supervised learning

Abstract: We propose a new contextual masking image modeling (MIM) approach called contrasting-aided contextual MIM (ccMIM), under the MIM paradigm for visual pretraining. Specifically, we adopt importance sampling to select the masked patches with richer semantic information for reconstruction, instead of random sampling as done in previous MIM works. As such, the resulting patch reconstruction task from the remaining less semantic patches could be more difficult and helps to learn. To speed up the possibly slowed convergence due to our more difficult reconstruction task, we further propose a new contrastive loss that aligns the tokens of the vision transformer extracted from the selected masked patches and the remaining ones, respectively. The hope is that it serves as a regularizer for patch feature learning such that the image-level global information could be captured in both masked and unmasked patches, and notably such a single-view contrasting avoids the tedious image augmentation step required in recent efforts of introducing contrastive learning to MIM (to speedup convergence and discriminative ability). Meanwhile, the attention score from the contrastive global feature can also carry effective semantic clues to in turn guide our above masking patch selection scheme. In consequence, our contextual MIM and contrastive learning are synergetically performed in a loop (semantic patch selection-token alignment contrasting) to boost the best of the two worlds: fast convergence and strong performance on downstream tasks without ad-hoc augmentations, which are verified by empirical results on ImageNet-1K for both classification and dense vision tasks. 

Published as a conference paper at ICLR 2023
CONTEXTUAL IMAGE MASKING MODELING VIA SYN- ERGIZED CONTRASTING WITHOUT VIEW AUGMENTA- TION FOR FASTER AND BETTER VISUAL PRETRAINING
Shaofeng Zhang- Feng Zhu", Rui Zhao 1.3, Junchi Yan 1 Moo Key Lab of Artificial Intelligence, Shanghai Jiao Tong University
Shanghai AI Laboratory SSense Group Limited
(sherrylone, yanjunchi j ftt.juuuuuc.o edu. (zhufeng, zhaorui] @senset ime com
Code: attps://githi com/ http///seerlrcccccceoeoe CCMM
ABSTRACT
We propose a new contextual masking image modeling (MIM) approach called contrasting-aided contextual MIM (ceMIM), under the MIM paradigm for visual pretraining. Specifically, we adopt importance sampling to select the masked patches with richer semantic information for reconstruction, instead of random sampling as done in previous MIM works. As such, the resulting patch recon- struction task from the remaining less semantic patches could be more difficult and helps to learn. To speed up the possibly slowed convergence due to our more difficult reconstruction task, we further propose a new contrastive loss that aligns the tokens of the vision transformer extracted from the selected masked patches and the remaining ones, respectively. The hope is that it serves as a regularizer for patch feature learning such that the image-level global information could be captured in both masked and unmasked patches, and notably such a single-vier -iew contrasting avoids the tedious image augmentation step required in recent efforts of introducing contrastive learning to MIM (to speedup convergence and discriminative ability). Meanwhile, the attention score from the contrastive global feature can also carry effective semantic clues to in turn guide our above masking patch selection scheme. In consequence, our contextual MIM and contrastive learning are synergetically performed in a loop (semantic patch gelection-token alignment contrasting) to boost the best of the two worlds: fast convergence and strong performance on downstream tasks without ad-hoc augmentations, which are verified by empirical results on ImageNet-I K for both classification and dense vision tasks.
INTRODUCTION
sulf-supervised learning (SSL) (Zbontar et al., 2021; Jin et al., 2022; Chen et al., 2020b) has been attracting increasing attention recently in deep learning, due to its label-free property and capability of learning rich holistic representations. Recent SSL methods mainly fall into two classes. Contrastive methods (He et al., 2020; Chen et al., 2020b; Zhang et al., 2021) construct multiple views of the given image to increase the variance and align the representations
these views in latent space. The pair-wise learning paradigm endows strong linear evaluation accuracy of contrastive methods. However, these methods tend to extract global information but ignore local information, which limits their performance downstream dense vision tasks, c.g. detection, segmentation, etc. Besides, the two-view learning paradigm is usually sensitive to the choice of augmentation function (Chen et al., 2020c), batch size (Chen et al., 2020b; Zhang al., 2021) and output dimension (Zbontar et al., 2021; Zhang et al., 2022) etc. With the success recent ViT-based vision backbones, which divide images into several patches, Mask- Miakk-mmae--Mddde
(MIM) methods (He et al., 2021; Fang et al., 2022; Chen et al., 2022) randomly mask some patches and use the self-attention mechanism to
Junchi Yan is the correspondence author. Rui Zhao is also with Qing Yuan Research Institute, Shanghai Jiao Tong University. This work was in part supported by NSFC (62222607), Shanghai Municipal Science and Technology Major Project (2021 (021SH2D7X010)) and SenseTime Collaborative Research Grant.

---

Published as a conference paper at ICLR 2023
recover pixel-wise information from the remaining un-masked patches. These MIM-based methods are shown can obtain a better pre-trained encoder even than contrastive approaches, and especially transfer learning and finetuning tasks thanks to the ability capture
show strong performance local information. However, these methods can suffer from the limited discriminability for global representation with less competitive linear accuracy (Gao al., 2022), slow convergence speed (1,600 epochs for MAE), and GPU-consuming (e.g. with batch size 4096).
Therefore, recent attempts (Wang et al., 2022; Chen et al., 2022; Yi al., 2022) are devoted to combining contrastive learning and MIM. The hope is that both (global) discriminative information (by contrasting) and
information (by MIM) can be captured. Although these approaches show improved convergence speed (in the sense of fewer epochs needed for convergence) compared with using MIM alone, they can hardly further boost the performance of MIM on downstream tasks given even more (e.g. 800, 1600) pretraining epochs. Meanwhile, such direct combing contrasting with MIM can also bring about side effects e.g. the increased sensitivity to data augmentation and more overhead at each training epoch. In this paper, we aim to improve both convergence speed and accuracy by devising a new contextual MIM scheme, which is synergistically aided by contrastive learning. The highlights of our proposed ccMIM are:
1) Novel framework for synergizing MIM and contrastive learning in a close-loop: We propose a novel contextual MIM framework by actively selecting the rich semantic patches as masking patches for reconstruction to improve the MIM learning difficulty, whereby the semantic clue is learned and measured by our devised vision transformer token alignment contrasting loss. As such, the synergizing of the two components can be fulfilled in the first place. In contrast, existing efforts on the combination of MIM and contrasting often perform these two components independently until their weighted loss is finally computed and show limited accuracy improvement (perhaps also partly due to their random patch selection strategy as observed in our ablation study in Sec. 4.3).
2) Cost-effective technical design for contextual MIM:
Under the above framework, we propose to use importance sampling to contextually select the patches with richer semantic information for masking and reconstruction, to increase the learning difficulty and effectiveness. The selection is guided by the attentional semantic score derived from the [CLS] token alignment as contrastive learning between the selected and un-selected patches. Compared to the widely adopted two-view augmentation for contrasting in recent MIM works, our contrasting is efficiently performed within a single view (for speedup MIM training). Moreover, our way of using contrasting also directly guides the MIM learning instead of being two independent components as done in the existing literature.
3) Improvement over MAE: Using ViT-B (Dosovitskiy al., 2020) standard protocol in MIM, ccMIM achieves 83.6%, 84.2% top-1 accuracy with 300 and 800 epochs pre-training, outperforming MAE (He et al., 2021) 0.8% (82.8% for 300 epochs) and 0.6% 83.6% for 1600 epochs) accuracy on ImageNet-1K, respectively. Source code will be made publicly available.
RELATED WORK
In our work, the proposed ccMIM divides image patches into two sets by information density by learning two objective functions. One is aligning the global representations of two sets and the other is reconstructing raw images from the visible set, so we briefly review contrastive learning (align representations of multiple augmented views in latent space) and the MIM-based methods (reconstruction).
Contrastive learning aims to learn instance discriminative representations distinguish an image from the others (Hjelm et al., 2018). This is achieved by pulling together the representations of different views of the same image and pushing away the other images. Thus, most contrastive methods adopt siamese network (He et al., 2020; Grill et al., 2020; Chen & Hc, 2021). To create different views for the same image, well-designed data augmentation functions have been deployed (e.g., those investigated in SimCLR (Chen et al., 2020b)). To increase the number of negative pairs, Moco (He et al., 2020; Chen et al., 2020c) constructs a large queue to store negative representation in memory. Besides, Moco and BYOL use momentum and stop-gradient mechanisms are adopted to prevent degenerate solutions (Tian et al., 2021; Wang & Isola, 2020). To simplify BYO,, SimSiam (Chen & He, 2021) discards the momentum updating and finds dtop-gradient and predictor head are the keys to preventing collapse. MoCo-v3 (Chen et al., 2021) and DINO (Caron et al., 2021) are based on the siamese network and extend MoCo (He et al., 2020) and BYOL (Grill et al., 2020) with Vision
2

---

Published as a conference paper at ICLR 2023
Transformer (ViT) (Dosovitskiy et al., 2020) their model backbones. Although contrastive learning methods provide discriminative features, most of them focus on learning global representations while lacking the spatial sential-eensitive and local representation that can be mitigated by the MIM techniques.
MIM-based methods (He et al., 2021; Xic et al., 2021) learn vision representation by reconstructing the masked patches from the partial observations. Based on the reconstruction objective, these methods can be divided into: pixel-wise reconstruction (He et al., 2021; Xie et al., 2021) and auxiliary features/tokens prediction (Dong et al., 2021; Chen et al., 2022). SimMIM (Xie et al., 2021) and MAE (He et al., 2021) are the first two methods applying mask modeling in the visual domain. They propose to reconstruct the raw pixel values from either the full set of image patches (mask tokens and visible patches for SimMIM) or partially observed patches (visible patches for MAE). Compared with SimMIM, MAE is more efficient because of dropping out large portion of input patches. Inspired by adversarial training (Goodfellow et al., 2014), CIM (Fang et al., 2022) adds perturbations to raw images to enhance robustness for reconstruction. While GreenMIM (Huang et al., 2022) applies divide-and-conquee with hierarchical VIT backbones (Liu et al., 2021). Departure from predicting on RGB space, iBOT (Zhou et al., 2022) and AttMask (Kakogeorgiou et al., 2022) proposed to randomly and selectively (by semantic importance) mask patches and perform contrasts in latent space, respectively.
Combination of Contrasting and MIM has been recently explored with the hope to achieve both fast convergence and good accuracy. For instance, CAE (Chen et al., 2022) randomly partitions the image into two sets of patches and utilizes the visible set reconstruct the masked patches. Then, an alignment objective is added to two sets in latent space. However, the reconstruction loss and alignment loss (as contrasting) are independent of each other and linearly combined for training, which we argue it lacks a synergistic effect between the two modules. Similar issues also exist in other attempts (Yi et al., 2022; Wang et al., 2022) with a línear combination of the two losses, although faster convergence is observed yet with little improvement on the accuracy of downstream tasks. In this paper, we aim to propose more synergetic approach for contrasting- MIM.
3 METHODOLOGY
3.1 PRELIMINARIES
We denote the set of N number of images X E XX and an encoder / for pretraining.
1) Contrastive Learning: The encoders usually quantified as CNNs and ViTs learn global features Z E ND from raw images. The raw images are firstly transformed by certain aug- mentation to generate two views. The key step is using encoders to learn the invariance of two views, i.e. maximizing the representation similarity of the two views (alignment term in (Chen et al., 2020b; Wang & Isola, 2020)). However, directly adding the alignment part will cause degenerate solutions (Tian et al., 2021; Wang & Isola, 2020), which means all the images are encoded to the same and collapsed representation. Two techniques have been developed to avoid degenerate solutions, i.e., adopting stop-gradient to build asymmetric framework (Grill et al., 2020; He et al., 2020; Chen et al., 2020c; Caron et al., 2021; Zhou et al., 2022) and using negative samples (Chen et al., 2020b; He et al., 2020; Chen et al., 2020c), which will be used in the two versions of our method, which are respectively based on: the contrastive forms of SimSiam (Chen & He, 2021) and SimCLR (Chen et al., 2020b).
i) Adopting the asymmetric framework: the objective of SimSiam can be formulated as:
Lvimsiam = 2 sim (StoopGrad(HH (HH(H))))))))))))))))))))))))) sim(StopGrad(HH )(H(A)))
(1)
where StopGrad(() means stop-gradient operations and sim(;-) is the similarity metric function (cosine similarity in this paper). p(-) is the predictor head (two layers MLP), which commonly used in previous asymmetric methods (Chen & He, 2021; Grill et al., 2020). H and H denotes the representations of view A and view B in contrastive space, which can be obtained by H gZ) and y is the projector head composed of two-layer MLP (LeCun et al., 2015).
ii) Adopting the negative samples: accordingly SimCLR (Chen et al., 2020b) adopts the negative- based objective: (InfoNCE (Hjelm et al., 2018)):
sim( 1/1/
Leimelr = -EA,B log N
(2)
Σ
esm(h? b2 + En= eimm(b? (ommbb/////))) ////TT
ビ==1
3

---

Published as a conference paper at ICLR 2023
Importance Ranking
Visible Set
Extract Vis & Mask Feature
Reconstruction
CLS token
Transformer Encoder
Positional Encoding & Encoder Inyer
CLS
Transfornce Decoder
Transfomner Encoder
Mask
Figure Framework of the proposed ccMIM. Stop gradient operation is added the bottom transformer, since the mask set includes richer contextual information (similar to teachers).
where h (h means the representation in contrastive space view (B) of image
2) Masking Image Modeling: the common embodiment of encoders are transformer- models, e.g. vallina VIT (Dosovitskiy al., 2020) and Swin Transformer (Liu et al., 2021). Current MIM- based methods (He et al., 2021; Chen et al., 2022; Xie et al., 2021) basically split raw images into several patches, and randomly select patches to mask. The remaining patches are used for reconstructing the masked patches &M, and the pixel-wise objective is added on masked prediction:
1
Lrecons
(((M) | T(xxss (xp
(3)
where () is the number of masked patches and p means the p-norm (p this paper). T is transformation function (for MAE, T is the identity function: for MaskFeat (Wei et al., 2021), T is the HOG transformation). In this paper, we mainly follow the settings MAE (He al., 2021).
3.2 THE PROPOSED CCMIM: CONTRASTING-AIDER CONTEXTUAL MIM
In our proposed ccMIM, the patches are divided into two sets: masked set (composed of patches with richer contextual information) and visible set (with less contextual information, e.g. background). The visible set is used for reconstruction, and the contrastive objective is regularized on the global tokens of the two sets to help divide patches by contextual information. As illustrated in Fig. ccMIM consists of five components:
Contextual Patch Modeling. Given an input image, ccMIM first divides the patches into visible and masked sets by contextual information. Patches in the masked set will have richer contextual information improve the difficulty of MIM. Both two will be fed the backbone.
Encoder architecture. Encoder backbone extracts latent feature representation for the masked image, which is then used to predict the original signals at the masked area. The learned encoder is expected to be transferable various vision tasks. this paper, following MAE (He al., 2021) 1),, mainly consider ViT (Dosovitskiy et al., 2020) as backbones.
Projector head. The projector head is applied in the contrastive branch, which forces the encoder to learn the global invariance of two sets (masked area and visible area). Specifically, we perform unidirectional alignment (Hjelm et al., 2018; Chen & He, 2021) on two sets (i.e., align visible set with masked set), since the masked set has richer contextual information.
Decoder. The decoder will be applied to the latent feature representation the visible set (with less contextual information) produce one form of the original signals of the masked areas.
Prediction target. This component defines the form of original signals predict. It can be either the raw pixel values or a transformation of the raw pixels (HOG features).
In the following, elaborate the details of approach.
4

---

Published as a conference paper at ICLR 2023
Selecting and dividing strategy. We denote the image as X R Under the ViT- based (Dosovitskiy et al., 2020) MIM paradigm, we first split each image to several patches X E RB*xCx # P. Then, each patch will be mapped to one vector through a linear layer W E R where D the hidden dimension. Inspired by the transformer (Vaswani al., 2017) in NLP, ViT adds one global token to extract global information of cach image. Denote the i-th patch as x, and let Xo be global token. Then, we can calculate the importance the patches x; by:
9.Ki VD
Softmar
EA
(4)
where En means taking the average attention value of multiple heads (Vaswani ct al., 2017). qo means the query vector of the global token, which be calculated by do x, W Q and W q refers the learnable weights. K1 is the key matrix ( (Vaswani et al., 2017) excluding the first row (global token), which is calculated by K XW к and W к is the learnable weights. After obtaining the importance of the patches, we can selectively divide the patches into two sets, i.c. masked set M and visible set V. where M mainly includes the patches with larger importance and V is composed of the remaining patches. The main rationale behind this design is that some background patches (less attention value) may still have a little contextual information, which may be helpful for learning contextual and fine-grained information for downstream tasks (e.g.. fine-grained classification, detection). Denote the probability density function of the importance is f(x and let p(x: be a random distribution. Note (x) ) is the distribution of function f on p. Then, for the expectation of f(x) æ,, we have:
E [ = -. T (x))
p((r) TTx))
(x)dr
(5)
p(x)
where a(x) is the importance sampling weights. For the new distribution
p(z) f()), we have:
(1l))
T(Φ)
| - Frum [(x)
x(x) p(z))
Varzup [(x) p(x)
= Eggup
p(x:)
fxx)2 (xx (xx) p(z) - (BEnn ((x)))
where ((x)
p(n) can be obtained by the observed importance values (calculated by self-attention) and random generator. Then, we can divide the patches into M and V sets by importance sampling.
Visible and masked patches alignment. For the importance masking strategy, one of the assumptions is the global token [CLS] can learn the global information of the whole image. However, for the previous methods like MAE (He et al., 2021), there's 's no regularization added on the [C'LS] token. Hence, directly using [CLS] to partition patches may still be random. To address this problem, we add another objective on the [CLSS token, i.e. contrastive loss (InfoNCE (Chen et al., 2021) or l2 regularization (Chen & He, 2021)). Denote the global representations (CCLS) token) in contrastive space of masked set and visible set as HM and HV, respectively. Then, the NCE-liked objective is:
sim(sG(h M ).bl//T
LNCE = -Ry log
sim(h? n/T Σ Žj=1ijs
where SG(.) is the stop gradient operation. Compared with SimCLR- objective, Eq. 7 only contrasts the visible branch. This design because the masked set mainly includes the important patches, which have richer semantic information than the visible set. Hence, h can be regarded as the teacher guide the h (student). We can also only add the alignment term on the [CT.5] token like SimSiam (Chen & He, 2021) and BYOL (Grill et al., 2020), which can be written as:
Latign
= E sim (Stoppaaaarah p,b)),
(8)
where sim((x y)
x y in this paper and p(-) is the projector head (Chen & He, 2021). ||x] ally 1a
Image reconstruction and contrastive learning. We adopt standard The masked patch recon- struction protocol in ccMIM is similar to MAE, by predicting the pixel values for patches in the masked set. Each element in the decoder's output vector of pixel values representing patch. The
5

---

Published as a conference paper at ICLR 2023
last layer of the decoder is prediction target W xDx ". Then, in line with MAE, the mean squared error (MSE) is computed between the reconstructed and original patches in the pixel space:
1
Lrecons
((MM) || 1
where (MM) the number of elements in the masked set M. The overall objective of ccMIM is:
L {{
Crecome + x Latigns for using asymmetric, for using negative samples
frameeeorr framework
(10)
Crecona + X LNCE.
where À is the using either of the contrastive loss (Eq. 7 and Eq. 8) is enough for regularizing the [CLS]] token. We mainly use LNCE by default in our main experiments which we find can be even more stable. The comparison of the two objectives is given in the ablation study.
to balance contrastive loss and reconstruction loss. Note that only
Remark, Note that though we also adopt the linear loss as in Eq. 10, while the reconstruction proce- dure is aided by the contrasting in a loop, which makes our method differs from other combination methods (Chen et al., 2022; Yi et al., 2022; Wang et al., 2022) without such synergetic connections.
3.3 METHODOLOOGY COMPARTSON
Note in Table 6 in Appendix, we compare recent SSL pretraining methods. In particular, here we elaborate on the connections the most related methods CAE, MST, Repre, and SeMMAE.
Relation to CAE (Chen et al., 2022). Both CAE and ccMIM partition the raw image into two sets, i.., visible set and mask set. However, CAE divides them randomly, which is similar MAE (He et al., 2021), limiting their performance. ccMIM partitions the two sets by importance sampling. Moreover, ccMIM adds another contrastive loss to regularize [CLS] token help partition.
Relation to MST (Li et al., 2021). Both MST and ccMIM use contrastive loss and reconstruction pretext task and selective strategy. However, the motivations of MST and ccMIM are completely different. MST mainly relies on the contrastive loss to learn the invariance of two views and reconstruction loss only works a little bit. Besides, they mask the unimportant patches, which only slightly influences their performance (Note that the random mask strategy will cause about 10% Top-1 accuracy drop). In contrast, ccMIM masks the important patches to enhance the difficulty of reconstruction. Besides, ccMIM does not require multiple views from tricky augmentation.
Relation to Repre (Wang et al., 2022). Repre is the other method using both contrastive loss and reconstruction loss. However, Repre also requires two views from tricky augmentation and the main gain is coming from the contrastive objectives. Besides, the reconstruction branch is similar to VAE (Kingma & Welling, 2013), which takes all patches input without any masking strategy.
Relation to SemMAE (Li et al., 2022). SemMAE is another concurrent work about attentive masking. Both ccMIM and SemMAE mask patches with richer contextual information increase the difficulty of reconstruction. SemMAE uses backbone pretrained by iBOT (Zhou et al., 2022) to help select contextual regions, where iBOT takes 1600 extra epochs for pretraining, which is computation-cost. In contrast, ccMIM directly adds alignment loss on [CLS) token to learn global information select masked regions, which does not require extra information and time.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset. We perform self-supervised pre-training on the ImageNet-IK (Deng et al., 2009) training set, as commonly used in SSL. (He et al., 2021; Zhang et al., 2021). includes 1 classes which well-balanced in distribution and the images contain an iconic view of objects. We evaluate our pretrained model on the ImageNet- Ik validation set, as well as on MS-COCO (Lin et al., 2014) object detection and segmentation, ADE20K (Zhou et al., 2017) segmentation, and other small-scale classification datasets (Van Horn et al., 2018).
Backbone. We the standard ViTs architecture (Dosovitskiy al., 2020). It has several blocks composed of multi-head self attentttentonn (MHSA) (Vaswani et al., 2017), MLP, and LayerNorm (Ba al., 6). ceMIM adds one (CL.S] token at both masked set and visible set (shared). The decoder

---

Published as a conference paper at ICLR 2023
Table 1: Linear and finetune top-1 accuracy on ImageNet-IK, means using the powerful multi-modie
DALL-E (Reddy et al., 2021) tokenizer in pre- training. I means methods from arXiv preprint to date.
Method
Model
#Params
PT Eps.
Linear
FT Eps.
FT Acc. (%)
Scratch, Dail (louvron al., 2021a)
VITB
86M
300
81.8
Training from scrach
Scratch, MAE
VIT-B
86M
300
82.3
Scratch, Swin
Swin-B
88M
300
83.5
Supervised Pre-training
Supervised, SimMIM
Swin-B
88M
300
100
83.3
Supervised. SimMIM
Swin-L
197M
300
100
83.5
Moco (Chen ol., 2021)
VTTBB
86M
800
76.5
100
83.2
Contrastive Leanring
DINO (Caron al. 2021)
VIT-B
86M
400
77.3
100
83.3
iBOT (2hou et 2022)
VIT-B
86M
1600
79.5
100
83.8
(HC 2021)
VIT-B
86M
300
62.4
100
82.
at,
VIT-B
86M
1600
67.8
100
83.6
eWei 2021)
VTT-B
83.6
ВETT (Bao 2021)
WT-B
86M
300
37.6
100
83,0
CAE (Chen al., 2022)
VII-B
86M
300
64.2
100
83.3
Masked Image Modeling
CAE (Chen ct 2022)
VIT-B
86M
800
68.3
100
83.6
CIM (Pang 2022)
VII-B
300
100
83.3
CIM (Fang et 2022)
VIT-B
86M
300
100
83.1
ConMIM (Yi ct al., 2022)
VII-B
86M
300
100
83.5
ConMIM (Yi al., 2022)
WTT-B
100
83.7
ccMIM (Ours)
VIT-B
86M
300
66.9
100
83.6
ccMIM (Ours)
VIT-B
86M
800
68.9
100
84.2
of ccMIM follows the settings of MAE (He et al., 2021), which includes two-layer MHSA blocks. At the evaluation stage, the average embedding of patches is used for finetuning and linear probing.
Evaluation protocols. i) Linear probing widely used as a proxy of pretraining quality evaluation for SSL (Chen et al., 2020b; He et al., 2021; Chen et al., 2022). It learns a linear classifier over the image-level representation output from the pretrained encoder by using the labels of the images, and then tests the performance on the validation set. ii) Fine tuning is often used to evaluate the backbone in reconstructed- based methods (He et al., 2021; Chen et al., 2022). We use the same hyper- pprammer with MAE, i.e., 100 epochs with learning rate 1e-3 and 1024 batch size. iii) Downstream tasks across different datasets and tasks (detection, segmentation, mne-grained classification) are also used to evaluate the transferability of the pre-trained encoder.
4.2
MAIN RESULTS
Linear probing and finetune on ImageNet-11. Table shows the linear probing and fine-tuning accuracy on ImageNet-Ie Ik dataset. We mainly compare our method with wansformer-based contrastive learning methods (Caron al., 2021; Chen et al., 2021) and MIM-based (He al., 2021; Xie et al., 2021) methods. For 300 epochs pretraining, our ecMIM outperforms baseline MAE +0.8 % top-1 accuracy. For 800 epochs pretraining. ccMIM achieves 84.2% top-1 accuracy, outperforms MAE +0.6% accuracy with 1600 epochs pretraining. For the concurrent method ConMIM, it uses two views by non non-trivial augmentation in the pre-training step, i.e., multiple views are required to enable contrastive objectives. Although ccMIM also has the contrastive objective, it only requires a single view. Besides, ConMIM gets 83.7% accuracy with 800 epochs pretraining, only improves 0.2% accuracy on the basis of 300 epochs pretraining. The reason may be that the contrastive objective in ConMIM only improves the convergence speed but not the final performance. In contrast, ccMIM gets 84.2% (0.6% higher than MAE and ccMIM with 300 epochs pretraining) top-1 accuracy in 800 epochs pretraining, which may be because the contrastive objective in ccMIM is to help select contextual patches and the final gain is coming from recovering patches with richer contextual information (ConMIM randomly masks patches with less contextual information). Besides, we also find ccMIM surpasses MAE +4.5% and +2.1 % top-1 accuracy under linear probing protocol. We guess that's because ccMIM adds the global-discriminttie objective (see ablation study in Sec. 4.3).
Object detection and segmentation on MS-COCO. Setups. Following CAE (Chen et al., 2022), we adopt Mask R-CNN (He et al., 2017) that produces bounding boxes and instance masks simulta- neously, with the ViT as the backbone (See the supplementary file for training details). We apply the same object detection system to the methods. We report the box AP for object detection and the mask AP for instance segmentation. We utilize multi-scale training and resize the image with the size of the short side between 480 and 800 and the longer side no larger than 1,333. The batch size is 32. For ViT-S, the learning rate is 3e-4, the layer-wise decay rate is 0.75, and the drop path rate is 0.1. For VIT-B, the learning rate is 3--4, the layer-wise decay rate is 0. 75, and the drop path
7

---

Published as a conference paper at ICLR 2023
Table 2: Object detection and instance segmentation MS-COCO. Mask R-CNN is adopted and trained with the Ix schedule. All the results are based on the same implementation for object detection and instance segmentation. 'Epoch' refers the number of pretraining epochs on ImageNet- IK. "SSL?" means whether use self- sulf-super pre-training.
Detection
Segmentation
Method
Backbone
Epoch
#Param.
SSL?
AP
AP
AP
Apm
APm
AP
DeiT (Touvron al., 2021a)
VIT-S/16
300
22M
43.1
65.2
46.6
38.4
61.8
40.6
MoCo V3 (Chen ct al., 2021)
VIT-S/16
300
22M
43.3
64.9
46.8
38.8
61.6
41.1
BEIT (Bao sl., 2021)
VIT-S/16
300
22M
35.6
56.7
38.3
32.6
34.2
CAE (Chen
2022)
VIT-S/16
300
22M
44.1
64.6
48.2
39.2
61.4
42.2
ccMI (Ours)
VIT-S/16
300
22M
44.3
64,9
48.3
39.5
61.4
42.4
Deil (Touvron ol, 202
VIT-B/16
300
86M
46.9
68.99
51.0
41.5
65.5
44,4
Maco V3 (Chen ct 2021)
VIT-B/16
300
86M
45.5
67.1
49.4
40.5
63.7
43.4
DINO (Caron
2021)
VIT-B/16
300
68.6
41.5
BET (Bao 2021)
VIT-B/16
300
86M
39.5
60.6
43.0
35.9
57.7
38.5
BETT (Bao 2021)
VTT-B/16
800
86M
42.1
63.3
46.0
37.8
60.1
40.6
MAE (He ct 2021)
VIT-B/16
300
86M
45.4
66.4
49.6
40.6
63.4
43.7
MAE (Hc ot al., 2021)
VIT-B/16
1600
86M
48.4
69.4
53.1
42.6
66.1
45.9
ConMIM (Yi et 2022)
VIT-B/16
800
43.6
CAE (Chen al., 2022)
2022)
VIT-B/16
300
86M
48.4
69.2
52.9
42.6
66.1
45,8
CAE (Chen et al., 2022)
VIT-B/I6
800
86M
49.8
70.7
$4.6
43.9
67.8
47.4
ccMIM (Ours)
VIT-B/16
300
86M
49.1
69.8
53.3
43.1
66.4
46.2
(Ours)
VIT-B/16
800
71.2
47,9
Table 3: Semantic segmentation on ADE20K dataset. #Epochs means pretraining epochs on
ImageNet-1K 1K. #Views means used views for pretraining. #lter the iterations for finetuning.
Method
Backbone
AEpochs
Supervised Self-supervieed
aAce
Moca V2 TChen et 20202]
ResNe
TPN
200
41k
35.8
17.6
45.1
DeiT (Touvron
2021a) 1o)
VIT-B/16
FPN
300
160k
47.0
Moco (Chen et al., 2021)
VIT.B/16
UperNet
300
160
46.2
56.4
82.4
DINO (Caron
2021)
VIT-B/16
UperNet
300
160h
45.0
55.1
82.1
(He
UperNer
MAE (Ile et 2021)
VIT-B/16
UperNet
1600
160k
46.9
57.4
82.7
CAE
al., 2022)
VIT-B/16
UperNet
300
56.6
CAE (Chen al., 2022)
VIT-B/16
UperNet
800
160%
47.2
57.9
83.0
ccMIM (Ours)
VIT-B/16
UperNet
300
160%
46.5
56.9
82.8
ocMIM (Ous)
VIT-8/16
UperNet
800
160k
47.7
58.3
Table 4: Transfer learning on fine-grained datasets. All the methods are under the same settings.
Dataset
ClusterFit
SNCA+
Grafit
DINO
Sup.
iBOT
BeiT
MAE (1600
ccMIM (800 eps)
IN INaturalistguss
49.7
69.2
69.8
72.6
73.2
74.6
72.3
75.4
75.6
IN Naturralista119
73.7
74.5
75.9
78.6
77.7
79.6
79.2
80.5
81.0
Flowers- 102
98.2
98.2
98.8
98.4
98.9
98.0
99.0
99.2
rate is 0.2. We train the network with the schedule 12 epochs with the learning rate decayed by 10x at epochs 9 and 11. Results. Table 2 shows the results on COCO detection and segmentation. ccMIM outperforms previous MIM- based methods with large range. Specifically, with 300 epochs pretraining, ccMIM surpasses MAE +3.7 % Apo and +2.5 % and Apm detection and segmenta- tion, respectively. With 800 epochs pretraining, ccMIM outperforms MAE +2.5 Apo AP36 and +1.9% and Apmkk We also find ccMIM surpasses CAE +0.5% Apbb and +0.6 % Apmk with 800 epochs pretraining, while only surpasses +0.2% Ap and +0.3% % Apmnk on ViT-S backbone, which we guess that's because the reconstruction in ccMIM (attentive selection) is more difficult than in CAE (random). Hence, ccMIM has better improvements when using larger backbones.
Semantic segmentation on ADE20K. Setups. We evaluate semantic segmentation performances of the tre-trained models on ADE20K (Zhou et al., 2017), which includes 150 fine- ine-grained semantic categories and 25k training data. In line with iBOT (Zhou al., 2022), we report the results by three metrics: mean intersection of union (mloU) averaged over all semantic categories, all pixel accuracy (aAcc), and mean class accuracy (mAcc). We pretrain ccMIM with 800 epochs and finetune it in FPN (Lin et al., 2017) with batch size 16. For VIT-B, the layer-wise decay rate is 0.65, and the drop path rate is 0.1. For other methods, we download their pretrained models and finetune them in setting for fair comparisons and Table 3 shows the finetune results. The proposed ecMIM gets state-of-the. accuracy among all the contrastive- based and MIM-based methods. Specifically, for 800 epochs pretraining, ccMIM outperforms MAE and CAE +1.7 % and +0.5% mloU, respectively.
8

---

Published as a conference paper at ICLR 2023
Transfer classification on other datasets. In line with previous SSL methods, conduct set of experiments to evaluate the transfer accuracy on une-grained classification datasets (iNatural- ist (Van Horn et al., 2018) and Flowers). We mainly compare 2020), + (Wu et al., 2018), Grafit (Touvron et al., 2021b), iBOT (Zhou et al., 2022), Dei'' (Tou- vron al., 2021a) and MAE (He al., 2021). We finetune the pretrained encoder on iNaturalist datasets in 360 epochs with the learning rate 3c-4. For all SSL methods, we choose ViT-B/16 for comparisons. The results in Table 4 show that ccMIM obtains the highest accuracy compared with previous supervised and self-supervised methods. Specifically, ccMIM gets 0.5%, 0.2% and 0.2% higher accuracy than MAE on iNaurralistgugg, iNaturalistguıs and Flowers- 102, respectively.
method with ClusterFit (Yan et al.,
4.3
ABLATION STUDY
Regularization on classification token and effect of X. Dif- ferent from ConMIM (Yi et al., 2022), ccMIM does not require multiple views for contrasting and the main improvement is due the contrasting -aseed contextual MIM. To verify this, we try different À on both Lalign and LNC in experiments. In detail, we try X from 0 (no regularization) to 1.0 by fixing training epochs as 400 and other hyper-paramete as le-5, batch size as 1024). We report top-1 linear probing accuracy in Fig, 4.3. We find that LNCE and Latign has sim- ilar properties (they gets best results when À = 0.2). When À > 0.2, with the increase of X, the accuracy drops. When A >> 0.8, ccMIM with LNCE gets lower accuracy than MAE, which we think this phenomenon is because the gain of ccMIM is from contextual masking, and contrastive loss is only used for selection semantic-richer patches, but not for direct improvement. When À = O, there's 's no regularization on [CLS] token, leading the lower accuracy than À 0.1 and X 0.2.
4:2
(e.g., learning rate
53.
3ta
350
0. 02
04 05 08 20 values
Figure 2: Impact of different A.
Table 5: Ablation study of the dividing strategy.
Dividing strategy. We further conduct a group of experiments to analyze the effect of different dividing strategies, i.e., random dividing, block-
Dividing strategy
Lincar
100
epochs
Finetune
Lincar
400 epochs
Finetune
random
33.6
79.4
63.1
82,9
wise dividing, importance-basse dividing, and
block-wise
32,9
79.2
62.0
82.8
anportance-samlling-aan based dividing. For block- wise masking, we follow BEiT (Bao et al., 2021).
importance
31.1
78.8
66.8
83.6
importance sampling
41.1
80.5
67.4
83.8
Specifically, block of image patches is masked each time. For each block, we set the minimum number of patches to 16. Then we randomly choose an aspect ratio for the masking block. We repeat the above two steps until obtaining enough masked patches. We distributed pretrain ccMIM on 32 GPUs in 100 and 400 epochs with 1024 batch size. We report linear and finetune accuracy in Table 5. In our experiments, the block-wise dividing strategy gets a bit lower accuracy than random dividing, which is in line with previous methods MAE (He et al., 2021) and SimMIM (Xie et al., 2021). We also find with 100 epochs pretraining, the importance-bast based dividing strategy gets much lower accuracy than random and block-wise methods. However, for 400 epochs pretraining, importance-bast based methods get higher accuracy than random and block-wise methods. We think that's ' because the importance-based strategy will slow down the convergence speed (patches with richer contextual information are more difficult to reconstruct). paporranccesamlling-bbsee
method outperforms other dividing methods in both 100 and 400 epochs pretraining. We guess that's because, for the first few epochs, the variance of attention value of different patches is small. Thus, when adding the random sampling operation, ccMIM also divides some patches with less contextual information in the masked set. Then, the reconstruction task is not too difficult, leading a faster convergence speed than the importance-based strategy. Then, for 400 epochs pretraining, (CT.S) token could easily divide patches with richer contextual information into the masked set, leading the higher accuracy (+4.3% %) than random and block-wise methods.
5 CONCLUSION
We have presented a novel contextual MIM framework dubbed ccMIM, whereby masked patch selection is aided by a contrast between the masked patches and unmasked ones. The reconstruction difficulty is increased as the masked patches are more semantically rich than random sampling as done in peer MIM methods. The resulting combined approach shows notable performance gain in both convergence speed and accuracy compared to basclines on public benchmarks across different tasks from classification dense tasks including detection and segmentation.

---

Published as a conference paper at ICLR 2023
REFERENCES
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint /007.6660.066 2016.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint wrXi::2/06.08254 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020a.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020b.
Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self-supervised representation learning, arXiv preprint 2rXiv:2202.33026. 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Xiaoyi Dong. Jianmin Bao, Ting Zhang. Dongdong Chen, Weiming Zhang. Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. arXiv preprint 1xXx::211111277: 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2020.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. arXiv preprint 2rX:v:2202.03382, 2022.
Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao. Convmae: Masked convolution meets masked autoencoders. arXiv preprint 22X5v:2005.03892 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27, 2014.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning, NewrIPS, 2020.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
R Devon Hjelm, Alex Fedorov, Samuel Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 1808 06670, 2018
Karan Grewal, Phil Bachman, Adam
10

---

Published as a conference paper at ICLR 2023
Lang Huang, Shan You, Mingkai Zheng, Fei Wang, Chen Qian, and Toshihiko Yamasaki. Green hierarchical vision transformer for masked image modeling. arXiv preprint arXiv:2205.13515, 2022.
Peng Jin, JinFa Huang, Fenglin Liu, Xian Wu, Shen Ge, Guoli Song, David A Clifton, and Jie Chen. Exppetation-maxinizzan contrastive learning for compact video-and-languaal representations. In NewrlPS, 2022.
Ioannis Kakogeorgiou, Spyros Gidaris, Bill Psomas, Yannis Avrithis, Andrei Bursuc, Konstantinos Karantzalos, and Nikos Komodakis, What to hide from your students: Attention-guided masked image modeling. In ECCV, 2022.
arXiv preprint Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 1rXivv1312.61 14, 2013.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 2015.
Gang Li, Heliang Zheng, Daqing Liu, Bing Su, and Changwen Zheng. Semmae: Semantic-guide masking for learning masked autoencoders. arXiv preprint arXiv:2206. 10207, 2022.
Zhaowen Li, Zhiyang Chen, Fan Yang. Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng, Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation.
NeurIPS, 2021.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Tsung-Y Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pp. 10012-10022, 2021.
Mr Murahari Reddy, Mr Sk Masthan Basha, Mr M Chinnaiahgari Hari, and Mr N Penchalaiah. Dall-e: Creating images from text. UGC Care Group Journal, 2021.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICMI, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021a.
Hugo Touvron, Alexandre Sablayrolles, Matthijs Douze, Matthieu Cord, and Hervé Jégou. Grafit: Learning fine-grained image representations with coarse labels. In ICCV, 2021b.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In CVPR, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
Luya Wang, Feng Liang, Yangguang Li, Wanli Ouyang, Honggang Zhang, and Jing Shao. Repre: Improving self-supervised vision transformer with reconstructive pre-training. arXiv preprint arXiv:2201.06857. 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICML, 2020.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:21 22.09/33, 2021.
Zhirong Wu, Alexci A Efros, and Stella Yu. Improving generalization via scalable neighborhood component analysis. In ECCV, 2018.
11

---

Published as a conference paper at ICLR 2023
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint arXiv:211 2021.
Xueting Yan, Ishan Misra, Abhinav Gupta, Deepti Ghadiyaram, and Dhruv Mahajan. Clusterfit: Improving generalization of visual representations. In CVPR, 2020.
Kun Yi, Yixiao Ge, Xiaotong Li, Shusheng Yang, Dian Li, Jianping Wu, Ying Shan, and Xiaohu Qie. Masked image modeling with denoising contrast. arXiv preprint arXiv:2205.004: 096666 2022.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICML, 2021.
Shaofeng Zhang, Feng Zhu, Junchi Yan, Rui Zhao, and Xiaokang Yang. Zero-cl: Instance and feature decorrelation for negative-free symmetric contrastive learning. In ICLR, 2021.
Shaofeng Zhang, Lyn Qiu, Feng Zhu, Junchi Yan, Hengrui Zhang, Rui Zhao, Hongyang Li, and Xiaokang Yang. Align representations with base: A new approach to self-supervised learning. In CVPR, 2022.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In CVPR, 2017.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, Image BERT pre-training with online tokenizer. In ICLR, 2022.
12

---

Published as a conference paper at ICLR 2023
Table Methodology comparison for recent SSL pre- treiirainngg methods. CE, NCE. and l2 mean cross-entropy loss, infoNCE loss, l distance loss and l2 distance loss, respectively. 'Aug twice' means augmenting the images twice to learn the invariance the two augmented views. "Extra knowledge" means whether use extra data or pretrained model.
Method
Extra knowldgg??
Aug twice?
Contrastive
Reconstruct
Mask siralegy
Objectives
DINO (Caron et al., 2021)
x
CE CE
IGPT (Chen al. 2020a)
x
X
random
Beif (Bao et al., 2021)
X
X
X
random
CE
MST (Li at. 2021)
X
V
selective
CE & h
iBOT (Zhou et al, 2022)
X
V
random
CE
MAE (Ile 2021)
X
X
X
rundom
SimMIM (Xic et ol., 2021)
X
X
X
random
PoCo (Dong ol., 2021)
X
X
random
CE
Repre (Wang et al. 2022)
X
NCE &
MaskFeat (Wei et 2021)
X
X
EXX
rundom
lg
CIM (Fing al., 2022)
X
random
CE
CAE (Chen 2022)
random
l/
ConMIM al,, 2022)
X
random
NCT.
SemMAR (1i 2022)
X
selective
12
ccMIM (Ours)
X
selective
la
Mask catio
Her (mundom))
Iter
Ber 500
Importacce niking
Viuba
0.50
10
20
30
Spachs
Mask illustration
(b) Reconstruction loss v.s. Epochs
Figure 3: Left figure illustrates the patches division strategy with different masking ratios, where visible set mainly includes semantic-less patches to increase the difficulty reconstruction. Right plot shows the Crecons in Eq. 9 v.s. epochs.
A METHODOLOGY COMPARISON
To better position our approach, we list recent contrastive and mask image modeling methods in Tab. 6.
B MASK I.LUSTRATION
For better understanding of our work, we illustrate how ccMIM divides the patches into visible and masked set in Fig. 3(a). We visualize four pretrained models, i.e., ccMIM with iteration 0, 800 and importance-basse dividing in Sec. 4.3. When Iter = O, the [CLSS token is randomly initialized, ccMIM randomly divides the patches into masked set and visible set. After 800 epochs pretraining, ccMIM could divide patches with richer contextual information into mask set, and the remain patches with less contextual information are divided into visible set to increase the difficulty of reconstruction. We further illustrate reconstruction loss curve of MAE and ccMIM in first 50 epochs pretraining in Fig. 3(b). We can find the loss values ccMIM is higher than MAE, which demonstrates the larger difficulty for reconstruction of ccMIM than MAE.
13

---

Published as a conference paper at ICLR 2023
C IMPLEMENTATION DETALS
line with MAE, we use A as optimizer with the base learning rate 1.5c-4 and weight decay 0.05. We the balance ratio 0.2 and use LNCE in Eq. as regularization loss as default. For computational tractability, we set batch size as 1024, distributed pretraining on 32 NVIDIA 1080Ti GPUs. Similar to previous methods in SSL (Xie et al., 2021), adopt cosine Ir scheduler and only use the random resized crop for augmentation.
Pre-training hyper-parameters.
14


Reviews:
[{"review_id": "UFLeW3T14B", "reviewer": "Reviewer_tKmv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "1, Adopt importance sampling to select the masked patches with richer semantic information for reconstruction, instead of random sampling as done in previous MIM works.\n2, Propose a new contrastive loss that aligns the tokens of the vision transformer extracted from the selected masked patches and the remaining ones.\n3, The proposed contextual MIM and contrastive learning are synergetically performed in a loop with fast convergence and strong performance on downstream tasks without ad-hoc augmentations", "summary_of_the_review": "The method provides a better version of the original MAE with better performance and faster convergence.", "strength_and_weaknesses": "Strength\n1, Novel framework for synergizing MIM and contrastive learning in a close-loop.\n2, Improvement over MAE: Better than MAE with shorter training epochs.\nWeaknesses\n1, Do you need to extract the mask set feature? Will that slow down the training?\n2, Why the s means the importance of the patches? Will the global token have global information at the beginning of the training?\n3, Can you compare the real training time? Your algorithm seems to be much more complicated than the original MAE. (thus More time over one epoch).\n4, Which token do you use for the downstreaming classifcation task?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EYn5xHlH6r", "reviewer": "Reviewer_pWuU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces self-supervised representation learning, which combines two well-known existing methods (i.e., contrast learning and mask autoencoder learning) in a way that appropriately obtains their respective advantages. Contrastive learning with Simese networks is faster in optimization compared to mask auto-encoder training, but is relatively poor at extracting spatially local information. On the other hand, masked encoder learning has performance advantages by making the learned representation have the ability to extract spatially local information, but the optimization is very slow. Since the main structural difference between the two existing methods is the preparation of the input, the proposed method uses visible patches and mask patches as two separate sets of patches required for contrastive learning. This method also proposed a novel mask region separation strategy that makes learning more difficult to obtain stronger representations. This strategy uses importance sampling using weights defined similarly to the attention weights for each value in ViT's self-attention module while input is the additional global token instead of local patches. Experiments have demonstrated that the learned representation through the proposed method (called ccSSL) was effective in terms of classification ability in linear probing task and finetuning task, and transferring ability in multiple downstream tasks.", "summary_of_the_review": "As this paper introduces a new and effective expression learning method by appropriately utilizing the advantages of the two well-known methods, I am on the positive side and I think it will attract the attention of many researchers.", "strength_and_weaknesses": "* Strength\n1. The advantages and disadvantages of the two existing representation learning methods are well described and appropriately utilized in the method design.\n2. The mask region separation strategy is novel and appears to be as effective as in tab 5. (The ablation experiments shown in tab 5 are well designed to support the use of the strategy.)\n3. The learned representation through the proposed method was effective in all the evaluation tasks which are widely used in verifying the effectiveness of the representation learning.\n\n\n* Weaknesses\n1. Some informative ablation studies may be necessary. E.g., the effect of various masking rate.\n2. Detailed information on how the [CLS] token is constructed is lacked or insufficient.\n3. It is necessary to experimentally confirm the advantages and disadvantages of the two existing representation learning methods and whether the proposed method has acquired each of these advantages well.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1XlFXVcIxy2", "reviewer": "Reviewer_ZDzp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a contrasting-aided contextual masked image modeling framework, termed ccMIM. The main motivation behind this paper is to use CLS tokens to mask semantic patches so that the reconstruction target could be more difficult. To make the CLS token contains more semantic context, ccMIM introduces a global alignment loss (InfoNCE or Self-Distillation) on the CLS token. Lots of experiments are conducted to show the superiority of ccMIM.\n", "summary_of_the_review": "This paper presents ccMIM. The core component behind ccMIM is to sample semantic-less patches for harder self-supervised learning objectives. Despite better results being achieved, the core contribution of ccMIM is not fully studied (see weakness).\n", "strength_and_weaknesses": "### Strengths\n\n1. The motivation makes sense. The paper writing is easy to understand.\n2. Results on image classification / object detection / semantic segmentation all show improvements.\n\n### Weaknesses\n\n1. As shown in Fig.1, the importance score of each patch is computed with the CLS token before feeding into ViT-Encoder. This means the CLS token has no interaction with image patches. So is the importance ranking generated from the CLS token really reliable? More descriptions and visualizations should be added for clear understanding.\n2. Bringing contrastive learning to MIM is already well-studied in many works. The more important component in this paper is the importance-based sampling strategy. However, sampling semantic-less patches for harder self-supervised learning objectives is suitable for not only MIM but also contrastive learning methods. It's important to conduct such a sampling strategy to contrastive learning methods (e.g., BYOL [Grill et al., 2020] or SimCLR [Chen et al., 2020b]) and see if consistent performance gains could be achieved.\n3. ccMIM feeds both visible patches and masked patches into ViT encoder. The extra computation costs are not mentioned in the paper.\n4. Improvements compared to previous methods are a bit marginal.\n5. Experiments on larger architectures (e.g., ViT-L) could bring more authority.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster