Architecture-Agnostic Masked Image Modeling -- From ViT back to CNN
Authors: Siyuan Li
Di Wu
Fang Wu
Zelin Zang
Lei Shang
Baigui Sun
Xuansong Xie
Stan Z. Li
Venue: Submitted to ICLR 2023 (2023)
Keywords: Self-supervised Learning
Vision Transformer
Representation Learning
Unsupervised Learning

Abstract: Masked image modeling (MIM), an emerging self-supervised pre-training method, has shown impressive success across numerous downstream vision tasks with Vision transformers (ViTs). Its underlying idea is simple: a portion of the input image is randomly masked out and then reconstructed via the pre-text task. However, the working principle behind MIM is not well explained, and previous studies insist that MIM primarily works for the Transformer family but is incompatible with CNNs. In this paper, we first study interactions among patches to understand what knowledge is learned and how it is acquired via the MIM task. We observe that MIM essentially teaches the model to learn better middle-order interactions among patches and extract more generalized features. Based on this fact, we propose an Architecture-Agnostic Masked Image Modeling framework (A$^2$MIM), which is compatible with both Transformers and CNNs in a unified way. Extensive experiments on popular benchmarks show that our A$^2$MIM learns better representations without explicit design and endows the backbone model with the stronger capability to transfer to various downstream tasks for both Transformers and CNNs.

Under review as a conference paper at ICLR 2023
ARCHITECTURE-AGNOSTA MASKED IMAGE MODELING - FROM VIT BACK TO CNN
Anonymous authors Paper under double-blind review
ABSTRACT
Masked image modeling (MIM), an emerging self-supervised pre-training method, has shown impressive success across numerous downstream vision tasks with Vi- sion transformers (ViTs). Its underlying idea is simple: portion of the input image is randomly masked out and then reconstructed via the pre-text task. However, the working principle behind MIM is not well explained. and previous studies insist that MIM primarily works for the Transformer family but is incompatible with CNNs. In this paper, we first study interactions among patches to understand what knowledge is learned and how it is acquired via the MIM task. We observe that MIM essentially teaches the model to learn better middle-order interactions among patches and extract more generalized features. Based on this fact, we propose an Architecture-Agnosie Masked Image Modeling framework (A MIMM) which is compatible with both Transformers and CNNs in unified way. Extensive experi- ments on popular benchmarks show that our A MIM learns better representations without explicit design and endows the backbone model with the stronger capability to transfer to various downstream tasks for both Transformers and CNNs.
INTRODUCTION
Supervised deep learning with large-scale annotated data has witnessed an explosion of success in computer vision (CV) (Krizhevsky et al., 2012a; He et al., 2016) and natural language processing (NLP) (Vaswani al., 2017). However, a large number of high-quality annotations are always available in real-world applications. Learning representations without supervision by leveraging pre-text tasks has become increasingly popular.
In CV. early self-supervised learning approaches (Zhang et al., 2016; Doersch et al., 2015; Gidaris al., 2018) aim to capture invariant features through predicting transformations applied to the same image. However, these methods rely vision ad-hoe heuristics, and the learned representations are less generic for downstream tasks. Recently, contrastive learning-based approaches (Tian et al., 2020; Chen al., 2020b; He et al., 2020) have witnessed significant progress, even outperforming supervised methods on several downstream tasks (Chen et al., 2020c; Grill et al., 2020; Zbontar et al., 2021). More recently, inspired by masked autoencoding methods (Radford et al., 2018; Devlin et al., 2018) in NLP, Masked Image Modeling (MIM) methods (Bao et al., 2022; He et al., 2022; Wei et al., 2021; Xie al., 2021b) have brought about new advances for self-supervised pre-training on CV tasks. The transition from human language understanding to NLP masked autoencoding is quite natural because the filling of missing words in sentence requires relatively comprehensive semantic understanding. In analogy, humans can understand and imagine masked content by visually filling the missing structures an image containing occluded parts.
Different from contrastive learning, which yields a clustering effect from pre-training by pulling similar samples and pushing away dissimilar samples, MIM pre-training methods have not been extensively explored in the context of the expected knowledge learned or how this knowledge is ac- quired. Existing works mainly focus on improving downstream tasks performance via explicit design such as trying different prediction target (Wei et al., 20,1), adopting pre-trained tokenizer (Zhou et al., 2021), 1), utilizing complex Transformer decoder (He et al., 2022) or combining with contrastive learning (El-Nouby et al., 2021). Moreover, the success of existing MIM methods is largely confined to Vision Transformer (ViT) structures (Dosovitskiy et al., 2021) since it leads to under-performing performance to directly apply mask token (Devlin et al., 2018) and positional embedding to CNNs.

---

Under review as a conference paper at ICLR 2023
In this work, we carry systematic experiments and show that MIM as pre- mee-traning task essentially teaches the model to learn better middle-order interactions between patches for more generalized feature extraction regardless of the underlying network structure. Compared to the local texture features learned by low-order interactions between patches, more complex features such as shape and edge could be extracted via middle-order interactions among patches. The interaction of patches could be considered as information fusion via both the convolution operation of a CNN and the self-attention mechanism of Transformer. That is to say, CNN and Transformer should both benefit from better middle- order interactions with MIM as the pre-text task.
To bridge the gap of MIM in terms of network architectures based on our extensive experimental Masked Image Modeling framework (A MA' that
analysis, we propose an Architecture-Agnssii focuses on enhancing the middle-order interaction capabilities of the network. Specifically, we mask the input image with the mean RGB value and place the mask token at intermediate feature maps of the network. addition, we propose loss in the Fourier domain to further enhance the middle-order interaction capability of the network. Our contributions are summarized as follows:
We conducted systematic experiments and showed the essence of MIM is to better learn middle-order interactions between patches but not reconstruction quality.
We proposed a novel MIM-based framework dubbed A MIM that bridges the gap between CNNs and Transformers. We are also the first to perform MIM on CNNs without adopting designs native to ViT's that outperforms contrastive learning counterparts.
Extensive experiments with both Transformers and CNNs on ImageNet- 1K and public benchmarks for various downstream tasks show that our method achieves performance improvement pre-trained representation quality than offte-of-the-arl methods.
RELATED WORK
Contrastive Learning. Contrastive learning learns instance-level discriminative representations by extracting invariant features over distorted views of the same data. MoCo (He al., 2020) and SimCLR (Chen et al., 2020b) adopted different mechanisms to introduce negative samples for contrast with the positive. BYOL (Grill et al., 2020) and its variants (Chen He, 2020; Chen et al., 2021) further eliminate the requirement of negative samples to avoid representation collapse. Besides pairwise contrasting, SWAV (Caron et al., 2020) clusters the data while enforcing consistency between multi-augmented views of the same image. Barlow Twins (Zbontar et al., 2021) proposed to measure the cross-correlation matrix of distorted views of the same image to avoid representation collapsing. Meanwhile, some efforts have been made top of contrastive methods to improve pre-training quality for specific downstream tasks (Xie et al., 2021a; Xiao et al., 2021; Selvaraju et al., 2021; Wu al., 2022). MoCo. V3 (Chen et al., 2021) and DINO (Caron al., 2021) adopted ViT (Dosovitskiy et al., 2021) in self-supervised pre-training to replace CNN backbones.
Autoregressive Modeling. Autoencoders (AE) are a typical type of network architecture that allows representation learning with no annotation requirement (Hinton & Zemel, 1993). By forcing denoising property onto the learned representations, denoising autoencoders (Vincent et al., 2008; 2010) are a family of AES that reconstruct the uncorrected input signal with a corrupted version of the signal input. Generalizing the notion of denoising autoregressive modeling, masked predictions attracted the attention of both the NLP and CV communities. BERT (Devlin et al., 2018) performs masked language modeling (MLM) where the task is to classify the randomly masked input tokens. Representations learned by BERT as pre-training generalize well to various downstream tasks. For CV. inpainting tasks (Pathak et al., 2016) to predict large missing regions using CNN encoders and colorization tasks (Zhang et al.. 2016) to reconstruct the original color of images with removed color channels are proposed to learn representation without supervision. With the introduction of Vision Transformers (ViT) (Dosovitskiy et al., 2021; Liu et al., 2021), iGPT (Chen et al., 2020a) predicts succeeding pixels given a sequence of pixels as input. MAE (He et al., 2022) and BEiT (Bao et al., 2022) randomly mask out input image patches and reconstruct the missing patches with ViTs. Compared to MAE, MaskFeat (Wei et al., 2021) and SimMIM (Xie et al., 2021b) adopt linear layers as the decoder instead of another Transformer in MAE. MaskFeat applied HOG the prediction target instead of the RGB value. Other research endeavors (EI-Nouby et al., 2021; Zhou et al., 2021; Assran et al., 2022; Akbari al., 2021; Sameni al., 2022) combine the idea of contrastive learning
2

---

Under review as a conference paper at ICLR 2023
Prstilopp
BoaNedo Razbee Prichltep
Ireenctoon Srregh Order
RoNeei brirnction Steaahh Order
ANORN PT Den
Mod +IT (MD
Tap Accemayy
kaaion -IT Kobkic
Aualon +I Buth
krabn -fT pum
destb IT (he) Fondss
6.1
0.3
3.6
6.
0.1
0.1
0.3
0.0 0.
0.1 1.6
Opd
(a)
(b)
(c)
(d)
Figure 1: (a)(b): Robustness against different occlusion ratios of images is studied for both VIT-S and ResNet-50 under different experimental settings (see Section 3.1). (c)(d:: Distributions of the interaction strength j(m) are explored for both ViT-S and ResNet-50 under different experimental settings. The label indicates the pre-training method + for random weight initialization. Appendix B provides more results and implement details.
augmentation used, random stands
(CL) with MIM. SplitMask (EI-Nouby et al., 2021) proposed to use half of the image pixels to predict the other half while applying InfoNCE loss (Van den Oord et al., 2018) across the corresponding latent features. MSN (Assran et al., 2022) matches the representation of an image view containing randomly masked patches and the original unmasked image. Similarly, iBOT (Zhou et al., 2021) adopts the Siamese framework to combine self- oeff-istillattinn with MIM. Moreover, Data2Vec (Baevski et al., 2022) proposed framework that applies the masked prediction idea for either speech, NLP. or CV. However, most MIM works are confined to ViT architectures, recently proposed CIM (Fang et al., 2022) adopts the output of pre-trained tokenizer as target and takes the prediction of a frozen BEiT as input to the encoder as a workaround to enable MIM on CNNs. In this work, we propose A with no components native to ViTs adopted to perform MIM with ViTs and CNNs.
3 INTRIGUING PROPERTIES OF MASKED IMAGE MODELING
3.1 Is MIM BETTER IMAGE A
Compared to CNN, Transformers gain tremendous performance improvement with carefully designed image augmentation techniques such as BaandAug(Cubula et al., 2020), CutMix et al., 2019) and random erasing(Zhong et al., 2020). Random erasing(Zhong et al., 2020) randomly removes part of the image and replaces it with Gaussian noise, while Cutmix randomly removes part of the image and replaces the corresponding region with patch from another image. Similarly, as in most MIM pre-training tasks, some image patches are masked out and replaced with a learnable mask token. Noticing the resemblance of the masking operations, we hypothesize that MIM a pre-training task and masking- -bsed data angmentations enhance the network's 's robustness towards occlusion, enabling the network with a more generalized feature extraction ability. To verify our hypothesis, we design an occlusion robustness test. Let X E Hx be an input image and E RCC be its corresponding label, where C is the class number. Consider a classification task y = fxx where I denotes neural network, the network is considered robust if the network outputs the correct label given an occluded version of the image x', namely y = f(x'). For occlusion, we consider the patch-based random masking as adopted in most MIM works (He et al., 2022; Xie et al., 2021b; Wei et al., 2021). In particular, we split the image of size 224 x 224 into patch size 16 x 16 and randomly mask MM patches out of the total number of N patches. The occlusion ratio could then be defined as Af We conduct experiments on ImageNet- and CNN with different settings. We choose VIT-S (Dosovitskiy et al., 2021) and 5es et al., 2016) as the network architecture. Robustness is compared under the following settings: (i) random weight initialization with no image augmentation applied: (ii) random weight initialization with different image augmentations applied; (iii) MIM pre-training as weight initialization with and without image augmentations applied. In Fig. 1, we report the average top-1 accuracy across five runs trained with different settings under various occlusion ratios. Fig. (aa) and 1(b) show that both MIM and patch- patch-removing alike augmentations significantly improve model occlusion robustness for both ViT-S and ResNet-50. Nevertheless, MIM yields more robust feature extraction than adopting augmentations. Although MIM and patch-removing alike augmentations share similar masking mechanisms, MIM explicitly forces the model to learn the interactions between patches in order to reconstruct missing patches enabling more robust feature extraction. Comparing Fig. ((a) and 1(b), the convex trend of accuracy from ViT-S indicates better robustness than the concave trend from ResNet-50. The self-attention mechanism of ViTs is able to model the interactions between patches with high degrees of freedom compared to CNNs constrained by convolution priors. We claim that
N
(IN- (Krizhevsky et al., 2012b) for both Transformer
3

---

Under review as a conference paper at ICLR 2023
(a) Original Image
(b) 30% Masked
(c) 50% Masked
(d) 70% Masked
(e) Inpainting Mask
Figure 2: (a) Four patches (i, k; 1) interact with each other and forms contour or edge pattern of the fox for image categorization. (b) Image with 30% masking ratio. Masked patches i and & interact with neighboring patches and to predict the missing patches. (c) Image with 50% masking ratio. Masked patches are forced to interact with middle-order interactions for the MIM task. (d) Image with 70% masking ratio. Masked Patch i interacts with longer-range patches j and k forming an edge pattern. (e) A typical masking pattern for existing inpainting tasks.
the success of MIM on ViTs can be seen resonance in terms of better putch interactions imposed by MIM while supported by the self-attention mechanism of ViTs.
3.2 MIDDLE-ORDER INTERACTIONS FOR GENERALIZED FEATURE EXTRACTION
Next, we show that MIM essentially enables better middle-order interactions between patches. Note that existing MIM works adopt a medium or high masking ratio (Xie et al., 2021b; He et al., 2022) (e.g., 60% or 70%, see Fig. 2) during pre-training, and in these settings, the pairwise interactions between patches are under a middle-size context measured by the order m. Early inpainting work based on CNN (Pathak al., 2016) resembles MIM but attracts little attention due to the much inferior performance to contrastive learning methods. The inpainting task adopts the masking strategy as illustrated in Fig. I(C), which masks full large region instead of random small patches. Such masking mechanisms ignore patch interaction and focus only on reconstruction leading to poor, learned representation quality. To investigate whether MIM makes the model more sensitive to patch interactions of some particular orders, we resort to the tool of multi-order interactions introduced by (Deng et al., 2022; Zhang et al., 2020). Intuitively, m interactions of patches refer to inference patterns (deep features) induced from m number of patches of the original image in the input space. With a small value of m (low-order interactions), the model simply learns local features such as texture. Formally, the multi-order interaction (mm) (i, is to measure the order of interactions between patches i and j. We define ((m)) (i, to be the average interaction utility between patches i and on all contexts consisting of patches. indicates the order of contextual complexity of the interaction. Mathematically, given an input image x with a set of n patches N = {1...., n2 (e.g., an image with n pixels), the multi-order interaction ((m) (im) is defined as:
(mm) (i.j) (GC.
where (i, - S(S (i.j}) - (SSUU)) - J(S (i)) + S(S). J(S) indicates the score of output with patches in N \ S kept unchanged but replaced with the baseline value (Ancona et al., 2019), where the context S C N. See Appendix B.2 for details. To measure the interaction complexity of the neural network, we measure the relative interaction strength J(m) of the encoded m-th order interaction as follow:
j(m) Ex I (m)) (i.jl=)
(2)
Em Even Bi / (mm (i,
where Q the set of all samples and S m 2 n - 2. ((m is the average value over all possible pairs of patches of input samples. ((m) is normalized by the average value of all interaction strengths. (mm then indicates the distribution (area under curve sums up one) of the order of interactions of the network. In this work, use j(m) as the metric evaluate and analyze interaction orders of the network with MIM pre- pre-training. We conduct experiments on IN-100 with image size 224 x 224 and use ViT-S (Dosovitskiy al., 2021) and ResNet-50 (He et al., 2016) as the network architecture. We consider a patch of size 16 x 16 as input patch. For the computation of (mm), we adopt the sampling solution following previous works (Deng et al., 2022: Zhang et al., 2020). As can be seen from Fig. I(c) that VIT-S with random weight initialization tends learn simple interactions with few patches (c.g., less than 0.05n patches) while MIM pre-trained models show a stronger interaction for relative middle-order (from 0.05n to 0.5m). Similarly, observed from (d)) MIM pre-trained
4

---

Under review as a conference paper at ICLR 2023
Ground Truth FFT
Predicted
Mask Taken
Ground Truth t Lags Patic
Ground Truth Expe Predicted Palch
fro
RGB Mean
Spato Domain
Frequerccy Domain
Decoder
Dacoder
Intermediste Stage a Layer
Encoder
Encoder VIT, Swin, CNN
(e.g. VIT, Swin)
Mask
Mask
(a) Existing MIM Framework
(b) Our Proposed A'MMM
Figure 3: An illustration comparison between the existing MIM framework and our proposed framework. For the existing MIM framework, the input image is patchfied into sequence of patches without overlapping with masked patches that are replaced with learnable mask tokens. The sequence then input the Transformer encoder. The Lapa is applied between the ground truth patches and the reconstructed patches from the decoder in the spatiotemporal domain. Our proposed framework uses the mean RGB value of the image instead of the mask token in the input space. We then add learnable mask token onto the intermediate feature map of layer-/ of stage-s of the encoder instead of replacement in the input space. The encoder could either be of the Transformer or the CNN family. In addition to the Lepa. we adopt frog in the Fourier domain to enhance the encoder to learn more middle-order interactions. Specifically, we apply DFT on both the ground truth image and the predicted image and then use Mean square error (MSE) to measure the difference.
ResNet-50 enhances the middle-order interactions from OnIn to 0.55n compared to random initialized models. Stronger middle-order interactions form more complex features such as shape and edge compared to local texture features learned from low-order interactions (Naseer et al., 2021).
APPROACH
We propose a generic MIM framework following two design rules: (a) No complex or non-generic
designs adopted to ensure compatibility with all network architectures. (b) Better middle- order interactions between patches for more generalized feature extraction. Figure 3 highlights
the difference between proposed framework and existing MIM frameworks in terms of three key components: masking strategy, encoder/decoder architecture design and prediction targets.
4.1 ARCHITECTURE AGNOSTIC FRAMEWORK
Mask Where Middle-order Interactions Occur. Existing works (El-Nouby et al., 2021; 1; He et al., 2022; Xie et al., 2021b; Wei et al., 2021) adopt the masking strategy where the input image is divided into non-overlapping patches, and random subset of patches masked. MAE utilizes a Transformer as a decoder and takes only the visible patches into the encoder. Masked tokens appended to the decoder reconstruct the masked patches. SimMIM (Xie al., 2021b) and MaskFeat (Wei et al., 2021) utilize a fully connected layer as the decoder and feed the mask token into the encoder together with the visible patches. The mask token (Devlin et al., 2018) is token-shared learnable parameter that indicates the presence of missing patches to be predicted. Despite different choices of decoder structures, the mask token is either placed the input to the encoder or the decoder. Mathematically. the masking process of MIM is defined as Imost = z @ (1 M) + T O M, where M is the random occlusion mask, and represents the learnable mask token. Such masking at the patch embedding layer aligns with the attention mechanism of Transformers, which is robust against occlusion. However, masking at the stem layer undermines the context extraction capability of CNN, which relies on local inductive biases. Moreover, masking at input stages of the network leads to low- order interactions. Thus, we propose to mask intermediate features where the output feature contains both the semantic and spatial information and the mask token encode interactions with the medium number tokens. More concretely, our masking operation defined as = = z + T & D(M), where z' the intermediate feature map of at layer-/ in the Transformer encoder (or for stage-l in CNNs) and D(-) is the corresponding down-sampling function of the occlusion mask.
5

---

Under review as a conference paper at ICLR 2023
Filling Masked Tokens with RGB Mean. It worth noting that existing works directly replace the occluded patches with the mask token in the input space or after the patch embedding (Bao et al., 2022; Xie al., 2021b). In contrast, the input to the encoder and add the mask token onto the intermediate feature maps of the encoder. The masking mechanism originates from NLP where languages are of high-level semantics and do not require low-level feature extraction as image processing. The introduction of a zero mask at the early stages of the network where low-level feature extraction happens is harmful in terms of feature extraction. From the view of Fourier domain, the RGB mean is the DC component of images. It not only brings about minimum local statistics variation caused by the masking operation but also forces the network to model rather more informative medium frequencies instead of filling the mask patches with blurry color blocks of low frequencies. The proposed masking strategy is generic to both convolution and self-attention in that it accommodates low-level to semantic-level feature extraction.
use the average RGB value to fill the occluded patches as
4.2 MIDDLE-ORDED INTERACTIONS FROM FOURIER PERSPECTIVE
Current works (El-Nouby et al., 2021; He et al., 2022; Xie et al., 2021b) adopt raw RGB values as the prediction target. However, raw pixels in the spatial domain are heavily redundant and often contain low-order statistics (Bao et al., 2022; Wei et al., 2021; Zhou et al., 2021). MaskFeat (Wei et al., 2021) adopts the Histogram of Oriented Gradients (HOG) as the prediction target outperforming MAE and SiMMIM. HOG is discrete descriptor of medium or high-frequency features which captures shape patterns based on middle-order interactions. ViTs and CNNs have low-pass and high-pass filtering properties, respectively (Park & Kim, 2022; 2021). VITs and CNNs have certain frequency bands that they each cannot model well, and both cannot model middle-order interactions well (detailed in Appendix B.3). The observation of the medium frequency descriptor HOG improves middle-order interactions and leads to the hypothesis that learning medium frequencies would help the model learn more middle-order interactions. Given RGB image R3xHxW the discrete Fourier transform (DFT) of each channel is defined as:
h= w= W
F F(u.o))
Σ
Σ
xth, w)e 2 11 +
A=1
In addition to the common MIM loss in the spatial domain Cspa, we propose & frog in Fourier domain:
c=3== W=
Direg= ΣΣ
w(u, |DFF(xErre @ M de(xgred o (1 M)) DFT(= (4)
where pred is the predicted image, de(-) is detach gradient operation, and w(u, v) is the frequency weighting matrix. w(u, v) enables both ViTs and CNNs model features of medium frequencies rather than local textures and noise corresponding to high frequencies. Inspired by Focal Frequency loss (Jiang et al., 2021), we define adaptive w(u, v) as follows:
(5) where a is a scaling factor, and we set a = 1. Fig. B.3 verifies that Eq. (5) allows the model to learn previously ignored frequencies (mostly the medium frequency components). Note that L introduces negligible overhead by using Fast Fourier Transform (FFT) algorithms with O(n log n) complexity to achieve DFT. The overall loss function of M is then defined as:
(u, v) = |DFFT(xpee pred C M + det(z)red) O (1 M)) - DFT((=))1
L Espa + AL
(6)
where Capa = Iatred 2|| O M and is loss weighting parameter. We set to 0.5 by default.
EXPERIMENTS
5.1 PRE-TRAINING SETUP
We adopt ResNet-50 (He et al., 2016) and Vision Transformer (Dosovitskiy al., 2021) (ViT- S/16 and VIT-B/16) as the backbone. We pre-train on ImageNet-1K 1K (IN-1K) training set with AdamW (Loshchilov & Hutter, 2019) optimizer with basic learning rate of 1.5 10-4 adjusted by

---

Under review as a conference paper at ICLR 2023
cosine learning rate scheduler and batch size of 2048. The input image size is 224 224 with patch size of 32 x 32. We use a random masking ratio of 60%. By default, the learnable mask tokens placed at stage-3 in ResNet-50 and layer-5/layer- in ViT-S/ViT-B, respectively. We adopt linear prediction head as the decoder (Xie et al., 2021b). MIM+ indicates adopting HOG as supervision and using the MLP decoder with depth-wise (DW) convolution. Our experiments implemented on OpenMixup (Li et al., 2022) by Pytorch and conducted on workstations with NVIDIA V100 GPUs. We report the average results of 3 trials for all experiments and bold and underline indicate the best and the second- best performance. See Appendix A for detailed pre- traininnnng settings.
5.2 IMAGE CLASSIFICATION ON IMAGENET-IK
Evaluation Protocols. We first evaluate the learned representation by end-to-end fine-tuning (FT) and linear probing (Lin.) protocols on IN-IK. For evaluation on CNN, we adopt RSB A2/A3 (Wight- man et al., 2021) training settings for fine-tuning on ResNet-50, which employs LAMB (You et al., 2020) optimizer with cosine scheduler for 300/100 epochs. For the linear probing setting ResNet-50, we freeze the backbone features and train a linear classifier with an initial learning rate of 30 and batch size of 256 following MoCo (He al., 2020). For evaluation Transformer, the fine-tuning as MAE (He et al., 2022), which uses DeiT' (Touvron et al., 2021) augmentation setting, an AdamW optimizer for 100-epoch training, and adopt layer-wise learning rate decay of 0.65 following (Bao et al., 2022). See Appendix A for detailed evaluation configurations.
employ
ResNet-50. We compare the proposed A MIM with classical self ceff-supervise learning methods (Inpainting (Pathak et al., 2016), Relative-Loo (Doersch et al., 2015), and Rotation (Gidaris et al., 2018)), contrastive learning (CL), and MIM methods with 100/300 pre-training epochs. We modified MIM methods to them ResNet-50: the learnable mask token is employed the encoder of BEiT (Bao et al., 2022), Data2 Vec (Bacvski et al., 2022), and SimMIM (Xie et al., 2021b) after the
Table 1: ImageNet- 1K linear probing (Lin.) and Table 2: ImageNet-IK fine-tuning (FT) top- ac- fine-tuning (FT) 1 accuracy (%) of ResNet-50. curacy (%) of ViT-S and ViT-B models.
Muuti-crop augmentation.
Our modified version for CNN.
Method
Supervision
ViT-S
VIT-B
Method
Fast Pre-training
Longer Pre-training
Epochs
FT
Epochs
FT
Epochs
Lin.
FT (A3)
Epochs FT (A3) FT (T
DetT (Sup.)
Label
300
79.9
300
81.8
Py Torch (Sup.)
18.8
MoCoV3
CL
300
81.4
300
83,2
Inpainting
70
40.1
78.4
300
78.0
DINO
CI.
300
81.5
400
83.6
Relative-Loc
38.8
77.8
300
77.9
BETT
DALLE
800
81.3
800
83.2
Rotation
70
48.1
77.7
300
78.2
SimCT
100
64.4
78.5
500
78.8
79.9
SplitMask
DALLE+CL
300
81.5
300
83.6
McCoV2
100
66.8
78.5
78.8
79.8
iBOT
Momentum
800
82.3
400
84.0
BYOL
100
68.4
78.7
400
78.9
80.1
MAE
RGB
1600
83.6
SWAVt
100
78.9
79.0
80.2
MaskFeat
HoG
800
84.0
Barlow Twins
100
67.2
78.5
300
78.8
79.9
Data2 Vec
Momentum
800
84.2
McCoV3
100
68.9
78.7
300
79.0
80.1
SimMIM
RGB
300
81.7
800
83.8
BETT
CAF
DALLE
300
81.8
800
83.6
Data2Veet
100
43.2
78.0
CIM
BETT
300
81.6
300
83.3
MAE
100
37.8
77.1
mc-BEiT
VQGAN
800
84.1
SiMMIM
100
47.5
78.2
300
78.1
79.7
CIM
300
78.6
80.4
BootMAE
Momentum
800
84.2
APMIM
100
48.1
78.8
300
78.9
80.4
A
RGB
300
82.2
800
84.2
M
78.9
79.0
80.5
A MIMM
HoG
300
82.4
800
84.5
Table 3: Performance of object detection and Table 4: Performance of object detection and seman- semantic segmentation tasks based on ResNet-tic segmentation tasks based on ViT-B on COCO and 50 on COCO and ADE20K.
ADE-20K.
Method
Fpochs
COCO
ADF-20K
Method
Supervision
Epochs
COCO Aplor
ADE-20K
Apl
Apminsk
mIoU
Aprusk
mloU
PyTorch (Sup.)
120
38.2
33.3
36.1
Deil (Sup.)
Label
300
47.9
42.9
47.0
SimCLR
800
37.9
33.3
37.6
MoCoV3
CL
300
47.9
42.7
47.3
McCoV2
400
39.2
34.3
37.5
DINO
CL
400
46.8
41.5
47,2
BYOL
400
38.9
34.2
37.2
BETT
DALLE
300
43.1
38.2
47.1
SwAV
800
38.4
33.8
37.3
iBOT
Momentum
400
48.4
42.7
48.0
SimSiam
400
39.2
34.4
37.2
MAE
RGB
1600
48.5
42.8
48.1
Balow Twins
800
39.2
34.3
37.3
MaskFeal
HoG
800
49.2
43.2
48.8
SimMIM
300
39.1
34.2
37.4
SimMIM
RGB
800
48.9
43.0
48.4
CIM
300
38.0
CAE
DALLE
800
49.2
43.3
48.8
A 'MM
300
39.8
34.9
38.3
A
RGB
800
49.4
43.5
49.0
7

---

Under review as a conference paper at ICLR 2023
Zero mask
mean mask
+T(r)
+L
Cucumber Fox
Raw image
Masked image
SimMIM
Masked image
MMM
Figure 4: Visualizations of predicted results from SimMIM (middle) and our A MIM (right) based on ViT-S pre-trained 300-epochs on IN-IK. Notice that T(!*) denotes the mask token T to the optimal layer-5 in ViT-S. We ablate the proposed components by adding them to the baseline. Compared to results from SimMIM, reconstruction results of the modified baseline (Cope) with the RGB mean mask relieves grid-like artifacts; adding the mask token T(l*) further improves the smoothness; using the proposed freg helps the model to capture more informative details and contours.
stem (the output feature of 56 x 56 resolutions); the encoder of MAE randomly selects 25% from 56 x 56 output features of the stem as unmasked patches and takes the reorganized 28 x 28 patches as the input of four stages. As shown in Tab. 1, our approach achieves competitive performance with state-of-the-art contrastive-based methods under 100-epoch RSB A3 fine-tuning. Note that MIM methods see fewer training samples per epoch than CL methods (40% vs. 200% patches) and usually require longer pre-training epochs. Based on longer fine-tuning evaluation using RSB A2, our method (300-epoch) outperforms contrastive-basel methods with even fewer training epochs. Meanwhile, A MIM also improves the baseline SimMIM (+0.8%) and the concurrent work CIM (+0.4%) in terms of RSB A3 fine-tuning for the longer pre-training. Besides, we also report the linear probing accuracy in the fast pre-training for reference, although our main focus is learn representations with better fine-tuning performances. The linear probing performance of our method lower than contrastive-ban based methods, still improves the baseline by 0.6%.
VIT. We then evaluate A MIM based on VIT-S/B in Tab. 2. We list the supervision target used by various pre-training methods the second column of Tab. DALL-E (Ramesh et al., 2021) and VQGAN (Esser et al., 2021) are pre-trained image tokenizers, while momentum refers to the momen- tum encoder. Our approach outperforms current state-of-the-ae methods with complex supervision, e.g., SplitMask (MIM with CL combined), iBOT (complex teacher-student architecture), and CIM (pre-trained BEiT as supervision). Based on ViT-S/B, MIM improves the baseline SimMIM by 0.5%/0.4% with RGB as supervision and 0.7% 7.7%% with the HOG feature supervision.
5.3 TRANSFER LEARNING EXPERIMENTS
Object detection and segmentation on COCO. To verify the transferring abilities, we benchmark CL and MIM methods on object detection and segmentation with COCO (Lin et al., 2014). For evaluation on CNN. we follow the setup in MoCo, which fine-tunes Mask R-CNN (He al., 2017) with ResNet-50-C4 backbone using 2x schedule on the COCO train2017 and evaluates on the COCO va/20/7. Results in Tab. indicate that our approach (300-epoch) outperforms contrastive- based methods with longer pre-training (+0.7% Aphoo and +0.6% Apmask). For evaluation on Transformer, we follow MAE and CAE, which efficiently fine-tunes Mask R-CNN with ViT-B backbone using Xx schedule. In Tab. 4, our approach (800-epoch) is superior to popular contrastive- based and MIM methods, e.g., outperforms MAE (1600-epoch) by 0.9% Aphoz and 0.8% Apmask
Semantic segmentation on ADE2KK. We then evaluate the transferring performances on semantic segmentation with ADE20K (Zhou et al., 2019) by fine-tuning UperNet (Xiao et al., 2018). Based on ResNet-50, all CNN models are fine-tuned for 160K iterations with SGD following MoCo. Results in Tab. show that our method outperforms CL methods by at least 0.9% mloU and outperforms CIM (required extra treitrained BETT (Bao et al., 2022)) by 0.3% mloU. Based on ViT-B, we fine-tune models for 80K iterations with Adam following MAE. Tab. 4 shows that our approach consistently improves MIM methods (e.g., outperforms MAE and SimMIM by 0.9% and 0.6% mloU).
8

---

Under review as a conference paper at ICLR 2023
Figure 5: Ablation mask token in various stages Table 5: Ablation of A MIM on IN-100 and IN- (S) layers (L) based SimMIM (without free)) IK. w/o denotes removing the re-weighting
on IN-100.
term w in Lireg and (((*) denotes adding the
ResNedo
V
mask token I' to the optimal layer-(".
Replaccing
Replacing
Additon
Adtittnn
Backbones
ResNet-50
VIT-S
ViT-B
McCoVV
SiebADM
SInMIM
MAE
Datasets PT Epochs
IN-100
IN-IK
IN-100
IN-IK
6.3
ep
100 ep
400 ep
400 ep
SimMIM
87.75
78.2
85.10
83.1
Lxpo
88.19
78.4
85.27
83.2
Liveg w/O
88.47
78.4
86.05
83.3
05.
Livey
88.73
78.6
86,41
83.4
Sive T(1))
88.86
78.8
86.62
83.5
3-2
54
4.4
1.5
Stages
Layer
5.4 ABLATION STUDY
We next verify the effectiveness of the proposed components. Ablation studies conducted with ResNet-50 and ViTs on IN-100 and IN-IK using the fine-tuning protocol. Based on the modified baseline SimMIM (La we first compare different mask token mechanisms: Replacing denotes the original way in most MIM methods, and Addition denotes our proposed way that adds the mask token intermediate feature maps of the backbone. As shown in Fig. 5, adding the mask token the medium stages (stage-3) or layers (layer-S) yields the best performance. Replacing masked patches in input images by RGB mean value slightly improves the baseline SimMIM, especially for ResNet-50 (88.19 19 vs. 87.75 on IN-100). Then, we verify the proposed frey in Tab. 5. We find that simply using Greg without the adaptive re-weighting w (Eqn. 5) brings limited improvements as the frequency constraint to Lypa. while employing further enhances performances by helping the model to learn more informative frequency components. Additionally, we visualize reconstruction results in Fig. to show the improvements brought by our proposed components (more results in Appendix B).
5.5 VERIFICATION OF A MIM DESIGN RULES
Table 6: Analysis of the scalability A for advanced components on IN- 1K.
We verify whether A MIM meets the intended de- sign rules using the same experiment settings as Sec. 5.4: (i) A A is generic incorporate ad- vanced components proposed in previous works (e.g., complex decoders, advanced prediction tar- gets), As for the decoder structure, we replace the original linear decoder with 2-layer MLP or Trans- former decoders, but find limited improvements or degenerated performances (similar to SimMIM) Tab. 6. Inspired by PVT.V2 (Wang et al., 2022),
Module
ResNet-50
VIT-B
Linear
78.8
82.4
2-layer MLP
78.8
82.4
Decoder
2-layer MLP (w/ DW)
78.9
82.5
2-layer Transformer
78.6
82.3
2-layer Transformer (w/ DW)
78.8
82.6
RGB
78.8
82.4
Target
HoGi Feature
78.9
82.6
DINO Feature
78.9
82.7
we introduce a depth-wise (DW) convolution layer (w/ DW) to the MLP decoder (adding 5 x 5 DW layer in between) and the Transformer decoder (adding a 3 x 3 DW layer in each FFN (Wang et al., 2022)), which brings improvements compared to the linear decoder. As for the prediction target, follow MaskFeat to change the RGB target to the HoG feature or the output feature from ViT-B/16 pre-trained 1600-epoch by DINO (Caron et al., 2021). Tab. 6 shows that using advanced targets significantly improves the performance of A for both ResNet-50 and ViT-B. Therefore, we can conclude MIM is generally applicable framework. (ii) A enhances occlusion robustness and middle-order interaction among patches from experiments on ImageNet-IKG in Fig. A3.
CONCLUSION
In this paper, we delved deep into MIM and answered the question of what exactly is learned during MIM pre-training. We adopted multi-order interactions to study the interaction order among image patches. We discovered that MIM essentially teaches the network to learn middle-orde interactions among image patches for more complex feature extraction regardless of the network architecture. Based on our findings, we further proposed a general framework A MIM that is compatible with both Transformers and CNNs for MIM tasks aiming at enhancing patch interactions during self-supervised pre- pre-training. Besides a different mask token mechanism, we proposed a loss in the Fourier domain to better learn the middle-orde interaction. Experimental results have shown that proposed framework improves the representations learned for both CNNs and Transformers yielding superior performance than state-of-the-arts various downstream tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Hassan Akhari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Marco Ancona, Cengiz Oztireli, and Markus Gross. Explaining deep neural networks with a polynomial time algorithm for shapley value approximation. In International Conference on Machine Learning (ICML), pp. 272-281, PMLR, 2019.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning. arXiv preprint arXiv:2204.07 141,, 2022.
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli, Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint arXiv:2202.03555, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. International Conference Learning Representations (ICLR), 2022.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems (NeurIPS), 33:9912 119991299944 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the International Conference Computer Vision (ICCV), 2021.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning (ICML), pp. 1691-1703. PMLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint (xXiv:2002.05709 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. arXiv preprint arXiv:2011.10566. 2020.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04299 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9640-9649, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 702-703. 2020.
Huiqi Deng, Qihan Ren, Xu Chen, Hao Zhang, Jie Ren, and Quanshi Zhang. Discovering and explain- ing the representation bottleneck of dnns. In International Conference on Learning Representations (ICLR), 2022.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep 2018.
bidirectional transformers for language understanding. arXiv preprint
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the International Conference on Computer Vision (ICCV), 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations (ICLR), 2021.
10

---

Under review as a conference paper at ICLR 2023
Alaaeldin El-Nouby, Gautier Izacard, Hugo Touvron, Ivan Laptev, Hervé Jegou, and Edouard Grave. Are large-scale datasets necessary for self-supervised pre-training? arXiv preprint arXiv:2112.10740, 2021.
Patrick Esser, Robin Rombach, and Bjor Ommer. Taming transformers for high-resolution im- age synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12873-12883, June 2021.
Yuxin Fang, Li Dong, Hangbo Bao, Xinggang Wang, and Furu Wei. Corrupted image modeling for self-supervised visual pre-training. arXiv preprint arXiv:2202.03338, 2022.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations (ICLR), 2018.
Priya Goyal, Piotr Dollr, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jía, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in I hour. arXiv preprint
2020.
2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach self-supervised learning. In Advances in Neural Information Processing Systems (NeurlPS), 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Geoffrey E Hinton and Richard Zemel. Autoencoders, minimum description length and helmholtz free energy. Advances neural information processing systems (NeurIPS), 6, 1993.
Liming Jiang, Bo Dai, Wayne Wu, and Chen Change Loy. Focal frequency loss for image reconstrue- tion and synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13919-13929, 2021.
Alex Krizhevsky, llya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Pereira, C. J. Burges, L. Bottou, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc., 2012a.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems (NeurlPS), pp. 1097-1105, 2012b.
Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, and Stan Li. Openmixup: Open mixup toolbox and benchmark for visual representation learning, https: /ttpttitt...//ggg/// com/ West laketaaaeeaee AI/ openmi xup, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context, In Proceedings of the European Conference Computer Vision (ECCV), 2014.
11

---

Under review as a conference paper at ICLR 2023
Tsung- Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE International Conference on Computer Vision (ICCV), 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations (ICLR), 2019.
Muhammad Muzammal Naseer, Kanchana Ranasinghe, Salman H Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Intriguing properties of vision transformers. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.
Namuk Park and Songkuk Kim. Blurs behave like ensembles: Spatial smoothings to improve accuracy, uncertainty, and robustness. arXiv preprint 1rXiv:2105.12633 2021.
Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations (ICLR), 2022.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context encoders: Feature learning by inpainting, In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp. 2536-2544, 2016.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training (2018), 2018.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv, abs/2102. 12092, 2021.
Sepehr Sameni, Simon Jenni, and Paolo Favaro. Dilemma: Self-supervised shape and texture learning with transformers. arXiv preprint arXiv:2204.0470 2022.
Ramprasaath R Selvaraju, Karan Desai, Justin Johnson, and Nikhil Naik. Casting your model: Learning to localize improves self-supervised representations. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11058-11067, 2021.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. In International Conference on Machine Learning (ICML), pp. 10347-10357. PMLR, 2021.
Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pp. a8Xiv-1807, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems (NeurIPS), 30, 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning (ICMI.), pp. (096-1103, 2008
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoine Manzagol, and
Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research. 11(12), 2010.
Wenhai Wang. Enze Xie, Xiang Li. Deng- Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvtv2: Improved baselines with pyramid vision transformer. arXiv preprint arXiv:2 106.. 13797, 2022.
12

---

Under review as a conference paper at ICLR 2023
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:2112209113,, 2021.
Ross Wightman, Hugo Touvron, and Herv Jgou. Resnet strikes back: An improved training procedure in timm, 2021.
Di Wu, Siyuan Li, Zelin Zang. and Stan Z Li. Exploring localization for self-supervised fine- tne-grained contrastive learning. In Proceedings of the British Machine Vision Conference (BMVC), 2022.
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for
scene understanding. In European Conference on Computer Vision (ECCV). Springer, 2018.
Tete Xiao, Colorado Reed, Xiaolong Wang, Kurt Keutzer, and Trevor Darrell. Region similarity representation learning. arXiv preprint arXiv:2 203. 22902, 2021.
Enze Xie, Jian Ding, Wenhai Wang, Xiaohang Zhan, Hang Xu. Zhenguo Li, and Ping Luo. Detco: Unsupervised contrastive learning for object detection. In Proceedings of the International Conference on Computer Vision (ICCV), 2021a.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. arXiv preprint 2021b.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training BERT in 76 minutes. In International Conference on Learning Representations (ICLR), 2020.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6023-6032, 2019.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning (ICML), pp. 12310-12320. PMLR, 2021.
Hao Zhang, Sen Li, Yinchao Ma. Mingjie Li, Yichen Xie, and Quanshi Zhang, Interpreting and boosting dropout from a game-theoretir view. arXiv preprint arXiv:2009. 1129, 2020.
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. Proceedings of the European Conference on Computer Vision (ECCV), 2016.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmenta- tion. In Proceedings of the AAAI conference on artificial intelligence (AAAI), pp. 13001-13008, 2020.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision (IJCV), 2019.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong, ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832,
2021. 2021.
13

---

Under review as a conference paper at ICLR 2023
A DETAILS OF COMPARISON EXPERIMENTS
This section provides experimental details for Sec. 5, e.g. pre-training and evaluation on ImageNet-IK and transfer learning settings on downstream tasks.
A.1 IMAGENET-IK 1K EXPERIMENTS
Pre-training. The default settings of A MIM for ResNet-50 and ViTs are provided in Tab. AI, fol- lowing SimMIM (Xie et al., 2021b). We use AdamW (Loshchilov & Hutter, 2019) optimizer with the cosine scheduler and the linear learning rate scaling rule (Goyal al., 2020): Ir base Jr brathhssizze / 256. Similar to current MIM methods, we only use RondomResizeddCrpp with the scale of (0.67, 1.0) and do employ other complex augmentations (e.g., Rand Augment (Cubuk al., 2020), mix- ups (Yun et al., 2019), or stochastic depth) during pre-training. As for ViTs, we adopt Cosine decay for 100 and 300 epochs pre-training while using Step decay (the learning rate multiplied 0.1 at 700-epoch) for 800-epoch pre-training.
End-to-end fine-tuning, Our fine-tuning settings follow common practices of supervised image classification ImageNet- IK. As shown in Tab. A2, we fine-tune pre-trained ViTs for 100 epochs using the DeiT' (Touvron et al., 2021) training recipe, which employs Adam (Loshchilov & Hutter, 2019) optimizer with the cross-entropy (CF) loss; fine-tune pre-trained ResNet-50 for 100/300 epochs using RSB A3/A2 (Wightman et al., 2021) settings, which employs LAMB (You et al., 2020) optimizer with the binary cross-entropy (BCE) loss, Additionally, we use layer-wise learning rate decay as (Bao et al., 2022) for fine-tuning ViT models.
Table A1: ImageNet- IK A MIM pre-training settings Table A2: ImageNet- IK fine-tuning recipes for ResNet-50 (RSB A2/A3) and ViTs (DeiT).
ResNer-50 and ViT models.
Configuration
R5
ViTs
Configuration
VIIs
ResNet-S0 50
Pre-training resolution
224 224
224 x 224
RSB
Mask patch size
32 32
32 x 32
FT epochs
100
300
100
Mask ratio
60%
60%
Training
224
Testing resolution
224
224
224
Optimizer
AdamW
AdamW 10
Testing crop
0.95
Base learning rate
1.5 10
Optimizer
AdamW
LAMB
LAMB
Weight decay
0.05
0.05
Rase learning rate
10
1.5 x 10-8
10
Optimizer momentum
3.09, 0.999
Bi 8220099 0.999
Weight decay
0.05
0.02
0.02
Batch size
2048
2048
2048
Learning rate schedule
Cosine
Cosine / Step
Learning rate schedule
Cosine
Cosine
Cosine
Warmup epochs
Warmup epochs
10
10
Label smoothing
0.1
X
X
RandomeesizeddCrp
Stochastic depth
Rand Augment
Gradient clipping
5.0
X
X
Stochastic Depth
Rand Augment
0.5)
(7, 0.5)
(6, D.5)
Gradient Clipping
norm= 5
Mixup alphu
0.8
0.1
0.1
CutMix alpha
Loss function
CE loss
BCB loss
BCE loss
A.2 OBJECT DETECTION AND SEGMENTATION ON COCO
We adopt Mask-RCNN (He et al., 2017) framework perform transfer learning to object detection and segmentation on COCO (Lin et al., 2014) in Detectron2 For evaluation on ResNet-50, we follow MoCo (He al., 2020) and fine-tune tune Mask R-CNN with the pre-trained ResNet-50-CC backbone using 2x schedule (24 epochs). For evaluation of ViTs, we follow MAE (He et al., 2022), which employs the pre-trained ViT backbone and an FPN neck (Lin et al., 2017) in Mask R-CNN, and fine-tune the model using 1 x schedule (12 epochs). For a fair comparison, we follow (Bao et al., 2022; Xie et al., 2021b) to turn on relative position bias in VIT (Dosovitskiy et al., 2021) during both pre-training and transfer learning, initialized as zero.
A.3 SEMANTIC SEGMENTATION ON ADE-20K
We adopt UperNet (Xiao et al.. 2018) to perform transfer learning to semantic segmentation on ADE-20K and use the semantic segmentation implementation in MMSSemmentttion We initialize
'https:/ / http:bccooaacbbbooooo http://ccmeccccoooeeeeeeeeeeeeee Phttps:// // thub. com hoamm/ppn--nnee wtt.mm/a////mm///att......m
14

---

Under review as a conference paper at ICLR 2023
Vir-S Handere Paidkinp
BeaNedo Handom Pachilopp
VIF-5 intraction Senngh Order + Mudl
Keshic50 Irtmntion Strengh of Order
IP Predoo
fasco IP (H-I
MAd
Sunt IT Nedaa
dedick ceadie
2T14 SAAT
11-45--1T (N-7)
OcOuure 1300
Oeder Fe
Osla/a
(a)
(b)
(c)
(d)
Figure A1: (a)():: Robustness against different occlusion ratios of images (CL, vs. MIM) is studied
for both ViT-S and ResNet-50 on ImageNet-100. (c)(d): Distributions of the interaction strength J(m)
(CL vs. MIM) explored for both ViT-S and ResNet-50 on ImageNet- 100. The label indicates the
pre-training method + fine-tuning augmentation used, random stands for random weight initialization.
the UperNet using the pre-trained backbones (ResNet-50 or ViTs) on ImageNet- IK. For ViTs, we fine-tune end-to-end for 80K iterations by Adam with a batch size of 16. We search a optimal layer- wise decay from {0.8. 0.9} and search optimal a learning rate from {1 x 10-2, 2 x 10-1,3 1,3 x 10-") for all competitors. Similar to fine-tuning settings on COCO, we use relative position bias in VIT (Dosovitskiy et al., 2021) during both pre-training and transfer learning as (Bao et al., 2022; Xie al., 202 For ResNet-50, we follow MoCo (He et al., 2020), i.e., all CNN models are fine-tuned for 160K iterations by SGD with the momentum of 0.9 and a batch size of 16.
B EMPIRICAL EXPERIMENTS
This section provides background information and experimental details for Sec. 3. We also provide additional results of occlusion robustness evaluation and multi-order interaction strength.
B.1 OCCLUSION ROBUSTNESS
In Sec. 3.1, we analyze robustness against occlusion of fine-tuned models on ImageNet-100 100 (a subset on ImageNet-1 KK divided by (Tian et al., 2020)) using the official implementation" provided by Naseer et al. (2021). Both MIM and contrastive-basel methods are pre- pre-trained 400 epochs on ImageNet- 100 using their pre-training settings on ImageNet-IK. We adopt the fine-tuning training recipe as DeiT in Tab. A2 and use the same setting (100-epoch) for both ViT-S and ResNet-50. Note that modified SimMIM for ResNet-50 (replacing masked patches in the input image with the RGB mean) in all experiments.
the
As shown in Fig. and A1, we compared MIM pre-trained models supervised methods with various augmentations and contrastive learning pre-trained methods in terms of the top-1 accuracy under various occlusion ratios. We find that MIM methods show better occlusion robustness on both Transformers and CNNs. In addition to Sec. 3.1, we also provide results of salient occlusion for ViT-S and ResNet-50 on ImageNet- 100 in Fig. A2. Note that the occlusion ratio means the ratio of dropped and total patches and we plot the mean of accuracy across runs. We can conclude that MIM pre-trained models have stronger robustness against random and salient occlusions than supervised and contrastive-be based methods.
B.2 MULTI-ORDER INTERACTION
In 3.2, interpret what is learned by MIM by multi-order interaction (Deng et al., 2022: Zhang al., 2020). The interaction complexity can be represented by [(m)) (defined in Eqn. 1), which measures the average interaction utility between variables i, on all contexts consisting of m variables. Notice that the order m reflects the contextual complexity of the interaction (im) j). For example, a low-order interaction (e.g., = 0.05n) means the relatively simple collaboration between variables i,j. while a high-order interaction (e.g. m = 0.05n) corresponds to the complex collaboration. As figured out in the representation bottleneck (Deng et al., 2022), deep neural networks (DNNs) are more likely to encode both low-order interactions and high-order interactions, but often fail to learn middle-order interactions. We hypothesize that MIM helps models learn more middle-order
Phtpps: / / thub. com Momzamamma 1 Na Naseer Ittprr/Inrigggg wtwppiigggrooopeeeeeeeeeeee ooless-o--issse -n noformers
15

---

Under review as a conference paper at ICLR 2023
VIT-S Runden DackDry
VIT-5 Salten Deih
BosNerSO Recdiv Pashhoop
RosNato Sélier PuckDooy
Vocun #7 (ril)
LV5- (5) x
Noraas (H-T)
Top- Loooo (80
Twg Aucuum (5)
Funden (iit)
+ 2Ai
Eavie 777 Doede
Voew PT
Radin
15 41
0
100
10
31
52
15
(a)
(b)
(c)
(d)
Figure A2: Robustness against various random or salient occlusion ratios of images is studied in (a)(b) for ViT-S, and (c) for ResNet-50 using various experimental settings on ImageNet- The label indicates the pre- vre-training method + fine-tuning setting used, random stands for random weight initialization.
VI Randern PrsDDrrp
BeaNedo Handom Pachhltop
VIT -5 betracion Stmpph Osler
Keshicbti brianatenn Strength Crter
MiCWi (AX)
PyTwa- (1100)
Pylicd -1
StotMefT JAN
30-15M--1 (3UT)
20
50 90 DatA4 (5)
12
58
40
130
0.5
Obd
Ooder
Oeder/n
(b)
(c)
(d)
Figure A3: Verification of robustness and interaction of A with ViT-S and ResNer-50 on ImageNet- IK (a)(b): Robustness against different occlusion ratios of images is studied for A MIM and various methods. (c): Distributions of the interaction strength ((m) are explored.
interactions since MIM has a natural advantage in cases where some parts of the image are masked out. In Fig. 1, we calculate the interaction strength ((m) (defined in Eqn. 2) for fine-tuned models on ImageNet- 100 using the official implementation" provided by Deng et al. (2022). Specially, we use the image of 224 x 224 resolution as the input and calculate ((m) on 14 x 14 grids, i.e., n = 14 x 14. P(j=ylans)
And we set the model output as f(z) log 1- P()=vl=s)) given the masked sample æs, where denotes the ground-trute -ruth label and P(O = y و denotes the probability of classifying the masked sample Vy to the true category.
B.3 MIM FROM FREQUENCY PERSPECTIVE
We first plot the log magnitude of Fourier transformed feature maps of ResNet-50 with different pre- training methods using the tools provided by Park & Kim (2022) on ImageNet- K. Following (Park & Kim, 2022), first convert feature maps into the frequency domain and represent them on the normalized frequency domain (the highest frequency components are at {-T, +)) In Fig. 4(a), we report the amplitude ratio of high-frequency components by using A log amplitude. As shown in Fig. 4(a), inpainting and MIM show similar low-pass filtering effects at convolution layers as compared to contrastive learning. This indicates that inpainting and MIM reduce noise and uncertainty induced by high-frequency features. We argue that the reconstruction performance of MIM is mainly related to low or high-order interactions of patches (Deng et al., 2022), while reconstruction performance is not directly related to the learned representation quality. Then, we provide the standard deviation of feature maps by block depth as (Park & Kim, 2022; 2021), which first calculates the feature map variance on the last two dimensions and then averages over the channel dimension for the whole dataset. Fig. 4(b) shows the feature variance of each layer of ResNet-50 with different pre-training methods on IN- This figure indicates that MIM tends to reduce the feature map variance, and conversely, supervised training, inpainting, and contrastive learning based on CNN tend to increase variance. Compared to MIM, which learns better middle-order interactions, the inpainting task fails to filter out low-order interactions and thus leads to higher variance. To conclude, MIM methods learn middle-order interactions and reduce the feature map uncertainty (high frequencies) based on the CNN encoder for a generalized and stabilized feature extraction.
'https: http://b///tttbb...///// wtw.:tttt..oooo.eoeoeoooooooeooe www./owo///llaaoooee 'https: / /i thub. com/xxxne1 1 hhww//hww//o do-vi ta-work
122000//botteem
16

---

Under review as a conference paper at ICLR 2023
BYOL
BYOL
3.5
4.00
Mocovs
Inpaiting
Feature Map Vari arfaece
Inpanting
M Log Ampittude
SimMIN DelT (Sup.)
DelT (Sup.)
SimMIM
Random
4.8-
5.8
3.0
0.0
34 366 38
0.0
5.4 Du8
3.0
Normalized Depth
Normallizdd Depth
(a)
(b)
Figure A4: (a) Fourier transformed feature maps. The vertical axis the relative log amplitudes of
the high-frequency components, and the horizontal axis the normalized depth of the network. The blue columns indicate the pooling layers, while the white columns indicate the convolution layers. (b) Feature maps variance. The vertical axis the average variance value of feature maps. DeiT (Sup.) is supervised pre-tre traininggn The results of the randomly initialized network are plotted for reference.
Raw image
Prediction image
L frog wlo
L freg
Fox
Fourier spectrum
Fourier spectrum
Loss weight (Fourier spectrum)
Figure A5: Visualization of predicted images and freg loss weight in Fourier domain. From the view of the Fourier spectrum, the raw image (left) contains 99% low-frequency components (usually present contents) and rich medium-frequeng, (structural patterns) and high- figgufrryeency components (local details and noises), while the predicted result (middle) provides fewer medium or high- frequency components. Calculated in the Fourier domain, the loss weights (right) of f w/o w help the model to learn the full spectrum while fireg focusing on the low and medium-fremt medium-frequency parts, which are more likely to be low-order or middle-order interactions.
C MORE EXPERIMENT RESULTS
C.I ABLATION OF THE PROPOSED MODULES
In addition to ablation studies in Sec. 5.4, we provide ablation study on the proposed frreg in the Fourier domain, as shown in Figure A5, As we discussed Sec, 4, we hypothesize that learning medium frequencies would help better learn middle-order interactions. tackle the dilemma of Lepas which tends learn cow-frquency components (i.e., contents reflected by high-order interactions). Although the reconstruction loss in the Fourier domain has global perception, the figh-freeuency components are usually constructed by local details and noises (i.e., low-order interactions), which might hurt the generalization abilities. Therefore, we introduce the reweight w(u, v) to force the model to learn more medium medium-frequency components, which are identical middle-order interactions. Then, we perform further analysis of the masked patch size for M in Tab. A3. Note that we pre-train ResNet-50 for 100 epochs and ViT-B for 400 epochs on ImageNet- IK and report the fine-tur finettuning results. As shown in Tab. A3, when the mask ratio is 60%, the optimal masked patch size is 32 x 32 for A MIM, which is the same as SimMIM.
thereby propose L to
Table A3: Ablation of masked patch size for A MIM based on ResNet-50 and ViT-B ImageNet-IK
Model
Masked
Mask
PT
Top-1 Accuracy (%)
patch size
ratio
epoch
ResNet-50
8 16633//66 64
0.6
100
78.2/78.6 78.8 78.7
VIT-B
8/ 16 32 64
0.6
400
82.9/ 83.4 83.5 83.3
17

---

Under review as a conference paper at ICLR 2023
C.2 ANALYSIS OCCLUSION ROBUSTNESS AND INTERACTION OF A
We further analyze occlusion robustness and interaction strength of A with VIT-S (pre-training 400-epoch) and ResNet-50 (pre-training 100-epoch) on ImageNet- IK shown in Fig. A3. Fig. 3(a) and 3(b) shows that A MIM is more robust to occlusion than the baseline SimMIM and contrastive learning methods with both Transformers and CNNs. Meanwhile, we find that MIM methods learn more balanced interaction strength than both supervised and contrastive learning methods in Fig. 3(c) and 3()). MIM further improves SimMIM by capturing more middle-order interactions (0.2n to 0.6n) with Transformers and CNNs. Therefore, we can conclude that MIM helps the model to learn better middle-order interactions between patches for more generalized visual representation.
C.3 SCALING-UP UP MIM
Additionally, scale up the model size of backhone encoders verify the performance of MIM with ResNet and ViT on ImageNet-IK, As shown in Table A4, our proposed MIM and its advanced variant MIM consistently improve both the contrastive-based and MIM methods on all scale architectures, e.g., AIMI outperforms SimMIM by 1.5%/0.5%/0.5% ViT-S/B/L/H and ResNet-50/ 101, demonstrating that MIM is an architecture acchtecture-agnostii and salable framework for MIM pre- pre-training.
and 0.6%/0.49% based on
Table A4: ImageNet- 1K fine-tuning (FT) top-1 accuracy (%) with ResNet (R) and ViT of various model scales. We adopt the 100-epoch fine-tuning protocols for both architectures.
Methods
Supervision
ViT-S
ViT-B
ViT-L
ViT-H
R-50
R-101
Sup.
Label
79.9
81.8
82.6
83.1
78.1
79.8
MoCoV3
CL
81.4
83.2
84.1
78.7
DINO
CL
81.5
83.6
78.7
MAE
RGB
83.6
85.9
86.9
77.1
SimMIM
RGB
81.7
83.8
85.6
86.8
78.2
80.0
MaskFeat
HoG
84.0
85.7
78.4
MIM
RGB
82.2
84.2
86.1
87.0
78.8
80.4
A M
HoG
82.4
84.5
86.3
87.1
78.9
80.5
18

---

Under review as a conference paper at ICLR 2023
D VISUALIZATION EXPERIMENTAL DETAILS
In addition to visualization results in Sec. 5.4, we visualize more reconstruction results of MIM here. Similar to Fig. ablate the proposed components in A based on ResNet-50 Fig. A6, which demonstrates that A MIM helps ResNet-50 learn more spatial details, i.e., more middle-order interactions. Moreover, we study the effects of the mask token both ViTs and CNNs in Fig. A7.
Zero mask
RGB mean mask
+T((*)
+
Fox
Cucumber
Balloon
Raw image
Masked image
SimMIM
Masked image
AIM
Figure A6: Visualizations predicted results from SimMIM (middle) and our A 2 (right) based on ResNet-50 pre-trained 100-epochs on ImageNet-IK. Notice that T(s*) denotes the mask token to the optimal stage-s in ResNet-50. the baseline SimMIM: replacing the mask with the RGB baseline) and adding the mask token T(s*) ) relieve grid-like artifacts in predicted results; adding the proposed freg helps the model to capture informative details.
ablate the proposed components by adding them to
mask (the modified SimMIM
VTT-B
VIT-B
ResNet-50
Coldfish
Balloon
Rew image
Masked Image
SimMIM
Remove
A' MIM
learned mask taken
A MIM
Remove mask token
mask token
Figure A7: Visualizations of predicted results with and without the mask token ImageNet- IK. Notice that mask tokens are adopted in the pre- treiteained models based on ViT-S (300-epoch) or ResNet-50 (100-epoch). Based on VIT-S, removing the mask token corrupts both contents of masked patches and overall colors in SimMIM while only corrupting the masked contents in AMMIM. Based on ResNet-50, removing the mask token slightly affects spatial details in the masked patches and causes grid-like artifacts in the unmasked patches. The different effects of the mask token in VIT-S and ResNet-50 might be because the two architectures use different spatial-ma -ixiin operators and normalization layers. As for ViTs, the self-attention operation captures informative details from unmasked patches, but the non-overlap patch embedding and layer normalization mask each patch isolated. The mask token learss the mean templates (contents) of masked patches and gathers spatial details from unmasked patches by the self-attention operation. As for CNNs, each patch shares the same contents extracted by batch normalization layers, and the convolution operation extract features from unmasked and masked patches equally. The mask token learns more high-freeg frequency and informative details.
19


Reviews:
[{"review_id": "wPGr2xr6AH", "reviewer": "Reviewer_dEJJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the problem that MIM is compatible with the Transformer family but is incompatible with CNNs. To this end, it proposed an Architecture-Agnostic Masked Image Modeling framework (A2MIM) that is compatible with both Transformers and CNNs in a unified way. Specifically, this paper used RGB Mean as masked tokens and added mask tokens on the intermediate feature maps. A frequency domain reconstruction loss is also applied to improve the ability of learned models. Experiments are conducted on ImageNet-1K, downstream COCO detection and segmentation, and ADE20K dataset with ViT models and ResNet-50.", "summary_of_the_review": "This paper conducted sufficient experiments. However, the method itself is not novel with marginal improvement, the novelty and originality of the method are basically not very strong. Also, many statements in this paper are slightly over-claimed (details please refer to \"Strength And Weaknesses\"). Thus, I tend to reject.", "strength_and_weaknesses": "Strengths:\n\n    - MIM is an interesting problem/framework to study, and understanding how MIM-based methods perform well in the vision domain is important for this field.\n\n    - The proposed method is clear to understand and is easy to follow by other researchers.\n\n    - The experiments are extensive on ImageNet-1K, COCO, and ADE20K datasets.\n\nWeaknesses:\n\n   - Some statements are incorrect or even a little bit over-claimed, such as:\n\n1)\tIn the abstract, the authors stated: “MIM primarily works for the Transformer family but is incompatible with CNNs.” And in the introduction section, this paper stated \"To the best of our knowledge, we are the first to carry out MIM on CNNs that outperforms contrastive learning counterparts.\" This is not true as ConvNext [1] is adequate to handle MIM learning strategy. And this paper with CNN is actually similar to it when using patchified images with CNN.\n\n[1] Liu, Zhuang, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. \"A convnet for the 2020s.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11986. 2022.\n\n2)\t“Based on this fact, we propose an Architecture-Agnostic Masked Image Modeling framework (A2MIM), which is compatible with both Transformers and CNNs in a unified way.” The proposed framework is basically similar to the regular MIM approach with trivial or not significant modifications (use RGB Mean as the Masked Tokens and add mask token on the intermediate feature map). Thus, the called \"a unified way\" seems incorrect. The used Fourier/Frequency domain is also not related to the core framework in this paper and can straightforwardly be applied in the vanilla MIM models.\n\n3)\tThe authors stated that the proposed framework has “(a) No complex or non-generic designs are adopted to ensure compatibility with all network architectures. (b) Better middle order interactions between patches for more generalized feature extraction.” I think this is because the proposed framework is too close to the original MIM architecture.\n\n4)\tThe authors claimed they “delved deep into MIM and answered the question of what exactly is learned during MIM pre-training.” This statement is very strong, however, after reading this paper carefully, I still did not get the insights or intuition about what MIM pre-training learns from the input data from the paper’s descriptions. Proper explorations and explanations are necessary to support this claim.\n\n - The performance in this paper is not competitive. As shown in Tables 1, 2, 4, etc., the improvement is fairly marginal (0%~0.2%). Considering that this paper used extra Fourier/Frequency domain reconstruction supervision/loss, I’m not sure whether the proposed strategy is truly effective or not.\n\n - The writing and organization of this paper can be improved. The used Fourier/Frequency domain reconstruction loss seems not related to the key approach of the architecture-agnostic masked image modeling framework, since it can also be applied to the regular MIM frameworks. Moreover, the insights of using this additional supervision are not clearly expressed. This part seems fragmented from others in the method.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "r63j3gCaCLQ", "reviewer": "Reviewer_RicF", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a variant of BERT-like pretraining method for computer vision (the so-called Masked Image Modeling). The method can generalize well to both vision transformers and CNNs, thus is described as architecture agnostic.\n\nThe main contributions of this work are:\n- using the mean color values to fill the corrupted pixels, which makes the self-supervised pipeline suitable for different neural network architectures;\n- giving a nice insight that the middle-order interactions are important for visual representation, and making two improvements to standard BERT algorithm to facilitate their learning (i. masking intermediate features and ii. introducing supervision in the frequency domain);\n- and providing some empirical evidence that supports its validity.\n\n&nbsp;", "summary_of_the_review": "Overall, I find this work provides some interesting insights, but the writing and presentation of the paper leaves much to be desired.\nIn addition, the methodological improvements to BERT pre-training in this paper seems to be trivial and piecemeal.\nI hope to understand this article better in further communication with the authors and that may influence my judgment.", "strength_and_weaknesses": "**[strengths]**\n- [S1] This work presents a *fresh* perspective to rethink the self-supervised learning in computer vision, say, the *middle-order* interaction. The \"interaction strength\" mentioned in the paper is a useful indicator to show how good the model is at modeling long-range dependency, which is actually considered to be a valuable aspect in advancing pre-training for NLP [1,2].\n- [S2] The authors have put considerable effort into how to fairly compare the established algorithms, which is appreciated and allows the hypotheses presented in their paper to be fully tested. The ablation experiments also demonstrate the validity of different components in their method.\n\n\n&nbsp;\n\n\n**[weakness]**\n\nThere are some parts of the article that are not very easy for the reader to follow, which could be the main weakness.\nI would suggest authors to explain more about the following aspects (which I find enlightening) and eventually add them to their manuscript or appendix:\n  - [W1] In the Introduction, the authors claim that \"it is not straightforward to directly apply mask token to CNNs\". Can we replace the masked part (e.g., 16x16x3=768 pixels) with a \"mask token\" (e.g., 768 learnable scalars) like BERT does? This is a straightforward way to mask. Or would it be more appropriate to describe this way as \"underperforming\" than \"not straightforward\"? (considering the mediocre performance of MAE, SimMIM, etc. in Tab. 1 of the paper)\n  - [W2] \"MAE takes the reorganized the unmasked input patches of 112x112 as the input\", can the authors explain more on this? I wonder if a substitution like the above (768 pixels to a 768-dimensional learnable vector) or some other operation has been performed.\n  - [W3] According to the analysis in Sec. 3.2, middle-order interactions seem to manifest a medium- or long-range inter-patch dependency. Why the authors say \"middle-order interactions could be enhanced via guiding the network to learn features of certain frequencies\"? Does \"certain frequencies\" refer to medium or high frequencies? Why does increasing the richness of these frequencies on feature maps can promote middle-order interactions?\n\n[W4] In addition, there appears to be some related work that has not been adequately discussed. See the \"Clarity, Quality, Novelty And Reproducibility\".\n\n&nbsp;\n\n**[open questions]**\n\n[O1] The methodological improvements to BERT-like pre-training in this paper seems to be incremental. \nConsidering that the authors propose some quantitative metrics to describe middle-order operations, is it possible to design a more principled algorithm to explicitly facilitate such middle-order interactions? I believe the insights on middle-order are valuable, but the solutions proposed in the article do not seem to fully exploit their values.\n\n&nbsp;\n\n------------------\n[1] Jawahar, Ganesh, Benoît Sagot, and Djamé Seddah. \"What Does BERT Learn about the Structure of Language?.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.\n\n[2] Xu, Jiacheng, et al. \"Discourse-Aware Neural Extractive Text Summarization.\" Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.\n\n&nbsp;\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VOpndu5oY8", "reviewer": "Reviewer_yVqw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper inspects MIM pre-training approaches and finds MIM essentially helps the model to learn better middle-order interactions between patches. Motivated by that, this paper proposes a novel MIM-based method, dubbed A$^2$MIM, that works well for both (small-sized) ConvNets & ViTs. Extensive experiments are conducted to verify the effectiveness of the proposed approach.", "summary_of_the_review": "Overall, I think this is a solid and well-written paper.\n", "strength_and_weaknesses": "### Strength \n1. This paper provides a thoroughgoing study of MIM, and highlights the essence of MIM pre-training is to help models learn middle-order interactions between patches.\n2. Based on the study of middle-order interactions of MIM, this paper proposes a novel approach that works well for both ConvNets as well as ViTs. What's more, I appreciate the management of technical details in this paper, such as ``Filling Masked Tokens with RGB Mean'' is technically sound. \n3. Extensive experiments on several downstream visual recognition tasks are conducted to verify the effectiveness of the proposed approach.\n   \n### Weaknesses\n1. The 2nd paragraph's name of Sec. 2 Related Work is inaccurate.``Autoregressive Modeling'' means that the output depends linearly on its own *previous* values (e.g, GPTs), while masked modeling is *bidirectional* modeling.\n2. I don't see a strong connection between Sec. 3.1 and the proposed approach.\n3. The scalability of A$^2$MIM is unknown, which is a crucial property for pre-training.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NZgVHfcUF1e", "reviewer": "Reviewer_nEcw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyzes the essence of MIM and introduces a universal MIM method that can be applied to both CNNs and Transformers. There are several designs that together build the introduced method, including the mean RGB replacement, the intermediate mask, the frequency reconstruction targets and the HOG targets. Experimental results on both ResNet-50 and ViTs are shown.", "summary_of_the_review": "I tend to weakly reject this paper in the initial comments due to the unfounded premises. I am looking forward to the authors' opinions in the rebuttal period on why we need MIM for CNNs.", "strength_and_weaknesses": "### Strengths\n1. The paper is well-organized and easy to follow.\n2. The figures, tables, and visualizations indeed help to understand the author's point of view.\n3. It is new to apply MIM on convolution neural networks.\n\n### Weaknesses\n1. I have the most concern about the motivation for this work. As we know, ViTs lack the capture of inductive bias and MIM solves this issue to some extent by enforcing visual context reasoning. However, CNNs are different, they are good at capturing inductive bias. Furthermore, as observed in the authors' results (Tab. 1), a lot of effort has been done to do MIM pre-training, however, the final results are almost the same as supervised counterparts. So is it a false proposition to apply MIM to CNNs?\n2. I have another concern about the authors' declaration that MIM enables the network with a better feature extraction ability. As a common observation in previous MIM works, the linearly probing results of MIM-pretrained models are unsatisfactory, worse than the contrastive methods. MIM is known to provide transferable model parameters rather than extracting out-of-the-box features.\n3. In Sec. 3.1, the authors analyze different augmentation methods, however, the setup is not fair. MIM is employed for pretraining while the others (e.g., CutMix) are for fine-tuning. \n4. There are many components in the introduced method, including the mean RGB replacement, the intermediate mask, the frequency reconstruction targets and the HOG targets. How much does each component affect?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject